전역-국소-문맥 정보의 어텐션 기반 게이팅 융합을 통한 멀티모달 유해 밈 탐지 모델

초록

Detecting harmfulness in multimodal memes is challenging because it often arises from implicit image–text interactions that single-modality models cannot capture. We propose a multimodal detector that progressively refines global, local, and contextual cues and fuses them via a lightweight gating mechanism. We evaluate the model on Harm-C (COVID-19 domain) and Hateful Memes, where harmfulness is revealed only through cross-modal composition. Our baselines include image-only models (ResNet-50, EfficientNet-B3, ViT-B/16), text-only models (BERT-base, RoBERTa-base), and multimodal baselines (Late Fusion, CLIP-Linear/MLP). Our method achieves an accuracy of 0.8430±0.0071 and an F1 score of 0.8297±0.0098 on Harm-C, as well as an accuracy of 0.7208±0.0124 and an F1 score of 0.6706±0.0279 on Hateful Memes, outperforming all baselines, including CLIP-based variants. Ablation studies identify MultiAttention and ContextGen as the primary contributors across both datasets.

키워드