Semantic-Conditioned TriStreamFusion for Cross-Dataset Deepfake Detection

초록

Deepfake detectors often degrade under changes in manipulation methods, compression, resolution, and capture pipelines. We present TriStreamFusion, which combines a ConvNeXtV2-Tiny RGB stream, a ResNet-50 wavelet stream using a nine-channel Sym4 level-2 stationary-wavelet representation, and semantic features from a frozen CLIP ViT-B/32 encoder. RGB and wavelet embeddings remain two main query tokens, while five CLS-guided CLIP patch tokens form an auxiliary key-value sequence, yielding a 2×5 attention map per head. The refined-main summary is concatenated with projected RGB, wavelet, and CLS-derived global semantic features for classification. Trained on FaceForensics++, the model is evaluated at video level on Celeb-DF v2, DeepfakeTIMIT, DFDC, andWildDeepfake without target-domain fine-tuning. Five-run evaluation yields 74.33% accuracy, 83.60% F1-Binary, 61.35% balanced accuracy, and 70.47% AUROC on the pooled test set. Excluding fake-only DeepfakeTIMIT yields 73.38%, 82.82%, 61.13%, and 70.44%, respectively. The predefined five-token setting gives the highest pooled balanced accuracy without DeepfakeTIMIT in the post-hoc sensitivity table. On DFDC, all 3,408 videos are classified as fake at threshold 0.5. A label-free Otsu threshold improves specificity to 48.21%, but balanced accuracy reaches only 51.13%, with AUROC unchanged at 52.25%. Overall, TriStreamFusion improves several pooled and WildDeepfake metrics, while calibration and domain-general ranking remain unresolved under severe distribution shifts. © 2013 IEEE.

키워드