CLIP Feature Selection Mechanism via Sparse Autoencoder for Generalized Deepfake Detection

Generating multimedia contents, particularly images, by using AI-based tool is becoming an everyday practice generally just for fair applications but always more for malevolent aims such as disinformation, defamation and blaming. Reliably detecting synthetic generated pictures is becoming crucial. However, new generative models are emerging at a much faster pace than the development of accurate deepfake detectors. According to this, it is fundamental to improve generalization capabilities in order to preserve performance in front of content created by unknown and new-emerging generation methods. This paper investigates in this direction proposing the idea to select a reduced set of CLIP-based features by resorting to a sparse autoencoder (SAE); such a set should ideally be termed the features of fake and could provide an improved generalization capacity. Experimental results carried out on extended datasets highlight this notable behavior and diversely designed detectors, based on this approach, achieve state-of-the-art performances.