Detection of face-swapped fake videos using human gaze information

Abstract

The relevance of fake face video detection is driven by the rapid improvement of face synthesis and face swapping techniques, which makes traditional forgery cues less robust when transferred to new generation methods and recording conditions. Common approaches based on the analysis of individual frames usually rely on local visual artifacts that may disappear after compression, re-encoding, and post-processing, and they also require a close match between the distributions of training and deployment data. Such methods do not exploit information about human behavior over time and do not make use of screen interaction scenarios. In this paper, we propose a cooperative approach in which the forgery cue is given by the calibration parameters of a gaze estimation model computed from a video track obtained while a subject sequentially fixates on points displayed on the screen. A simple classifier is trained using these parameters, and their fusion with the outputs of standard fake video detectors is also considered. It is shown that the calibration parameters alone are sufficient for reliable separation of authentic and synthesized tracks and, in several settings, outperform detectors trained only on visual features, while their fusion with the outputs of such detectors provides an additional gain in accuracy. Feature importance analysis indicates a dominant contribution of the parameters associated with the vertical component of gaze direction, which is consistent with the assumption that current generators are more vulnerable when synthesizing realistic eye appearance at extreme upward and downward gaze angles.