Towards Benchmarking and Evaluating Deepfake Detection

by   Chenhao Lin, et al.
Xi'an Jiaotong University
Tsinghua University
Wuhan University

Deepfake detection automatically recognizes the manipulated medias through the analysis of the difference between manipulated and non-altered videos. It is natural to ask which are the top performers among the existing deepfake detection approaches to identify promising research directions and provide practical guidance. Unfortunately, it's difficult to conduct a sound benchmarking comparison of existing detection approaches using the results in the literature because evaluation conditions are inconsistent across studies. Our objective is to establish a comprehensive and consistent benchmark, to develop a repeatable evaluation procedure, and to measure the performance of a range of detection approaches so that the results can be compared soundly. A challenging dataset consisting of the manipulated samples generated by more than 13 different methods has been collected, and 11 popular detection approaches (9 algorithms) from the existing literature have been implemented and evaluated with 6 fair-minded and practical evaluation metrics. Finally, 92 models have been trained and 644 experiments have been performed for the evaluation. The results along with the shared data and evaluation methodology constitute a benchmark for comparing deepfake detection approaches and measuring progress.


SoK: Comparing Different Membership Inference Attacks with a Comprehensive Benchmark

Membership inference (MI) attacks threaten user privacy through determin...

Image Matching: An Application-oriented Benchmark

Image matching approaches have been widely used in computer vision appli...

A Framework and Benchmarking Study for Counterfactual Generating Methods on Tabular Data

Counterfactual explanations are viewed as an effective way to explain ma...

FedEval: A Benchmark System with a Comprehensive Evaluation Model for Federated Learning

As an innovative solution for privacy-preserving machine learning (ML), ...

My Fuzzer Beats Them All! Developing a Framework for Fair Evaluation and Comparison of Fuzzers

Fuzzing has become one of the most popular techniques to identify bugs i...

Explainable Fuzzer Evaluation

While the aim of fuzzer evaluation is to establish fuzzer performance in...

A Survey on the Evaluation of Clone Detection Performance and Benchmarking

There are a great many clone detection tools proposed in the literature....

Please sign up or login with your details

Forgot password? Click here to reset