CN113822160A - Evaluation method, system and equipment of deep forgery detection model - Google Patents
Evaluation method, system and equipment of deep forgery detection model Download PDFInfo
- Publication number
- CN113822160A CN113822160A CN202110963494.3A CN202110963494A CN113822160A CN 113822160 A CN113822160 A CN 113822160A CN 202110963494 A CN202110963494 A CN 202110963494A CN 113822160 A CN113822160 A CN 113822160A
- Authority
- CN
- China
- Prior art keywords
- data
- detection model
- index value
- deep
- forgery
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Abstract
The invention belongs to the field of image processing, and discloses an evaluation method, a system and equipment of a depth forgery detection model, which comprises the following steps: respectively training the depth forgery detection models to be evaluated through preset depth forgery data sets of various types according to a training method of the depth forgery detection models to be evaluated to obtain the trained depth forgery detection models; and testing each trained depth forgery detection model through a preset diversified sample set, obtaining the accuracy index value, the generalization index value, the robustness index value and the practicability index value of the trained depth forgery detection model under the same distribution data, and the generalization index value, the robustness index value and the practicability index value under the non-same distribution data, and performing weighted superposition according to preset weights to obtain the evaluation result of the depth forgery detection model to be evaluated. An accurate, fair and comprehensive evaluation method is established, and the obtained evaluation result is more in line with the actual situation of the deep forgery detection model.
Description
Technical Field
The invention belongs to the field of image processing, and relates to an evaluation method, system and device for a depth forgery detection model.
Background
In recent years, artificial intelligence technology represented by a deep learning algorithm is continuously developed and innovated, so that solutions of many tasks in the field of computer vision are continuously broken through, and the successful application of the artificial intelligence technology brings convenience for life and social production, such as an intelligent video monitoring scene, an automatic driving scene, an intelligent medical scene and the like. However, abusing such technologies may pose a huge challenge to personal privacy protection, and recently proposed deep learning based deep forgery technology (deep fake) misleads people to believe false words in videos by tampering with or replacing face information of original videos, which constitutes a new threat to invasion of privacy, making false words, and disturbing national security. In the face of malicious propagation of deep-forged videos and pictures, related detection technologies are increasingly valued by researchers. At present, although a plurality of large-scale deep forgery data sets and detection methods are proposed, the detection accuracy of each model is difficult to judge fairly due to the fact that the training and reasoning deep forgery data sets selected by the detection models are inconsistent and the selected evaluation indexes are too single.
In view of the above problems, with the advent of large-scale deep-forgery-inhibited data sets in recent years, some research efforts have been made to make preliminary attempts at establishing a deep-forgery-inhibited detection reference. For example, a learner proposes a continuously updated online evaluation method, and an owner of the detection method can test a model of the online evaluation method by using a test deep forgery data set provided by a website and upload an inference result, and then the website audits the result and issues an evaluation index score to reference information maintained by the website. However, this benchmarking work has its limitations, firstly, it lacks detailed description of important information related to the benchmarking deep-falsification data set, such as data scale and falsification type of the falsification data therein, and secondly, it does not strictly control the training process and training data of the detection method taking part in the evaluation, but only provides a submission guide and guides the participants to submit offline detection results by themselves, which cannot guarantee fair evaluation of different methods. Another method of evaluation, a large scale depth forgery data set is presented and a depth forgery detection game is organized using the depth forgery data set. But because it only evaluates the methods that appear in its race and does not set any restrictions on the training process of the race method, it lacks a strict fair evaluation of the existing mainstream test methods.
In addition to this, the current evaluation of deep forgery detection models is unfair and inadequate for the following reasons, resulting in inaccurate results. First, many evaluation works of deep false detection models utilize models trained on different training deep false data sets for evaluation, for example, some methods directly apply publicly available trained models in evaluation instead of re-implementing the models and evaluating them using the same training data, and such evaluation work of inconsistent training deep false data sets may result in unfair and incorrect comparison among the methods. Secondly, most of the deep false detection models are trained and evaluated on a same-distribution deep false data set generated by only including a limited false generation method, so that the problems of overfitting and poor mobility exist, and the performance of most of the detection models with excellent performance is greatly reduced when the detection models are actually applied in a real scene.
Disclosure of Invention
The present invention is directed to overcome the above-mentioned shortcomings in the prior art, and provides a method, a system and a device for evaluating a deep forgery detection model.
In order to achieve the purpose, the invention adopts the following technical scheme to realize the purpose:
in a first aspect of the present invention, a method for evaluating a deep forgery detection model includes the steps of:
obtaining a training method of a to-be-evaluated deep forgery detection model;
respectively training the depth forgery detection models to be evaluated through preset depth forgery data sets of various types according to a training method of the depth forgery detection models to be evaluated to obtain the trained depth forgery detection models;
testing each trained deep forgery detection model through a preset diversified difficult sample set and various types of deep forgery data sets, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value and a robustness index value under the non-same distribution data;
and carrying out weighted superposition on the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
The evaluation method of the deep forgery detection model is further improved as follows:
the types of the depth forgery data sets include at least two of a depth forgery data set that generates dummy data based on a GAN generation method, a depth forgery data set that generates dummy data based on a self-coder generation method, a depth forgery data set that generates dummy data based on a graphical generation method, and a depth forgery data set that generates dummy data based on a GAN generation method, a depth forgery data set that generates dummy data based on a self-coder generation method, and a graphical generation method.
The preset deep forgery data sets of various types comprise a training set, a verification set and a test set; and the proportions among the training set, the verification set and the test set are the same in each type of deep forgery data set.
Performing video frame extraction, image face extraction or face correction processing on sample data in the depth forgery data sets before respectively training a depth forgery detection model to be evaluated through preset depth forgery data sets of various types;
or, a sample data preprocessing method for the deep forgery detection model to be evaluated is obtained, and the sample data in the deep forgery data set is preprocessed according to the sample data preprocessing method.
The diversified sample set comprises standard sample data and disturbance sample data;
the standard sample data comprises automatic standard sample data and manual standard sample data, and the automatic standard sample data is obtained in the following mode: predicting the false score value of each sample data in each type of deep forgery data set through a trained deep forgery detection model, and taking the sample data with the false score value smaller than a preset false score threshold value as automatic standard sample data; the manual standard sample data is obtained by the following method: observing the truth of each sample data in each type of deep forgery data set through a preset number of users, and taking more than half of sample data judged by the users as artificial standard sample data; the perturbation sample data is obtained by the following method: and adding a preset type of disturbance to the standard sample data to obtain disturbance sample data.
The preset type of disturbance includes one or more of gaussian blur, white gaussian noise, color contrast change, and color saturation change.
And selecting real sample data from the deep forged data sets of various types as standard sample data until the number of the false sample data in all the standard sample data is the same as that of the real sample data.
The accuracy index values comprise AUC, accuracy and precision; the generalization index value comprises AUC, accuracy and precision; the robustness index value comprises an area under the curve value of the disturbance degree-AUC curve; the practical index value comprises the ratio of a longitudinal axis to a transverse axis of the model parameter-AUC scatter diagram, the ratio of a longitudinal axis to a transverse axis of the model required calculation force-AUC scatter diagram and the ratio of a longitudinal axis to a transverse axis of the model inference time-AUC scatter diagram.
In a second aspect of the present invention, an evaluation system for a deep forgery detection model includes:
the acquisition module is used for acquiring a training method of the depth forgery detection model to be evaluated;
the training module is used for respectively training the depth forgery detection model to be evaluated according to the training method of the depth forgery detection model to be evaluated and through preset depth forgery data sets of various types to obtain various trained depth forgery detection models;
the testing module is used for testing each trained deep forgery detection model through a preset diversified difficult sample set and various types of deep forgery data sets, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value and a robustness index value under the different distribution data;
and the evaluation module is used for weighting and superposing the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
In a third aspect of the present invention, a terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the method for evaluating a deep forgery detection model when executing the computer program.
Compared with the prior art, the invention has the following beneficial effects:
the evaluation method of the deep forgery detection model completely reproduces the training process of the deep forgery detection model to be evaluated by the training method of obtaining the deep forgery detection model to be evaluated, and then trains on the preset deep forgery data sets of various types to obtain the training models of the deep forgery data sets of various types so as to ensure the fairness of evaluation in and among the deep forgery data sets of various types at the later stage. Then, based on a preset diversified sample set, the test set contains a forged video with extremely high deception degree for both human eyes and detection algorithm, the videos are generated by various classical depth forgery generation methods, have the characteristics of diversification and strong challenge, can comprehensively evaluate a depth forgery detection model to be evaluated, and provide four evaluation indexes, namely, an accuracy index value under the same distribution data, a generalization index value, a robustness index value and a practicality index value under the non-same distribution data, aiming at comprehensively and comprehensively evaluating a deep forgery detection model from the aspects of accuracy, generalization, robustness and practicability, and finally, the results of the four evaluation indexes are integrated to obtain the evaluation score of the deep forgery detection model, so that an accurate, fair and comprehensive evaluation reference is established, and the obtained evaluation result is more in line with the actual situation of the deep forgery detection model.
Drawings
FIG. 1 is a flow chart of an evaluation method of a deep forgery detection model according to the present invention;
FIG. 2 is a schematic diagram illustrating a process of generating a diversified hard sample set according to the present invention;
fig. 3 is a block diagram of a terminal device according to the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
The invention is described in further detail below with reference to the accompanying drawings:
referring to fig. 1, in an embodiment of the present invention, a method for evaluating a deep forgery detection model is provided, including the following steps.
S1: a training method for obtaining a deep forgery detection model to be evaluated.
Specifically, the depth forgery detection model generally includes an image depth forgery detection model, a video depth forgery detection model, and an audio depth forgery detection model. For each deep forgery detection model to be evaluated, the training method during evaluation, including the training process and the setting of parameters, strictly refers to the description in the detection method literature.
S2: according to the training method of the depth forgery detection model to be evaluated, the depth forgery detection model to be evaluated is respectively trained through preset depth forgery data sets of various types, and the trained depth forgery detection model is obtained.
Specifically, in most of the existing detection methods for deep forgery detection models, when different depth forgery detection models are evaluated, the depth forgery detection models are obtained by training different training depth forgery data sets, but due to the characteristics of poor mobility and poor generalization among sample data of different forgery types, the comparison among the depth forgery detection models involved in the inconsistent evaluation work of the depth forgery data sets is unfairly and incorrectly. Based on this problem, in this embodiment, under the strategy of the recurrent training method, training is performed on preset depth forgery data sets of various types to obtain a trained depth forgery detection model on the depth forgery data sets, so as to ensure the fairness of evaluation in and among the depth forgery data sets of various types in the later period.
Preferably, the preset depth forgery data sets include at least two of a depth forgery data set for generating false data based on a GAN generation method, a depth forgery data set for generating false data based on a self-encoder generation method, a depth forgery data set for generating false data based on a graphical generation method, and a depth forgery data set for generating false data based on a GAN generation method, a depth forgery data set for generating false data based on a self-encoder generation method, and a depth forgery data set for generating false data based on a graphical generation method.
Specifically, in the existing evaluation method of the deep forgery detection model, the deep forgery data set is a common mainstream and is disclosed, the deep forgery data set is classified according to the principle difference of the generation method applied when the false sample data contained in the deep forgery data set is generated, and the generation method of the false sample data in the collected deep forgery data set can cover the GAN generation method, the self-encoder generation method and the graphical generation method.
Preferably, the preset deep forgery data sets of various types all include a training set, a verification set and a test set; and the proportions among the training set, the verification set and the test set are the same in each type of deep forgery data set.
Specifically, when a training set, a verification set and a test set are divided for each type of collected depth forgery data set, a certain type of depth forgery data set is selected as a reference depth forgery data set, sample data volumes of the training set, the verification set and the test set of the reference depth forgery data set are determined, and then sample data volumes of the training set, the verification set and the test set to be divided are determined for other depth forgery data sets according to a multiple relation of the total sample data volume between the other depth forgery data sets and the reference depth forgery data set.
Preferably, before the depth forgery detection model to be evaluated is trained respectively through the preset depth forgery data sets of various types, the sample data in the depth forgery data sets is subjected to video frame extraction, image face extraction or face correction processing; or, a sample data preprocessing method for the deep forgery detection model to be evaluated is obtained, and the sample data in the deep forgery data set is preprocessed according to the sample data preprocessing method.
Specifically, because the depth forgery detection model usually uses a single frame of face image or multiple frames of face images as the model input, the data preprocessing process is usually essential for the training and testing of the depth forgery detection model, and the data preprocessing operations in this embodiment include operations such as video frame extraction, image face extraction, and face correction that are common to the methods, and also include data preprocessing operations specific to each depth forgery detection model, such as face feature point extraction and additional monitoring information generation.
S3: and testing each trained deep forgery detection model through a preset diversified difficult sample set and various types of deep forgery data sets, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value, a robustness index value and a practicability index value under the non-same distribution data.
The diversification difficult sample set comprises standard sample data and disturbance sample data; the standard sample data comprises automatic standard sample data and manual standard sample data, and the automatic standard sample data is obtained in the following mode: predicting the false score value of each sample data in each type of deep forgery data set through a trained deep forgery detection model, and taking the sample data with the false score value smaller than a preset false score threshold value as automatic standard sample data; the manual standard sample data is obtained by the following method: observing the truth of each sample data in each type of deep forgery data set through a preset number of users, and taking more than half of sample data judged by the users as artificial standard sample data; the perturbation sample data is obtained by the following method: and adding a preset type of disturbance to the standard sample data to obtain disturbance sample data.
Specifically, in order to simulate the threat of deep forgery data in a real scene, a diversified and difficult sample set is designed. Referring to fig. 2, the diversified difficult sample set includes standard sample data and perturbation sample data, wherein the standard sample data is from the deep forgery data set, and the false digital content category can cover the full category of the deep forgery data sets. The false digital content in the standard sample data is obtained through automatic model screening and manual screening, in the automatic model screening process, a sample false score threshold value is set firstly, then automatic model screening of the false sample data is carried out by utilizing a trained model, the sample data of which the model prediction false score is smaller than the set threshold value is selected and added into a diversification difficult sample set, and an initial diversification difficult sample set is obtained.
In the manual screening process, the user test is carried out on the false sample data in the initial diversified sample set, the user needs to observe the sample and predict the truth of the sample data under the condition that the correct label of the sample data is unknown in the test process, the user collects the results after the test is finished, only more than half of the sample data which is judged to be wrong by the user is added into the diversified sample set, then part of the real sample data is selected from the real sample data of each deep counterfeit data set and added into the diversified sample set, the consistency of the false sample data and the real sample data is ensured, the class balance of the data is ensured, and the standard data of the diversified sample set is obtained. And then adding a plurality of preset types of disturbance to the standard data to generate disturbance data with single disturbance and mixed disturbance. The preset type of disturbance comprises one or more of Gaussian blur, white Gaussian noise, color contrast change and color saturation change.
S4: and carrying out weighted superposition on the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
Since the widely used evaluation indexes (including AUC (area under ROC curve) and accuracy) cannot fully reflect the performance of the detection method, in previous studies, evaluation indexes related to time and space complexity are not used, which may result in that the actual efficiency of the detection method with excellent performance may be low in actual scenes including large-scale and diversified forged videos and pictures. Therefore, designing a correct and reasonable evaluation flow and formulating a comprehensive and practical evaluation index have important significance for understanding the advantages and limitations of the existing mainstream deep forgery detection method. In the embodiment, four evaluation measurement indexes are provided, and the deep forgery detection model is comprehensively and comprehensively evaluated from the aspects of accuracy, generalization, robustness, practicability and the like of the deep forgery detection model, and the results of the four evaluation indexes are finally synthesized to obtain the benchmark evaluation score of the method, so that the accurate, fair and comprehensive deep forgery detection model evaluation benchmark is established.
The uniformly distributed data specifically refers to data that is consistent with a deep forgery generation method of the false sample data included in the trained deep forgery data set, and the non-uniformly distributed data refers to data that is inconsistent with the deep forgery generation method of the false sample data included in the trained deep forgery data set. The accuracy index values include AUC (size of area under ROC curve), accuracy and precision; the generalization index value comprises AUC, accuracy and precision; the robustness index value comprises an area under the curve value of the disturbance degree-AUC curve; the practical index value comprises the ratio of a longitudinal axis to a transverse axis of the model parameter-AUC scatter diagram, the ratio of a longitudinal axis to a transverse axis of the model required calculation force-AUC scatter diagram and the ratio of a longitudinal axis to a transverse axis of the model inference time-AUC scatter diagram.
Specifically, in this embodiment, a comprehensive and fair deep forgery detection method metric index is formulated, and the purpose is to quantitatively evaluate the accuracy, generalization, robustness, and practicability of the method. During specific evaluation, the trained deep forgery detection model is used for evaluating and obtaining accuracy measurement index values corresponding to the trained deep forgery detection model on the same distribution data, namely the test set of each deep forgery data set, the non-same distribution data, namely the standard data of the diversified difficult sample set is evaluated and obtained to obtain generalization measurement index values corresponding to the trained deep forgery detection model, the disturbance data of the diversified difficult sample set is evaluated and obtained to obtain robustness measurement index values corresponding to the trained deep forgery detection model, and the standard data of the diversified difficult sample set is evaluated and obtained to obtain practical measurement index values corresponding to the trained deep forgery detection model. And then, the four measurement indexes are integrated, a comprehensive evaluation measurement index is formulated, the final evaluation benchmark score of each trained deep forgery detection model is calculated, and the quality of each trained deep forgery detection model is evaluated according to the final evaluation benchmark score. Specifically, in this embodiment, the weight of each index is assigned to 1, so as to obtain a final comprehensive evaluation index result.
Subsequently, according to the evaluation result of each depth forgery detection model, namely the index value after weighted superposition, the depth forgery detection model with the best evaluation result can be selected, the image depth forgery detection of each image to be detected is carried out, and then a more accurate detection result is obtained.
In summary, the evaluation method of the deep forgery detection model of the present invention completely reproduces the training process of the deep forgery detection model to be evaluated by the training method of obtaining the deep forgery detection model to be evaluated, and then performs training on the preset deep forgery data sets of various types to obtain the training models of the deep forgery data sets of various types, so as to ensure the fairness of evaluation in and among the deep forgery data sets of various types at a later stage. Then, based on a preset diversified sample set, the test set contains a forged video with extremely high deception degree for both human eyes and detection algorithm, the videos are generated by various classical depth forgery generation methods, have the characteristics of diversification and strong challenge, can comprehensively evaluate a depth forgery detection model to be evaluated, and provide four evaluation indexes, namely, an accuracy index value under the same distribution data, a generalization index value, a robustness index value and a practicality index value under the non-same distribution data, aiming at comprehensively and comprehensively evaluating a deep forgery detection model from the aspects of accuracy, generalization, robustness and practicability, and finally, the results of the four evaluation indexes are integrated to obtain the evaluation score of the deep forgery detection model, so that an accurate, fair and comprehensive evaluation reference is established, and the obtained evaluation result is more in line with the actual situation of the deep forgery detection model.
The following are embodiments of the apparatus of the present invention that may be used to perform embodiments of the method of the present invention. For details of non-careless mistakes in the embodiment of the apparatus, please refer to the embodiment of the method of the present invention.
In another embodiment of the present invention, an evaluation system of a deep forgery detection model is provided, which can be used to implement the above evaluation method of a deep forgery detection model.
The acquisition module is used for acquiring a training method of the to-be-evaluated deep forgery detection model; the training module is used for respectively training the depth forgery detection model to be evaluated according to the training method of the depth forgery detection model to be evaluated and through preset depth forgery data sets of various types to obtain various trained depth forgery detection models; the testing module is used for testing each trained deep forgery detection model through a preset testing set and a diversified difficult sample set of each type of deep forgery data set, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value and a robustness index value under the different distribution data; the evaluation module is used for weighting and superposing the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
Referring to fig. 3, in yet another embodiment, the invention provides a terminal device, which may be a computer device, including a processor, an input device, an output device, and a computer-readable storage medium. The processor, input device, output device, and computer-readable storage medium may be connected by a bus or other means.
The computer-readable storage medium is for storing a computer program comprising program instructions, and the processor is for executing the program instructions stored by the computer-readable storage medium. The Processor may be a Central Processing Unit (CPU), or may be other general-purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable gate array (FPGA) or other Programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc., which is a computing core and a control core of the terminal, and is specifically adapted to implement one or more instructions, and specifically adapted to load and execute one or more instructions in a computer-readable storage medium, so as to implement a corresponding method flow or a corresponding function, and the Processor may be used for the operation of the evaluation method of the depth falsification detection model.
In yet another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a Memory device in a computer device and is used for storing programs and data. It is understood that the computer readable storage medium herein can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space storing an operating system of the terminal. Also, one or more instructions, which may be one or more computer programs (including program code), are stored in the memory space and are adapted to be loaded and executed by the processor. It should be noted that the computer-readable storage medium may be a high-speed RAM memory, or may be a non-volatile memory (non-volatile memory), such as at least one disk memory. One or more instructions stored in the computer-readable storage medium may be loaded and executed by a processor to implement the corresponding steps of the method for evaluating a deep forgery detection model in the above embodiments.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. An evaluating method of a deep forgery detection model is characterized by comprising the following steps:
obtaining a training method of a to-be-evaluated deep forgery detection model;
respectively training the depth forgery detection models to be evaluated through preset depth forgery data sets of various types according to a training method of the depth forgery detection models to be evaluated to obtain the trained depth forgery detection models;
testing each trained deep forgery detection model through a preset diversified difficult sample set and various types of deep forgery data sets, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value and a robustness index value under the non-same distribution data;
and carrying out weighted superposition on the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
2. An evaluating method for a deep forgery detection model according to claim 1, wherein the types of the deep forgery data sets include at least two of a deep forgery data set that generates dummy data based on a GAN generation method, a deep forgery data set that generates dummy data based on a self-coder generation method, a deep forgery data set that generates dummy data based on a graphical generation method, and a deep forgery data set that generates dummy data based on a GAN generation method, generates dummy data based on a self-coder generation method, and generates dummy data based on a graphical generation method.
3. The method for evaluating the deep forgery detection model according to claim 1, wherein the preset deep forgery data sets of each type include a training set, a verification set, and a test set; and the proportions among the training set, the verification set and the test set are the same in each type of deep forgery data set.
4. The method for evaluating the depth forgery detection model according to claim 1, wherein before the depth forgery detection model to be evaluated is trained respectively through the preset depth forgery data sets of various types, the sample data in the depth forgery data sets is subjected to video frame extraction, image face extraction or face correction processing;
or, a sample data preprocessing method for the deep forgery detection model to be evaluated is obtained, and the sample data in the deep forgery data set is preprocessed according to the sample data preprocessing method.
5. The method for evaluating the deep forgery detection model according to claim 1, wherein the diversified difficult sample set includes standard sample data and disturbance sample data;
the standard sample data comprises automatic standard sample data and manual standard sample data, and the automatic standard sample data is obtained in the following mode: predicting the false score value of each sample data in each type of deep forgery data set through a trained deep forgery detection model, and taking the sample data with the false score value smaller than a preset false score threshold value as automatic standard sample data; the manual standard sample data is obtained by the following method: observing the truth of each sample data in each type of deep forgery data set through a preset number of users, and taking more than half of sample data judged by the users as artificial standard sample data; the perturbation sample data is obtained by the following method: and adding a preset type of disturbance to the standard sample data to obtain disturbance sample data.
6. An evaluating method for a deep forgery detection model according to claim 5, characterized in that the preset type of disturbance includes one or several of gaussian blur, white gaussian noise, color contrast change and color saturation change.
7. The method for evaluating the deep forgery detection model according to claim 5, further comprising selecting real sample data from each type of deep forgery data set as standard sample data until the number of false sample data in all standard sample data is the same as the number of real sample data.
8. The method for evaluating a deep forgery detection model according to claim 1, wherein the accuracy index value includes AUC, accuracy rate, and precision rate; the generalization index value comprises AUC, accuracy and precision; the robustness index value comprises an area under the curve value of the disturbance degree-AUC curve; the practical index value comprises the ratio of a longitudinal axis to a transverse axis of the model parameter-AUC scatter diagram, the ratio of a longitudinal axis to a transverse axis of the model required calculation force-AUC scatter diagram and the ratio of a longitudinal axis to a transverse axis of the model inference time-AUC scatter diagram.
9. An evaluation system for a deep forgery detection model, comprising:
the acquisition module is used for acquiring a training method of the depth forgery detection model to be evaluated;
the training module is used for respectively training the depth forgery detection model to be evaluated according to the training method of the depth forgery detection model to be evaluated and through preset depth forgery data sets of various types to obtain various trained depth forgery detection models;
the testing module is used for testing each trained deep forgery detection model through a preset diversified difficult sample set and various types of deep forgery data sets, and acquiring an accuracy index value, a generalization index value, a robustness index value and a practicability index value of the trained deep forgery detection model under the same distribution data, and a generalization index value and a robustness index value under the different distribution data;
and the evaluation module is used for weighting and superposing the accuracy index value of the trained deep forgery detection model under the same distribution data, the generalization index value, the robustness index value and the practicability index value under the non-same distribution data according to preset weight to obtain the evaluation result of the deep forgery detection model to be evaluated.
10. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for evaluating a deep forgery detection model according to any one of claims 1 to 8 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110963494.3A CN113822160B (en) | 2021-08-20 | 2021-08-20 | Evaluation method, system and equipment of depth counterfeiting detection model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110963494.3A CN113822160B (en) | 2021-08-20 | 2021-08-20 | Evaluation method, system and equipment of depth counterfeiting detection model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113822160A true CN113822160A (en) | 2021-12-21 |
| CN113822160B CN113822160B (en) | 2023-09-19 |
Family
ID=78923148
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110963494.3A Active CN113822160B (en) | 2021-08-20 | 2021-08-20 | Evaluation method, system and equipment of depth counterfeiting detection model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113822160B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115457374A (en) * | 2022-11-09 | 2022-12-09 | 之江实验室 | Deep pseudo-image detection model generalization evaluation method and device based on reasoning mode |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019218621A1 (en) * | 2018-05-18 | 2019-11-21 | 北京市商汤科技开发有限公司 | Detection method for living being, device, electronic apparatus, and storage medium |
| WO2020258667A1 (en) * | 2019-06-26 | 2020-12-30 | 平安科技（深圳）有限公司 | Image recognition method and apparatus, and non-volatile readable storage medium and computer device |
| CN112766189A (en) * | 2021-01-25 | 2021-05-07 | 北京有竹居网络技术有限公司 | Depth forgery detection method, device, storage medium, and electronic apparatus |
| CN112784790A (en) * | 2021-01-29 | 2021-05-11 | 厦门大学 | Generalization false face detection method based on meta-learning |
| CN113128619A (en) * | 2021-05-10 | 2021-07-16 | 北京瑞莱智慧科技有限公司 | Method for training detection model of counterfeit sample, method for identifying counterfeit sample, apparatus, medium, and device |
-
2021
- 2021-08-20 CN CN202110963494.3A patent/CN113822160B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019218621A1 (en) * | 2018-05-18 | 2019-11-21 | 北京市商汤科技开发有限公司 | Detection method for living being, device, electronic apparatus, and storage medium |
| WO2020258667A1 (en) * | 2019-06-26 | 2020-12-30 | 平安科技（深圳）有限公司 | Image recognition method and apparatus, and non-volatile readable storage medium and computer device |
| CN112766189A (en) * | 2021-01-25 | 2021-05-07 | 北京有竹居网络技术有限公司 | Depth forgery detection method, device, storage medium, and electronic apparatus |
| CN112784790A (en) * | 2021-01-29 | 2021-05-11 | 厦门大学 | Generalization false face detection method based on meta-learning |
| CN113128619A (en) * | 2021-05-10 | 2021-07-16 | 北京瑞莱智慧科技有限公司 | Method for training detection model of counterfeit sample, method for identifying counterfeit sample, apparatus, medium, and device |
Non-Patent Citations (1)
| Title |
|---|
| 梁瑞刚;吕培卓;赵月;陈鹏;邢豪;张颖君;韩冀中;赫然;赵险峰;李明;陈恺;: "视听觉深度伪造检测技术研究综述", 信息安全学报, no. 02 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115457374A (en) * | 2022-11-09 | 2022-12-09 | 之江实验室 | Deep pseudo-image detection model generalization evaluation method and device based on reasoning mode |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113822160B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111126622B (en) | Data anomaly detection method and device | |
| CN111275491B (en) | Data processing method and device | |
| CN107545245A (en) | A kind of age estimation method and equipment | |
| CN108509976A (en) | The identification device and method of animal | |
| CN110705996B (en) | User behavior identification method, system and device based on feature mask | |
| CN108717663A (en) | Face label fraud judgment method, device, equipment and medium based on micro- expression | |
| CN106650670A (en) | Method and device for detection of living body face video | |
| CN110544168A (en) | weak supervision internet financial anti-fraud method based on generation countermeasure network | |
| CN111881706B (en) | Living body detection, image classification and model training method, device, equipment and medium | |
| CN108198072A (en) | A kind of system of artificial intelligence assessment financial product feature | |
| CN116843196A (en) | Intelligent training method and system applied to military training | |
| CN113822160B (en) | Evaluation method, system and equipment of depth counterfeiting detection model | |
| CN108549654A (en) | A kind of big data analysis method based on image procossing | |
| CN111310918A (en) | Data processing method and device, computer equipment and storage medium | |
| CN111737438B (en) | Data processing method and device based on text similarity and electronic equipment | |
| CN108510483A (en) | A kind of calculating using VLAD codings and SVM generates color image tamper detection method | |
| CN109101883A (en) | A kind of Depression trend evaluating apparatus and system | |
| CN112084936A (en) | Face image preprocessing method, device, equipment and storage medium | |
| CN113673811B (en) | On-line learning performance evaluation method and device based on session | |
| CN113807541B (en) | Fairness repair method, system, equipment and storage medium for decision system | |
| CN116956005A (en) | Training method, device, equipment, storage medium and product of data analysis model | |
| CN115457374A (en) | Deep pseudo-image detection model generalization evaluation method and device based on reasoning mode | |
| CN115510969A (en) | Public health data reliability assessment system based on machine learning | |
| CN112599246B (en) | Vital sign data processing method, system, device and computer readable medium | |
| CN114529794A (en) | Infrared and visible light image fusion method, system and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |