CN115482595B - Specific character visual sense counterfeiting detection and identification method based on semantic segmentation - Google Patents

Abstract

The invention discloses a specific character visual sense counterfeiting detection and identification method based on semantic segmentation, belongs to the technical field of depth counterfeiting and detection, and provides a counterfeiting video detection mode which takes depth counterfeiting video detection of specific characters as a research target, constructs a personal characteristic model of a target task based on a basic method of semi-supervision and semantic segmentation, selects and classifies attribute masks of constructed human face regions, and outputs results by integrating classification weights of various attributes. Firstly, constructing a mask data set of a target character region based on semantic segmentation; and secondly, establishing a personal semantic model for visual forgery detection and identification to detect a deep forged video. In the process of manufacturing the data set, the data set is amplified by using a semi-supervised machine learning algorithm, so that the problem of insufficient data set of the specific person is solved, and the manual labeling cost is reduced.

Description

A Person-specific Visual Forgery Detection and Identification Method Based on Semantic Segmentation

technical field

The invention belongs to the technical field of deep counterfeiting and detection, and relates to a counterfeit video detection method, specifically a semantic segmentation-based specific person visual forgery detection and identification method.

Background technique

Deep fake is translated from the word "DeepFake", which is a combination of "Deep learning" and "Fake", that is, the combination of deep learning and forgery. Deepfake is a deep learning-based technology that refers to the creation of fake videos and images by swapping people's faces. The term DeepFake stems from a machine learning algorithm released by Reddit user deepfakes in 2017, which he claimed could help him convert celebrity faces into pornographic videos. Once the algorithm was released, it was hotly discussed by the public and the media, followed by an upsurge in research on visual deepfake algorithms. In 2018, BuzzFeed released a deepfake video of Barack Obama giving a speech, created using FakeApp software created by a Reddit user. From 2017 to 2020, the number of papers related to deep forgery has increased from the original 3 to more than 250. At the same time, FakeApp, Faceswap, Zao, FaceApp and other fast deep forgery software for the public that can achieve no technical cost have also been successively published. Various categories of fake videos produced by visual deepfakes have also raised concerns about identity theft, impersonation, and the spread of disinformation on social media.

Existing visual deep forgery methods can be roughly divided into three types: synthetic new faces, face modification and face swapping. Among them, synthesizing a new face refers to using GAN to create a face image that does not exist; face modification refers to modifying some parts of the original existing face; face swapping refers to partially or Whole exchange.

The method of synthesizing new faces uses the powerful generation confrontation network GAN to completely create the entire face image that does not exist. At present, the databases for synthesizing new faces are based on the ProGAN and StyleGAN architectures, and each fake created Each image will carry its specific GAN fingerprint. The facial modification method is mainly to add some facial modifications to the target face, such as changing the hair color or skin color, modifying the gender of the target person, adding glasses to the target person, etc. This method also needs to be based on the generation confrontation network GAN, the latest StarGAN technology Faces can be divided into multiple domains and retouching operations can be performed on them at the same time. The face swap method consists of two parts. The first is to use another person’s face to replace the face of the target person in the video. This is currently the most popular method in the direction of visual deep forgery, such as DeepFakes and FaceSwap. This method, different from the first two methods that put the facial synthesis operation on the image, this method can be used for the synthesis of deep fake videos; the second method is facial expression exchange, also known as facial reproduction, that is, another The facial expression on the human face is replaced by the facial expression of the target person, such as by changing Obama's expression and movements to make it complete a fake "speech".

The visual deep forgery detection technology is mainly carried out by the steps of feature extraction, model building, detection and classification. First, researchers preprocess the image or video data to be detected, and determine the features to be detected based on prior knowledge or image processing methods. Then, the corresponding algorithm is designed to extract certain features, and a network model matching the detection task is established. Finally, the performance of the detection algorithm is tested by using the data to be detected, so as to verify the scientificity of the selected features and the validity of the classification model. Among them, the key to determining the detection performance is how to select relevant features that can effectively distinguish between real and fake faces, and how to establish a model with good classification effect.

Different deep forgery detection methods have different emphases in the detection algorithm process, so the detection methods can be classified as follows:

The visual deepfake detection technology based on specific artifacts focuses on the feature determination part in the detection flow chart. From the perspective of image processing, it captures the abnormal phenomena such as blur, jitter, and ghosting in the generated image or video with pixel-level granularity. The degree of discrimination of artifact features directly affects the performance of detection algorithms.

The data-driven visual deep forgery detection technology focuses on the model building part of the detection flow chart, and uses a well-designed neural network to train and classify the time domain and frequency domain information in the extracted fakes. Excellent network design can extract potential subtle features more effectively.

The visual deep forgery detection technology based on information inconsistency focuses on capturing the inconsistency between forgery and objective laws from the high-level semantics such as biological inherent characteristics, time continuity, and motion vectors. Because the extraction process of advanced semantic features is relatively complex, this technology focuses on the two parts of feature determination and feature extraction in the detection flow chart.

Since a specific person has a large amount of available real face data, according to the real face, a large amount of training can be performed using the generative confrontation network GAN to create a very realistic deep fake face. Deep counterfeit products are easy to cause adverse effects, and the current general-field counterfeit detection methods are not good enough to identify counterfeit products of specific people. Therefore, it is necessary to conduct research on deep counterfeit detection for specific people.

Contents of the invention

In view of the above problems, the present invention proposes a method for visual forgery detection and identification of a specific person based on semantic segmentation, which effectively improves the ability of forgery detection and identification.

The visual forgery detection and identification method of a specific person based on semantic segmentation in the present invention is divided into a semantic segmentation part and a forgery detection and identification part.

The semantic segmentation part performs semantic segmentation on the deep fake face: the image of the target person is marked according to eleven features of the face to form an initial training set; the initial training set is used to generate a mask of the target person using a semi-supervised semantic segmentation model. Membrane dataset.

The forgery detection and identification part performs dot product according to the mask data of the target person after semantic segmentation and the face picture corresponding to the mask data to obtain the designated image attribute area, and further model the obtained area attribute, specifically as :

For the original face image z of each target person, the image mask a separated by the facial features in the mask dataset is combined with the manually selected facial features region of interest vector V to obtain the region of interest of the facial features, and then combine it with The corresponding original face image is dot-multiplied to generate the conditional tensor T of the required face region of interest.

Dot product the input picture z and the tensor T corresponding to the picture z, and input the dot product processing result p(z) into the generative confrontation network for pose-independent recognition processing. The formula for dot product processing the tensor T of the above selected region of interest with the original image z is as follows:

p(z)=z·T=z·a·V

Input p(z) and the given pose to the generator G; the generator G uses the given pose to generate corresponding fake pictures, and uses the discriminator D to judge the pose and identity of the generated fake pictures, and continuously conduct confrontation training , until it reaches the critical state where the discriminator D believes that the fake picture generated by the generator G has the same identity as the original input picture, and a pose-independent face picture is obtained.

After the pose-independent recognition process, each face segmentation attribute area of the face image x that has undergone pose change is input into a single convolutional neural network for classification processing, and a new CNN binary classification classifier is constructed. Among them, the image features are learned through the convolutional network, the output dimension is reduced through the pooling layer, and the depth features are fused through the fully connected layer, and finally the classification result is formed and output to achieve the purpose of identifying whether the input image is a positive sample or a negative sample.

The advantages of the present invention are:

1. The method for detecting and discriminating visual forgery of a specific person based on semantic segmentation in the present invention constructs a face mask data set based on a specific person, which can expand the data of face forgery without increasing the cost of manual labeling.

2. The semantic segmentation-based visual forgery detection and identification method for a specific person in the present invention can effectively use the attention mechanism, thereby greatly improving the detection accuracy of the classifier for forgery samples.

3. The present invention is based on the semantic segmentation-based visual forgery detection and identification method for a specific person, constructs a pose-independent module, realizes the detection of input pictures of various poses, and increases the robustness of forgery detection.

4. The method for visual forgery detection and identification of a specific person based on semantic segmentation of the present invention can detect pictures and videos generated by various fake face forgery technologies, increasing the generalization of forgery detection.

Description of drawings

Fig. 1 is a flow chart of the method for detecting and discriminating visual forgery of a specific person based on semantic segmentation in the present invention;

Figure 2 shows the semantic segmentation network structure.

Detailed ways

The present invention will be described in further detail below in conjunction with the accompanying drawings.

The method for detecting and discriminating visual forgery of a specific person based on semantic segmentation in the present invention, as shown in Figure 1, the specific steps are:

Step 1: Based on the semantic segmentation method, generate a mask dataset for facial feature segmentation of a specific person.

101. Collect a large amount of data on the target person to form an initial training set.

The data set needs to be collected and includes the following four parts: the real face of the target person, the real face of other people who are not the target person, the deep fake face of the target person, and the imitators and actors of the target person. In order to build the basic resources for model training.

For the consideration of the subsequent processing of video data, the real face of the target person can be selected from the public video website with a high-definition video with the front of the target person facing the camera as much as possible, and download it. Select the real face of the target person as the target collection object, and collect about 60 hours of video to ensure that the model can effectively learn the real face features of the target person. For the imitators and actors of the target person, as well as the real face of the non-target person, they are collected by referring to the real face collection method of the target person; while the deep fake face of the target person is collected through the real face of the target person respectively. FaceSwap, Wav2lip and First Order Motion are produced by three forgery methods. As for the real face of the above-mentioned non-target person, a face whose facial features are as similar as possible to the target person can be selected to enhance the model's ability to identify similar faces in real scenes.

Select the target person as the target object of protection, then the video data set of the real face of the target person is used as a positive sample, and the imitator and actor of the target person, the deep fake face of the target person, and the counterfeit product data of the non-target person set as a negative sample.

102. Perform facial semantic segmentation on the collected real face datasets and counterfeit product datasets of the target person.

For the faces of all positive samples and negative samples, select N pictures of randomly selected frames in the video, and use the LabelMe labeling tool to manually label them, as shown in Figure 1. During the manual labeling process, the areas where 11 parts are located need to be identified. Mark, respectively: left eye, right eye, left eyebrow, right eyebrow, nose, upper lip, lower lip, hair, left ear, right ear and neck. Afterwards, the json format file generated by the annotation is processed together with the original image of the annotation to obtain a mask dataset of different facial category labels.

103. Use the pictures in the mask dataset manually marked in the previous step to train the machine labeling method by constructing a semantic segmentation network Deeplabv3+ for semi-supervised machine learning.

Select the semantic segmentation network Deeplabv3+ for training, input the N mask dataset pictures generated manually in step 102, and use the deep learning model to machine the remaining M unmarked pictures in the positive and negative samples collected in step 1. Automatic labeling, to achieve semi-supervised machine learning labeling, the method is as follows:

A. As shown in Figure 2, two semantic segmentation networks P ₁ and P ₂ with the same structure and different weight initial values are constructed:

P ₁ = f(X; θ ₁ )

P ₂ = f(X; θ ₂ )

P ₁ and P ₂ are two semantic segmentation networks Deeplabv3+, and the two are only different in the initial value of the weight parameter.

Among them, X represents the input image after data enhancement of N marked images; θ ₁ and θ ₂ represent the weights of P ₁ and P ₂ networks respectively; Y represents the pseudo-label obtained by two semantic segmentation networks, that is, the preliminary Split results. Among them, the two semantic segmentation networks P ₁ and P ₂ use Deeplabv3+, and the classifiers Y1 and Y2 use the ResNet101 network. Compared with other semantic segmentation network structures, Deeplabv3+ can enable the upper layers of the network to encode context information of different scales by convolution or pooling of input features, and at the same time enable the lower layers of the network to gradually recover Spatial information captures clear object boundaries and is suitable for semantic segmentation of faces. For the two segmentation networks, the corresponding one-hot labels Y ₁ and Y ₂ are obtained through the argmax operation. Then these two pseudo-labels are used as supervision signals, _Y2 is used as the supervision of _P1 , _Y1 is used as the supervision of _P2 , and constrained by the cross-entropy loss function to improve the performance of the semantic segmentation network.

Finally, the M images labeled by the semantic segmentation network machine will be combined with the original manually labeled N images to form the Mask mask dataset.

Step 2: Build a personal semantic model for visual forgery detection and identification

The present invention constructs a method for visual forgery detection and identification of a specific person, which is divided into data preprocessing, posture-independent recognition, and model construction according to the process; the details are as follows:

201: Preprocessing of original target face data

The original face picture of the target person collected in step 101 is marked as z ∈ Z, where Z is all the face data collected in step 101, which is a collection of faces including positive samples and negative samples. For each face Figure 1 shows the image mask a of the facial features separated from the Mask mask data set obtained in step 1 and the manually selected facial features area vector V (such as the nose, eyes, and ears represent a vector, and the selected nose is the vector V as [ 1, 0, 0]) to obtain the region of interest of facial features, and then dot-multiply it with the corresponding original face image to generate the conditional tensor T of the required facial region of interest.

Dot product the input picture z and the tensor T corresponding to the picture z, and input the dot product processing result p(z) into the generative confrontation network for pose-independent recognition processing. The formula for dot product processing the tensor T of the above selected region of interest with the original image z is as follows:

p(z)=z·T=z·a·V

202: Attitude-independent recognition

Each picture z is multiplied by the tensor T point corresponding to the picture z and the result p(z) and the given attitude are input to the generator G; in the present invention, the attitude is set as the direction in which the front of the face faces the camera, indicating that the face and the front The front angle is 0°. The generator G uses the given pose to generate the corresponding fake picture, uses the discriminator D to judge the pose and identity of the generated fake picture, and continuously conducts confrontation training until the discriminator D believes that the fake picture generated by the generator G is the same as the original When the critical state of the same identity of the input image is obtained, the pose-independent face image is obtained.

203: Deep forgery detection model construction

After the pose-independent recognition process, each face segmentation attribute area of the face image x that has undergone pose change is input into a single convolutional neural network for classification processing, and a new CNN binary classification classifier is constructed. Among them, the image features are learned through the convolutional network, and the output dimension is reduced through the pooling layer. The deep features are fused through the fully connected layer, and finally the classification result is formed and output to achieve the purpose of identifying whether the input picture is a positive sample or a negative sample.

Among them, in the design of the loss function, the binary cross entropy (BCE) loss function is used to measure the classification loss, and the formal definition of _LY is as follows:

L _Y (x, y) = BCE (p, y) = -(y*log(p))+(1-y)*log(1-p)

Among them, p is the predicted classification output of the classifier, y ∈ {0, 1} is the true and false label, and the sigmoid activation function is used to process the output in the binary classification task.

In summary, the present invention is based on semantic segmentation-based specific person visual forgery detection and identification method, establishes a mask training data set for facial image segmentation of a specific person, uses a semi-supervised machine learning algorithm to amplify the data set, and solves the problem of insufficient specific person data sets. problem and reduce the cost of manual labeling; add semantic segmentation module and attitude change module before the binary classification detection model, adopt different feature modeling and fitting methods, improve the ability of forgery detection and identification, and can improve the accuracy of the binary classification detection model5 -10 percentage points.

Claims (3)

1. A specific person visual forgery detection and identification method based on semantic segmentation, characterized in that: it is divided into a semantic segmentation part and a forgery detection and identification part; The semantic segmentation part performs semantic segmentation on the deep fake face: the image of the target person is marked according to eleven features of the face to form an initial training set; the initial training set is used to generate a mask of the target person using a semi-supervised semantic segmentation model. Membrane dataset; The forgery detection and identification part performs dot product according to the mask data of the target person after semantic segmentation and the face picture corresponding to the mask data to obtain the designated image attribute area, and further model the obtained area attribute, specifically as : For the original face image z of each target person, the image mask a separated by the facial features in the mask dataset is combined with the manually selected facial features region of interest vector V to obtain the region of interest of the facial features, and then combine it with The corresponding original face image is dot-multiplied to generate the conditional tensor T of the required facial region of interest; Dot multiplication of the input picture z and the conditional tensor T corresponding to the picture z, and input the dot product processing result p(z) into the generation confrontation network for attitude-independent recognition processing; the above-mentioned required conditions for the facial region of interest The formula for dot product processing tensor T and original image z is as follows: p(z)=z·T=z·a·V Input p(z) and the given pose to the generator G; the generator G uses the given pose to generate corresponding fake pictures, and uses the discriminator D to judge the pose and identity of the generated fake pictures, and continuously conduct confrontation training , until it reaches the critical state where the discriminator D believes that the fake picture generated by the generator G has the same identity as the original input picture, and a pose-independent face picture is obtained; After the pose-independent recognition process, input the face segmentation attribute regions of the face image x that has undergone pose changes into a single convolutional neural network for classification processing, and construct a new CNN binary classification classifier; where, by The convolutional network learns image features, reduces the output dimension through the pooling layer, and fuses the deep features through the fully connected layer, and finally forms a classification result and outputs it to achieve the purpose of identifying whether the input image is a positive sample or a negative sample. 2. A kind of specific character visual forgery detection and discrimination method based on semantic segmentation as claimed in claim 1, is characterized in that: semantic segmentation method is: Manually annotate N pictures of randomly selected frames in the selected face image video. During the manual annotation process, the areas where 11 parts are located need to be marked; after that, the json format file generated by the annotation will be processed together with the original image of the annotated image. , to get mask datasets of different face category labels; Select the semantic segmentation network Deeplabv3+ for training, input N mask dataset pictures generated by manual labeling, and use the deep learning model to automatically label the remaining M unlabeled pictures in the face image to realize semi-supervised machine learning label, specifically: Construct two semantic segmentation networks P ₁ and P ₂ with the same structure and different weight initial values: P ₁ = f(X; θ ₁ ) P ₂ = f(X; θ ₂ ) Among them, X represents the input image after data enhancement for N marked images; θ ₁ and θ ₂ represent the weights of P ₁ and P ₂ networks respectively; Y represents the pseudo-label obtained by two semantic segmentation networks; A segmentation network, the corresponding one-hot labels Y ₁ and Y ₂ are obtained through the argmax operation; then these two pseudo-labels are used as supervision signals, Y ₂ is used as the supervision of P ₁ , Y ₁ is used as the supervision of P ₂ , and crossover Entropy loss function constraints; finally, the M pictures generated and marked by the semantic segmentation network machine will be combined with the original manually marked N pictures to form the mask data set of the target person. 3. the forgery detection and discrimination method based on the specific person's vision of semantic segmentation as claimed in claim 1 is characterized in that: on the design of classifier loss function, measure its classification loss with binary cross entropy BCE loss function, L _Y The formal definition of is the following formula: L _Y (x,y)=BCE(p,y)=-(y*log(p))+(1-y)*log(1-p) Among them, x represents the input image, p is the predicted classification output of the classifier, y∈{0,1} is the true and false label, and the sigmoid activation function is used to process the output in the binary classification task.

Images