CN114360034A - Method, system and equipment for detecting deeply forged human face based on triplet network - Google Patents

Abstract

The invention discloses a method, a system and equipment for detecting a deep forged face based on a triplet network

Preprocessing into 299 x 3 size, inputting into the main feature extraction network of the three-cell network to obtain the depth feature of the face imageNet(I)，Net(I)Is a characteristic vector of 2048 dimensions; then using the classification network pairNet(I)And (3) classifying, outputting a 2-dimensional feature by a classification network for the input 2048-dimensional feature vector, converting the numerical value of the 2-dimensional feature into a probability through Softmax processing, and expressing the relative probability of authenticity of the picture, wherein the picture with the probability greater than a preset value is a forged face picture. The three-cell network can extract more effective true and false face identification characteristics and has more accurate deep fake face detection effect.

Description

Deep forgery face detection method, system and device based on triplets network

technical field

The invention belongs to the technical field of artificial intelligence security, and relates to a deep forgery face detection method, system and equipment, in particular to a deep forgery face detection method, system and equipment based on a triplets network.

Background technique

Deepfake is a technology to create or synthesize fake content (such as images, videos) based on intelligent methods such as deep learning. In recent years, with the development of deep learning technology, deepfakes are developing at an unprecedented speed. At present, deepfake technology can not only generate face-changing images and imitate the facial expressions of real people, but also create characters that do not exist in reality and are difficult to distinguish, subverting the traditional concept of "seeing is believing".

Once deepfake technology is abused, it will bring great harm to individuals, society and the country.

The best way to fight "deepfake" is "deepfake detection". The purpose of deepfake detection technology is to detect whether an image or video has been forged through deepfake technology. The current mainstream detection methods include detection methods using traditional image features and detection methods based on deep learning. With the development of deep learning technology, more and more novel deepfake detection techniques are applied. By building different convolutional neural network structures, the researchers extract the depth features in the image, and use the depth features to determine whether the face is deep forged. In order to improve the expressiveness of features, researchers continue to propose new network architectures, the mainstream ones including Xception network, residual network and DenseNet. Some researchers have also introduced frequency domain information in convolutional neural networks to increase the expressiveness of features.

However, although these convolutional neural network structures can extract the main features of the image well, these single-sample input networks are easy to pay attention to the feature expressions that are not related to the authenticity of the image, such as background features, skin color features, etc. And it is difficult to capture the intrinsic feature expression related to the authenticity attributes of images, especially when multiple images have similar appearance but different authenticity attributes, these networks can easily extract similar image features, thus affecting the accuracy of detection.

SUMMARY OF THE INVENTION

In order to solve the above technical problems, the present invention proposes a deep face forgery detection method, system and electronic device based on triplets network. The three coupled samples of the original image, the target image and the fake image are used to learn the input network, which enables the network to capture feature expressions with similar appearance but different true and false attributes.

The technical scheme adopted by the method of the present invention is: a deep forgery face detection method based on triplets network, comprising the following steps:

Step 1: Preprocess the fake face image I to be detected into a preset size and input it into the backbone feature extraction network of the triplets network to obtain the depth feature Net(I) of the face image; Net(I) is a 2048-dimensional feature vector;

Step 2: Use the classification network to classify Net(I) . For the input 2048-dimensional feature vector, the classification network outputs a 2-dimensional feature, and the value of the 2-dimensional feature is converted into a relative probability through Softmax processing, which is used to represent the authenticity of the picture. The relative probability of ; wherein, the picture with the probability greater than the preset value is a fake face picture;

The backbone feature extraction network of the triplet network adopts the skeleton of the Xception network, including the entry flow, the intermediate flow and the exit flow; the entry flow contains two 3×3 convolutions and uses the ReLU activation function to activate and three convolutions. block; the middle stream contains 8 convolution modules; the outlet stream contains a convolution block and two 3×3 depthwise separable convolutions, activated with the ReLU function, and finally performs an average pooling operation; three backbone features Extract the network share a weight;

The classification network adopts BP neural network, including three layers, the first layer is the input layer, including 2048 nodes; the middle layer includes 1024 nodes, and the output layer includes 2 nodes; ReLU activation function is used between each layer activation.

The technical scheme adopted by the system of the present invention is: a deep forgery face detection system based on triplet network, comprising the following modules:

Module 1 is used to preprocess the forged face image I to be detected into a preset size and be input into the backbone feature extraction network of the triplets network, obtain the depth feature Net(I) of the face image; Net(I) is a 2048-dimensional feature vector;

Module 2 is used to classify Net(I) by using the classification network. For the input 2048-dimensional feature vector, the classification network outputs a 2-dimensional feature, and the value of the 2-dimensional feature is converted into a relative probability through Softmax processing, which is used to represent the picture. Relative probability of authenticity; among them, pictures with a probability greater than the preset value are forged face pictures;

The backbone feature extraction network of the triplet network adopts the skeleton of the Xception network, including the entry flow, the intermediate flow and the exit flow; the entry flow contains two 3×3 convolutions and uses the ReLU activation function to activate and three convolutions. block; the middle stream contains 8 convolution modules; the outlet stream contains a convolution block and two 3×3 depthwise separable convolutions, which are activated using the ReLU function, and finally perform an average pooling operation; three backbone features Extract the network share a weight;

The classification network adopts BP neural network, including three layers, the first layer is the input layer, including 2048 nodes; the middle layer includes 1024 nodes, and the output layer includes 2 nodes; ReLU activation function is used between each layer activation.

The technical scheme adopted by the device of the present invention is: a deep forgery face detection device based on triplet network, including:

one or more processors;

A storage device for storing one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the triplets-based network The deepfake face detection method.

Advantages and positive effects of the present invention:

(1) The present invention uses three coupled samples of the original image, the target image and the fake image to learn the input network. Compared to a single-sample input network, using triples enables the network to capture feature representations that look alike but differ in their true and false attributes.

(2) The present invention realizes the identification of deep forged faces, and solves the security problems caused by forged faces in practical application scenarios.

Description of drawings

1 is a schematic diagram of a method according to an embodiment of the present invention;

2 is a schematic diagram of a triplets network constructed in an embodiment of the present invention;

3 is a schematic diagram of a classification network constructed by an example of the present invention;

FIG. 4 is an experimental result diagram of a network constructed by an example of the present invention.

Detailed ways

In order to facilitate the understanding and implementation of the present invention by those of ordinary skill in the art, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention, but not to limit it. this invention.

The existing convolutional neural network structure can only extract the depth features of the image, but these depth features do not pay attention to the feature expression related to the authenticity of the image. The original image, target image and fake image, three coupled samples, are sent to the triplets network for learning, which enables the triplets network to capture feature expressions with similar appearance but different true and false attributes.

See Fig. 1, a kind of deep forgery face detection method based on triplets network provided by the present invention comprises the following steps:

Step 1: Preprocess the fake face image I to be detected into a size of 299×299×3 and input it into the backbone feature extraction network of the triplets network to obtain the depth feature Net(I) of the face image; Net( I) is a 2048-dimensional feature vector;

Step 2: Use the classification network to classify Net(I) . For the input 2048-dimensional feature vector, the classification network outputs a 2-dimensional feature, and the value of the 2-dimensional feature is converted into a relative probability through Softmax processing, which is used to represent the authenticity of the picture. The relative probability of ; wherein, the picture with the probability greater than the preset value is a fake face picture.

Please refer to Figure 2. The backbone feature extraction network of this embodiment adopts the skeleton of the Xception network, including inlet flow, intermediate flow and outlet flow; the inlet flow is to convert a 299×299×3 picture into a 19×19× The feature map of 728 contains 2 convolutions of 3×3 and uses the ReLU activation function to activate and 3 convolution blocks; the intermediate stream contains 8 convolution modules; the outlet stream is a 19×19×728 feature map Converted into a 2048-dimensional feature vector, containing a convolution block and two 3×3 depthwise separable convolutions, activated using the ReLU function, and finally performing an average pooling operation; three backbone feature extraction networks share one Finally, through the backbone feature extraction network, a 299×299×3 image is converted into a 2048-dimensional feature vector.

Please refer to Figure 3, the classification network of this embodiment adopts BP neural network, including three layers, the first layer is the input layer, including 2048 nodes; the middle layer includes 1024 nodes, and the output layer includes 2 nodes; each layer Use the ReLU activation function to activate between them.

The backbone feature extraction network of this embodiment is a trained backbone feature extraction network; the training process includes the following substeps:

Step 1.1: Collect several original image-target image-fake image triples, denoted as ( original , target , fake );

In this embodiment, the forged face video, the original face video and the target face video are down-sampled first, and frame images starting from a specific frame and with a fixed frame interval are selected in the video, and the forged face video and the original person from which it is derived are kept. There is a one-to-one correspondence between the face video and the video frames obtained by downsampling the target face video. This embodiment selects a total of 10 frames of images starting from 0 seconds, one frame per second, and each video as the original data set;

In this embodiment, the RetinaFace face detection algorithm is used to detect the face region in the acquired frame image, and the face image is cropped. Extract 5 facial feature points, including left eye, right eye, nose, left corner of mouth and right corner of mouth; align the face through facial feature points so that the aligned face is in the center of the image; resize the image to 299 pixels long by opencv , 299 pixels wide, 3 channels, organize the processed face image into original face-target face-fake face triple data, denoted as ( original , target , fake ).

In this embodiment, the processed face image is organized into the original face-target face-forged face triple data obtained in step 1.1 as input and supervision samples respectively, and the backbone feature extraction network is continuously trained.

Step 1.2: For each group of triples ( original , target , fake ), the images in the triples are sequentially sent to the backbone feature extraction network Net( ) of the triplet network; the depth features of the images are obtained respectively, denoted as ( Net ( original ), Net ( target ), Net ( fake )); wherein, the weights of the backbone feature extraction network Net ( ) are shared;

Step 1.3: Calculate the feature distance Dis ( Net ( original ), Net ( target )) of the depth feature of the original image and the depth feature of the target image and the feature distance Dis ( Net ( target ), Net of the depth feature of the target image and the depth feature of the fake image ( fake )), where Dis ( a , b ) represents the characteristic distance between two eigenvectors a , b , and the calculation formula of the eigenvector distance is as follows:

；

;

代表L2距离； in

represents the L2 distance;

Step 1.4: Calculate the loss function loss of the triplets network according to the feature distance calculated in step 1.3. In order to drive the network model, the focus is not on the appearance-like features (even if the appearance is not the same, the depth features should be close), but the true and false attribute expression (Even if the appearance is similar, the depth feature should be far away), I hope that the distance between the two real image features Dis ( Net ( original ), Net ( target )) should be as small as possible, and the distance between the real and fake images is Dis ( Net ( target ) , Net ( fake )) should be as large as possible. Therefore, the loss function calculation formula is as follows:

loss = max ( Dis ( Net ( original ), Net ( target )) - Dis ( Net ( target ), Net ( fake )) + margin , 0);

Among them, margin is a hyperparameter, used to set the interval between two feature distances;

In this embodiment, margin = 0.2 is set, which means that no loss occurs when the feature distance between the original image and the target image is 0.2 smaller than the feature distance between the target image and the fake image.

Step 1.5: After calculating the loss, use the Adam optimizer to back-propagate and optimize the backbone feature extraction network;

Step 1.6: Repeat steps 1.1-1.5 to train the backbone feature extraction network to convergence, and obtain the trained backbone feature extraction network.

The classification network of the present embodiment is a trained classification network; its training process includes the following substeps:

Step 2.1: Preprocess the fake face image I to be detected into a size of 299×299×3 and input it into the backbone feature extraction network in the trained triplets network to obtain the depth feature of the face image Net(I) ; Net(I) is a 2048-dimensional feature vector;

Step 2.2: Use the classification network to classify Net(I) . For the input 2048-dimensional feature vector, the classification network outputs a 2-dimensional feature, and the value of the 2-dimensional feature is converted into a relative probability through Softmax processing, which is used to indicate the authenticity of the picture. the relative probability of ;

Step 2.3: Calculate the cross-entropy loss function loss between the predicted result and the actual result. The calculation method of loss is as follows:

；

;

where pi represents the probability that sample _i is true, yi represents the label of sample i , if sample _i is a fake image, then yi ₌ 0, otherwise yi = ₁ ;

Step 2.4: After calculating the cross entropy loss, use the SGD optimizer, backpropagation with gradient descent method and optimize the classification network;

Step 2.5: Repeat steps 2.1 and 2.4 until the classification network converges, and a trained classification network is obtained.

After completing the above steps, experiment with the network. The fake and real images are mixed and fed into the trained triplets network for feature extraction. The extracted feature vector is mapped on the two-dimensional plane rectangular coordinates, and the result is shown in Figure 4. It can be seen from Figure 4 that the features of fake faces are significantly different from those of real faces. The present invention can detect the authenticity of the face image, and the detection result has high reliability.

It should be understood that the above description of the preferred embodiments is relatively detailed, and therefore should not be considered as a limitation on the protection scope of the patent of the present invention. In the case of the protection scope, substitutions or deformations can also be made, which all fall within the protection scope of the present invention, and the claimed protection scope of the present invention shall be subject to the appended claims.

Claims (6)

1. A deep fake face detection method based on a triplet network is characterized by comprising the following steps:

step 1: counterfeit human face image to be detectedIPreprocessing the image into a preset size and inputting the preset size into a trunk feature extraction network of a three-cell network to obtain the depth feature of the face imageNet(I)；Net(I)Is a characteristic vector of 2048 dimensions;

step 2: using a classification network pairNet(I)Classifying, namely outputting a 2-dimensional feature by a classification network for the input 2048-dimensional feature vector, and converting the numerical value of the 2-dimensional feature into relative probability through Softmax processing to express the relative probability of authenticity of the picture; wherein, the picture with the probability larger than the preset value is a forged face picture;

the main feature extraction network of the three-cell network adopts a framework of an Xconvergence network, and comprises an inlet flow, a middle flow and an outlet flow; the ingress stream contains 2 convolutions of 3 × 3 and is activated using the ReLU activation function and 3 volume blocks; the intermediate stream contains 8 convolution modules; the exit stream comprises a volume block and two times of 3 x 3 depth separable convolutions, and is activated by using a ReLU function, and finally an average pooling operation is performed; three trunk characteristic extraction networks share one weight;

the classification network adopts a BP neural network and comprises three layers, wherein the first layer is an input layer and comprises 2048 nodes; the middle layer comprises 1024 nodes, and the output layer comprises 2 nodes; and a ReLU activation function is used between each layer for activation.

2. The method for detecting the deep forged face based on the triplet network as claimed in claim 1, wherein: the trunk feature extraction network of the three-cell network in the step 1 is a trained trunk feature extraction network; the training process comprises the following substeps:

step 1.1: acquiring a plurality of triplets of original image-target image-forged image, and recording as (original,target,fake)；

Step 1.2: for each set of triplets: (original，target，fake) Sequentially sending the images in the triplets into a trunk feature extraction network of the triplet networkNet( )(ii) a Obtaining the depth features of the images respectively, and recording as (Net(original)，Net(target)，Net(fake) ); wherein the trunk feature extraction networkNet( )Sharing the weight of (1);

step 1.3: calculating the feature distance between the depth feature of the original image and the depth feature of the target imageDis(Net(original),Net(target) And feature distances of target image depth features and forged image depth featuresDis(Net(target),Net(fake) Therein), whereinDis(a,b) Representing two eigenvectorsa,bThe feature distance between them, the feature vector distance calculation formula is as follows:

；

wherein

Represents the L2 distance;

step 1.4: calculating the network loss function of the triplet according to the characteristic distance calculated in the step 1.3loss，The loss function calculation formula is as follows:

loss=max (Dis(Net(original),Net(target))-Dis(Net(target),Net(fake))+margin,0)；

wherein,marginsetting the interval between two characteristic distances for the hyper-parameter;

step 1.5: after calculating the loss, performing back propagation and optimization on the trunk feature extraction network by using an Adam optimizer;

and 1.6, repeating the step 1.1-1.5, and training the trunk feature extraction network until convergence to obtain the trained trunk feature extraction network.

3. The method for detecting the deep forged face based on the triplet network as claimed in claim 2, wherein the step 1.1 comprises the following steps:

step 1.1.1: selecting frame images starting from specified frames and having fixed frame number intervals from the forged face video, the target face video and the original face video; enabling the original face video frame, the target face video frame and the forged face video frame to correspond one to one; generating a group of original face-target face-fake face images;

step 1.1.2: preprocessing the images in the step 1.1.1, and identifying and cutting a face area of each image through a face detection technology; aligning the human face through the human face feature point, so that the aligned human face is positioned in the center of the image; organizing the obtained image into a triplet of an original image, a target image and a fake image, and marking as (original,target,fake)。

4. The method for detecting the deep forged face based on the triplet network as claimed in claim 1, wherein: the classification network in the step 2 is a trained classification network; the training process comprises the following substeps:

step 2.1: counterfeit human face image to be detectedIPreprocessing into 299 x 3 size, inputting into the main feature extraction network in the trained three-cell network to obtain the depth feature of the face imageNet(I)； Net(I)Is a characteristic vector of 2048 dimensions;

step 2.2: using a classification network pairNet(I)Classifying, namely outputting a 2-dimensional feature by a classification network for the input 2048-dimensional feature vector, and converting the numerical value of the 2-dimensional feature into relative probability through Softmax processing to express the relative probability of authenticity of the picture;

step 2.3: calculating intersections between predicted results and actual resultsEntropy loss functionloss，lossThe calculation method of (2) is as follows:

；

whereinp _i Representative sampleiThe probability of being true is the probability that,y _i representative sampleiIf the sample is a label ofiTo forge a picture, theny _i =0, otherwise y _i =1；

Step 2.4: after the cross entropy loss is calculated, an SGD optimizer is used, and a gradient descent method is used for back propagation and optimizing a classification network;

step 2.5: and (5) repeating the step 2.1 and the step 2.4 until the classification network is converged to obtain the trained classification network.

5. A depth fake face detection system based on a three-cell network is characterized by comprising the following modules:

module 1 for detecting a fake face image to be detectedIPreprocessing the image into a preset size and inputting the preset size into a trunk feature extraction network of a three-cell network to obtain the depth feature of the face imageNet(I)；Net(I)Is a characteristic vector of 2048 dimensions;

module 2 for utilizing a classification network pairNet(I)Classifying, namely outputting a 2-dimensional feature by a classification network for the input 2048-dimensional feature vector, and converting the numerical value of the 2-dimensional feature into relative probability through Softmax processing to express the relative probability of authenticity of the picture; wherein, the picture with the probability larger than the preset value is a forged face picture;

the main feature extraction network of the three-cell network adopts a framework of an Xconvergence network, and comprises an inlet flow, a middle flow and an outlet flow; the ingress stream contains 2 convolutions of 3 × 3 and is activated using the ReLU activation function and 3 volume blocks; the intermediate stream contains 8 convolution modules; the exit stream comprises a volume block and two times of 3 x 3 depth separable convolutions, and is activated by using a ReLU function, and finally an average pooling operation is performed; three trunk characteristic extraction networks share one weight;

the classification network adopts a BP neural network and comprises three layers, wherein the first layer is an input layer and comprises 2048 nodes; the middle layer comprises 1024 nodes, and the output layer comprises 2 nodes; and a ReLU activation function is used between each layer for activation.

6. A depth forgery face detection device based on three-cell network is characterized by comprising:

one or more processors;

storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the network-based deep false face detection method according to any one of claims 1 to 4.

Images