CN111476200A - Face de-identification generation method based on generation of confrontation network - Google Patents

Abstract

The invention discloses a face de-identification generation method based on a generative confrontation network, which acquires N pairs of face images, extracts feature vectors from each face image respectively, and uses the facial features of each face image as The occluded face image is obtained by occlusion with random noise, and the occluded face image is combined with the corresponding face image feature vector. , which constitutes a training sample; the generator and discriminator are trained using the training sample. After the training is completed, in the application stage, for each face image to be de-identified, the occluded face image and the feature vector of the face image are also obtained, and then combined and input to the generation of the trained face de-identification generation model. to obtain the de-identified face image. The present invention can generate high-quality face images of virtual users while protecting user privacy.

Description

Generative Adversarial Network-Based Face De-identification Generation Method

technical field

The invention belongs to the technical field of face recognition, and more particularly, relates to a method for generating face de-identification based on a generative confrontation network.

Background technique

With the rapid development of network information technology, face recognition technology and applications have gradually developed from academia to government departments and industry, playing an important role in more and more applications. These roles are usually used to replace or assist ID cards, passwords, and other documents are used to verify the user's identity information by virtue of the identification. However, regardless of the training or practical application of face recognition models, a large amount of high-quality labeled data is often required, and these data often carry the privacy of users' personal portraits, which are obtained by third-party operators during training or use. Impact on user privacy. It is this demand that makes the need for face de-identification generation come into being, providing unique identification of users without revealing personal privacy, enabling face recognition model training and practical application of face recognition models.

The face de-identification generation method is mainly composed of two parts, face de-identification and face image generation. The traditional methods in the past tend to focus more on the de-identification part of the face, such as K anonymity and other methods. These methods have certain shortcomings: first, after de-identifying the face with these methods, although the data can meet the de-identification requirements. requirements, but it can no longer uniquely identify users, so it cannot be used in the training and use of face recognition models, and its actual use value is low. Secondly, the clarity of these methods is poor, and the pictures are blurred, which is quite different from the real face pictures. In addition, for different face pictures of the same user, due to different shooting environments, user attire and other factors, the desensitized pictures may be completely different, that is, more user feature information is lost.

Therefore, only completing the task of face de-identification cannot meet the needs of actual face use. In the actual use of faces, the data owner needs to ensure that the user's portrait privacy is not leaked, but also to ensure the unique identification of the data and other characteristics, and to have high enough clarity and retain enough feature information to be able to use However, there is currently no effective solution to this demand in the industry.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method for generating face de-identification based on a generative confrontation network, which not only ensures that the privacy of the user's personal information is not leaked, but also ensures that the picture has a high enough definition and is as clear as possible. User features are preserved for training and application of face recognition models.

In order to achieve the above-mentioned purpose of the invention, the present invention includes the following steps in the method for generating face de-identification by generating adversarial network:

其中

表示第n对人脸图像中的第i张人脸图像，i＝1,2，n＝1,2,…,N；S1: Obtain N pairs of face images, the two face images in each pair of face images are different face images belonging to the same user, and each face image is adjusted to a preset size, and the face image pair is recorded.

in

Indicates the i-th face image in the n-th pair of face images, i=1,2, n=1,2,...,N;

所对应的特征向量为

S2: Input each face image into the pre-trained face feature extraction model, obtain the corresponding feature vector, and record each face image

The corresponding eigenvectors are

的人脸五官部分采用随机噪声进行遮挡，得到遮挡人脸图像

将其转化为向量，与人脸图像

的特征向量

进行组合作为输入，将原始人脸图像

作为真实人脸图像，构成一个训练样本的三元组

each face image

The facial features are occluded by random noise, and the occluded face image is obtained.

Convert it to a vector, with the face image

eigenvector of

Combining as input, the original face image

As a real face image, a triplet of training samples is formed

S3: Build a generative adversarial network, including a generator and a discriminator. The input of the generator is the combination of the occluded face image and the feature vector of the face image, and the output is to generate a virtual user face image. The real image used by the discriminator is The corresponding original face image;

S4: Train the generative adversarial network with the training samples obtained in step S2, select several pairs of face images from the set of face image pairs in each batch during the training process, and use the corresponding training samples as the current batch of training samples , the losses used include adversarial loss, gradient penalty loss, intra-user loss, inter-user loss and similarity loss. The calculation methods of various losses are:

The calculation method of the adversarial loss is: using the discriminator in the generative adversarial network to obtain the score of each real face image in the current batch of training samples, and the generated virtual user face corresponding to the real face image in the current batch of training samples The score of the image, the Wasserstein distance between the real face image score and the generated virtual user face image score is calculated as the adversarial loss LD;

The calculation method of gradient penalty loss is: calculate the gradient penalty loss of each training sample in the current batch of training samples, and average it as the gradient penalty loss LGP;

The calculation method of intra-user loss is as follows: using the face feature extraction model to obtain the feature vector pair of the generated virtual user face image corresponding to each pair of face images in the current batch of training samples, and calculating the two features in each feature vector pair. The cosine distance of the vector, the cosine distance corresponding to all feature vector pairs is averaged as the intra-user loss LFFI;

The calculation method of user-to-user loss is as follows: randomly select K pairs of face images in the current batch of training samples, two face images in each pair of face images belong to different users, and use the face feature extraction model to obtain K pairs of different users. The feature vector pair of the generated virtual user face image corresponding to the face image, the cosine distance of the two feature vectors in each feature vector pair is calculated, and the cosine distance corresponding to all feature vector pairs is averaged as the inter-user loss LFFO;

The calculation method of the de-identification loss is as follows: using the face feature extraction model to obtain each face image in the current batch of training samples and the corresponding feature vector of the generated virtual user face image, and calculating each face image and its corresponding generated face image. The cosine distance between the feature vectors of the virtual user face image is averaged as the de-identification loss LRF;

The calculation method of the structural similarity loss is: calculate the structural similarity of each face image in the current batch of training samples and the corresponding generated virtual user face image, and average it as the structural similarity loss Ls;

Set the discriminator loss as LD-θLGP and the generator loss as LD+αLFFI+βLs-γLRF+ηLFFO, where θ, α, β, γ, and η are preset parameters, and train the discriminator and the generator alternately;

将其转化为向量，与特征向量f′进行组合，然后输入至生成对抗网络的生成器，得到去识别化后的人脸图像p′_*。S5: Adjust the face image that needs to be de-identified and generated to a preset size, extract the feature vector f' of the obtained face image p' through the face feature extraction model, and use random noise for the face image p' Partial occlusion of the facial features to obtain a face image

It is converted into a vector, combined with the feature vector f', and then input to the generator of the generative adversarial network to obtain the de-identified face image p' _* .

The invention is based on the face de-identification generation method of generative confrontation network, obtains N pairs of face images, inputs each face image into a pre-trained face feature extraction model to obtain feature vectors, The facial features of the face are occluded by random noise to obtain the occluded face image, and the obtained occluded face image is combined with the corresponding feature vector of the face image. The real face image of the discriminator constitutes a training sample; the generator and the discriminator are trained with the training sample. After the training is completed, in the application stage, for each face image to be de-identified, the occluded face image and the feature vector of the face image are also obtained, and then combined and input to the generation of the trained face de-identification generation model. to obtain the de-identified face image.

The face image of the real user that can be used by the present invention is used to generate a high-quality face image of the virtual user, and on the basis of satisfying the de-identification, the gender, race, skin color, etc. of the user are kept to the greatest extent and have a small correlation with the identification. It ensures that the statistical information of the user set is not affected, and the pictures generated by the same user belong to the same virtual user, which ensures that the generated face image can still be used for the training and use of the face recognition model, so as to protect the user. Privacy also makes face images generated by de-identification highly usable.

Description of drawings

Fig. 1 is the specific implementation flow chart of the face de-identification generation method based on generative adversarial network of the present invention;

FIG. 2 is a comparison diagram of the original face image and the generated virtual user face image in this embodiment.

Detailed ways

The specific embodiments of the present invention are described below with reference to the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that, in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

Example

FIG. 1 is a flow chart of a specific implementation of the method for generating face de-identification based on generative adversarial network of the present invention. As shown in FIG. 1 , the specific steps of the method for generating face de-identification based on generative adversarial network of the present invention include:

S101: Obtain a face image sample:

其中

表示第n对人脸图像中的第i张人脸图像，i＝1,2，n＝1,2,…,N。Obtain N pairs of face images, the two face images in each pair of face images are different face images belonging to the same user, and each face image is normalized to a preset size, and the face image pair is recorded.

in

Indicates the ith face image in the nth pair of face images, i=1, 2, n=1, 2,...,N.

S102: Obtain training samples:

所对应的特征向量为

Input each face image into the pre-trained face feature extraction model, obtain the corresponding feature vector, and record each face image

The corresponding eigenvectors are

的人脸五官部分采用随机噪声进行遮挡，得到遮挡人脸图像

将其转化为向量，与人脸图像

的特征向量

进行组合作为输入，将原始人脸图像

作为真实人脸图像，构成一个训练样本的三元组

each face image

The facial features are occluded by random noise, and the occluded face image is obtained.

Convert it to a vector, with the face image

eigenvector of

Combining as input, the original face image

As a real face image, a triplet of training samples is formed

用于给模型提供人脸无关的背景信息，在生成过程中尽可能保留，人脸图像的特征向量为模型提供人脸信息，用于人脸去识别化生成。In the present invention, the face is blocked

It is used to provide the model with background information irrelevant to the face, and keep it as much as possible during the generation process. The feature vector of the face image provides the model with face information for face de-identification generation.

Through the occlusion of random noise, the original face information of the user is not directly input to the generative adversarial network, and the generative adversarial network learns to generate a virtual user face that is different from the original face according to the original face feature vector. The face features are similar, and the generative adversarial network can maintain this similarity during the training process, so that the de-identification still belongs to the same virtual user after generation.

S103: Build a Generative Adversarial Network:

Build a generative adversarial network, including a generator and a discriminator, where the input of the generator is the combination of the occluded face image and the feature vector of the face image, and the output is to generate a virtual user face image, and the real image used by the discriminator is the corresponding original face image.

S104: Train Generative Adversarial Networks:

The training samples obtained in step S102 are used to train the generative adversarial network. During the training process, several pairs of face images are selected from the set of face image pairs in each batch, and the corresponding training samples are used as the current batch of training samples. Since the setting of the loss is very important for the training of the generative adversarial network, in order to improve the performance of the generative adversarial network obtained by training, the losses used in the generative adversarial network in the present invention include adversarial loss, gradient penalty loss, intra-user loss, user loss Between loss and similarity loss, the calculation methods are as follows:

· Adversarial loss:

Use the discriminator in the generative adversarial network to obtain the score of each real face image in the current batch of training samples, and the score of the generated virtual user face image corresponding to the real face image in the current batch of training samples, and calculate the real face image score of the real person. The Wasserstein distance between face image scores and generated virtual user face image scores as adversarial loss LD.

Gradient penalty loss:

Calculate the gradient penalty loss for each training sample in the current batch of training samples, and average it as the gradient penalty loss LGP. Gradient penalty loss is a commonly used parameter in generative adversarial networks, and its specific calculation method will not be repeated here.

· Intra-user losses:

The face feature extraction model is used to obtain the feature vector pair of the generated virtual user face image corresponding to each pair of face images in the current batch of training samples, and the cosine distance of the two feature vectors in each feature vector pair is calculated. The cosine distance corresponding to the vector pair is averaged as the intra-user loss LFFI.

Losses between users:

K pairs of face images are randomly selected in the current batch of training samples. The two face images in each pair of face images belong to different users. The face feature extraction model is used to obtain the generated virtual images corresponding to K pairs of face images of different users. For the feature vector pairs of the user's face image, the cosine distance of the two feature vectors in each feature vector pair is calculated, and the cosine distances corresponding to all feature vector pairs are averaged as the inter-user loss LFFO.

De-identification loss:

The face feature extraction model is used to obtain each face image in the current batch of training samples and the corresponding feature vector of the generated virtual user face image, and the sum of each face image and its corresponding feature vector of the generated virtual user face image is calculated. The cosine distance between them is averaged as the de-identification loss LRF.

Structural similarity loss:

Calculate the structural similarity (Structural Similarity Index) of each face image in the current batch of training samples and the corresponding generated virtual user face image, and use the average as the structural similarity loss Ls. Structural similarity combines the contrast, brightness and structural similarity between two images, which can better measure the similarity of two images.

Set the discriminator loss as LD-θLGP, and the generator loss as LD+αLFFI+βLs-γLRF+ηLFFO, where θ, α, β, γ, and η are preset parameters, generally normal numbers, which can be set according to actual needs , train the discriminator and the generator alternately, that is, first fix the generator parameters, train the discriminator, then maximize LD-θLGP, then fix the discriminator parameters, train the generator, at this time, make LD+αLFFI+βLs-γLRF+ηLFFO minimum Finally, the two converge, and the final generated model is obtained. In this embodiment, θ=5, α=0.5, β=0.1, γ=0.1, and η=0.25.

Through the above loss settings, the face image generated by the generative adversarial network can be different from the original face image, but can retain the features in the original face image, and the generated face image obtained from the same user's face image can be generated. The virtual user face images are as similar as possible.

The end condition of the training can be set as required, and the end of the training in this embodiment is judged in the following manner: Calculate the cosine distance between the feature vector pairs corresponding to each pair of face images in the current batch of training samples, and average them as the true value. face image cosine distance, and then calculate the cosine distance between the corresponding feature vector pairs of the generated virtual user face image pair obtained by each pair of face images in the current batch of training samples, and average them as the generated virtual user face image Cosine distance, if the cosine distance of the generated virtual user face image is greater than the cosine distance of the real face image, continue training, otherwise the training ends.

S105: face de-identification generation:

将其转化为向量，与特征向量f′进行组合，然后输入至生成对抗网络的生成器，得到去识别化后的人脸图像p′_*。Normalize the face image that needs to be de-identified to a preset size, extract the feature vector f' of the obtained face image p' through the face feature extraction model, and use random noise for the face image p' Partial occlusion of the facial features to obtain a face image

It is converted into a vector, combined with the feature vector f', and then input to the generator of the generative adversarial network to obtain the de-identified face image p' _* .

In order to further improve the quality of the generated face image, the generated face image can also be verified, and the specific method is as follows:

将其转化为向量，与特征向量f″进行组合，然后输入至生成对抗网络的生成器，得到去识别化后的人脸图像p″_*。1) Acquire different face images belonging to the same user as the face image p', normalize them to a preset size, and obtain a face image p". Similarly, pass the obtained face image p" through the face The feature extraction model extracts its feature vector f", and uses random noise to occlude the facial features of the face image p" to obtain a face image.

It is converted into a vector, combined with the feature vector f″, and then input to the generator of the generative adversarial network to obtain the de-identified face image p″ _* .

2) Use the discriminator of the generative adversarial network to obtain the score corresponding to the de-identified face image p′ _* . If it is less than the preset threshold, it is considered that the de-identified face image p′ _* is not realistic enough, and the verification is not correct. Pass, otherwise go to step 3).

3) Use the face feature extraction model to extract the feature vectors of the de-identified face image p′ _* and the face image p″ _* , and judge the face image p′ _* and the face image p″ by the similarity of the feature vectors _* Whether it comes from the same user, if not, the verification fails, otherwise go to step 4).

4) Use the face feature extraction model to extract the face image p′ and the de-identified face image p′ _* feature vector, and judge whether the face image p′ and the face image p′ _* come from the similarity of the feature vectors For the same user, if it belongs to the same user, the verification fails, otherwise, go to step 5).

5) Calculate the structural similarity between the face image p' and the face image p' _* , if the structural similarity is greater than the preset threshold, the verification is passed, otherwise the verification fails.

In order to better illustrate the technical effect of the present invention, the present invention is experimentally verified by using a specific example. FIG. 2 is a comparison diagram of the original face image and the generated virtual user face image in this embodiment. As shown in Figure 2, the generated virtual user face image is significantly different from the original face image, which effectively protects user privacy, but retains the user's gender, race, skin color and other characteristics in the original face image, and its clarity The degree of loss compared to the original face image is also within the acceptable range for the application.

Although the illustrative specific embodiments of the present invention have been described above to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, As long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the inventive concept are included in the protection list.

Claims (3)

1. a face de-identification generation method based on generative confrontation network, is characterized in that, comprises the following steps: S1: Obtain N pairs of face images, the two face images in each pair of face images are different face images belonging to the same user, and each face image is adjusted to a preset size, and the face image pair is recorded.

in

Indicates the ith face image in the nth pair of face images, i=1,2, n=1,2,...,N; S2: Input each face image into the pre-trained face feature extraction model, obtain the corresponding feature vector, and record each face image

The corresponding feature vector is f _i ⁿ ; each face image

The facial features are occluded by random noise, and the occluded face image is obtained.

Convert it to a vector, with the face image

The feature vector f _i ⁿ is combined as input, the original face image

As a real face image, a triplet of training samples is formed

S3: Build a generative adversarial network, including a generator and a discriminator. The input of the generator is the combination of the occluded face image and the feature vector of the face image, and the output is to generate a virtual user face image. The real image used by the discriminator is The corresponding original face image; S4: Train the generative adversarial network with the training samples obtained in step S2, select several pairs of face images from the set of face image pairs in each batch during the training process, and use the corresponding training samples as the current batch of training samples , the losses used include adversarial loss, gradient penalty loss, intra-user loss, inter-user loss and similarity loss. The calculation methods of various losses are: The calculation method of the adversarial loss is: using the discriminator in the generative adversarial network to obtain the score of each real face image in the current batch of training samples, and the generated virtual user face corresponding to the real face image in the current batch of training samples The score of the image, the Wasserstein distance between the real face image score and the generated virtual user face image score is calculated as the adversarial loss LD; The calculation method of gradient penalty loss is: calculate the gradient penalty loss of each training sample in the current batch of training samples, and average it as the gradient penalty loss LGP; The calculation method of intra-user loss is as follows: using the face feature extraction model to obtain the feature vector pair of the generated virtual user face image corresponding to each pair of face images in the current batch of training samples, and calculating the two features in each feature vector pair. The cosine distance of the vector, the cosine distance corresponding to all feature vector pairs is averaged as the intra-user loss LFFI; The calculation method of intra-user loss is as follows: randomly select K pairs of face images in the current batch of training samples, two face images in each pair of face images belong to different users, and use the face feature extraction model to obtain K pairs of different face images. The feature vector pair corresponding to the user face image to generate the virtual user face image, calculate the cosine distance of the two feature vectors in each feature vector pair, and average the cosine distances corresponding to all feature vector pairs as the inter-user loss LFFO ; The calculation method of the de-identification loss is as follows: using the face feature extraction model to obtain each face image in the current batch of training samples and the corresponding feature vector of the generated virtual user face image, and calculating each face image and its corresponding generated face image. The cosine distance between the feature vectors of the virtual user face image is averaged as the de-identification loss LRF; The calculation method of the structural similarity loss is: calculate the structural similarity of each face image in the current batch of training samples and the corresponding generated virtual user face image, and average it as the structural similarity loss Ls; Set the discriminator loss as LD-θLGP and the generator loss as LD+αLFFI+βLs-γLRF+ηLFFO, where θ, α, β, γ, and η are preset parameters, and train the discriminator and the generator alternately; S5: Adjust the face image that needs to be de-identified and generated to a preset size, extract the feature vector f' of the obtained face image p' through the face feature extraction model, and use random noise for the face image p' Partial occlusion of the facial features to obtain a face image

It is converted into a vector, combined with the feature vector f', and then input to the generator of the generative adversarial network to obtain the de-identified face image p' _* . 2. The method for generating face de-identification according to claim 1, wherein the method for judging the end of training that generates an adversarial network in the step S4 is: calculating the corresponding value of each pair of face images in the current batch of training samples The cosine distance between the feature vector pairs is averaged as the cosine distance of the real face image, and then the distance between the feature vector pairs corresponding to the generated virtual user face image pairs obtained by each pair of face images in the current batch of training samples is calculated. If the cosine distance of the generated virtual user face image is greater than the cosine distance of the real face image, continue training, otherwise the training ends. 3. The method for generating face de-identification according to claim 1, wherein the step S5 also includes verifying the generated de-identified face image, and the specific steps include: 1) Obtain different face images belonging to the same user as the face image p', adjust it to a preset size, and obtain a face image p"; extract the obtained face image p" through a face feature extraction model. The feature vector f", the face image p" is partially occluded by random noise to obtain the face image

Convert it into a vector, combine it with the feature vector f", and then input it to the generator of the generative adversarial network to obtain the de-identified face image p"_*; 2) Use the discriminator of the generative adversarial network to obtain the score corresponding to the de-identified face image p′ _* , if it is less than the preset threshold, the verification fails, otherwise, go to step 3). 3) Use the face feature extraction model to extract the feature vectors of the de-identified face image p′ _* and the face image p″ _* , and judge the face image p′ _* and the face image p″ by the similarity of the feature vectors _* Whether it comes from the same user, if not, the verification fails, otherwise go to step 4); 4) Use the face feature extraction model to extract the face image p′ and the de-identified face image p′ _* feature vector, and judge whether the face image p′ and the face image p′ _* come from the similarity of the feature vectors For the same user, if it belongs to the same user, the verification fails, otherwise go to step 5); 5) Calculate the structural similarity between the face image p' and the face image p' _* , if the structural similarity is greater than the preset threshold, the verification is passed, otherwise the verification fails.

Images