CN114139198A - Face generation privacy protection method based on hierarchical k anonymous identity replacement - Google Patents

Face generation privacy protection method based on hierarchical k anonymous identity replacement Download PDF

Info

Publication number
CN114139198A
CN114139198A CN202111431904.6A CN202111431904A CN114139198A CN 114139198 A CN114139198 A CN 114139198A CN 202111431904 A CN202111431904 A CN 202111431904A CN 114139198 A CN114139198 A CN 114139198A
Authority
CN
China
Prior art keywords
image
face
anonymous
identity
generator
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202111431904.6A
Other languages
Chinese (zh)
Inventor
匡振中
陈超
俞俊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hangzhou Dianzi University
Original Assignee
Hangzhou Dianzi University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hangzhou Dianzi University filed Critical Hangzhou Dianzi University
Priority to CN202111431904.6A priority Critical patent/CN114139198A/en
Publication of CN114139198A publication Critical patent/CN114139198A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/60—Protecting data
    • G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
    • G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
    • G06F21/6245—Protecting personal data, e.g. for financial or medical purposes
    • G06F21/6254—Protecting personal data, e.g. for financial or medical purposes by anonymising data, e.g. decorrelating personal data from the owner's identification
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G06F18/232—Non-hierarchical techniques
    • G06F18/2321—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions
    • G06F18/23213—Non-hierarchical techniques using statistics or function optimisation, e.g. modelling of probability density functions with fixed number of clusters, e.g. K-means clustering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Bioethics (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Computing Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a face generation privacy protection method based on hierarchical k anonymous identity replacement, which comprises the steps of firstly preprocessing a face image data set, then constructing a hierarchical k anonymous generation confrontation network structure, and constructing a hierarchical k anonymous target function; then constructing a face replacement to generate a confrontation network structure, and constructing a target function of the face replacement; and finally, training and testing by using the public data set to obtain a trained hierarchical k anonymous generation countermeasure network and a human face replacement generation countermeasure network. The target face replaced by the method is also generated through the network, so that the privacy of others cannot be violated, and the method is more effective and visually friendly compared with the traditional mosaic shielding method. The experimental result obviously proves the high efficiency and the practicability of the method, and the privacy protection on the human images is more efficient and beautiful.

Description

Face generation privacy protection method based on hierarchical k anonymous identity replacement
Technical Field
The present invention relates to the field of face image privacy protection. With the advent of the big data artificial intelligence era, the privacy security problem of personal images is receiving more and more attention. Therefore, the invention provides a face anonymity-based method, which uses the generated proxy data set to replace the face image so as to achieve the purpose of protecting the image anonymity.
Background
Under the information era, with the rapid promotion of the internet technology, the real-time knowledge and idea spreading and the like become possible, and the most true communication among people becomes the reality. Today, the advent of high performance devices, the continual iteration of technology, has brought the development of artificial intelligence into the orbit. Among them, the image processing technique based on deep learning is frequently seen in daily life of people. Such as face recognition, image classification, object detection, intelligent monitoring, automated driving, and so forth.
The technology is like a double-edged sword, brings convenience to daily life of people, and simultaneously needs to ensure the safety of people. At present, the face recognition technology is used for functions of mobile phone password unlocking, access control opening and closing, work card punching, transaction payment, station passing and the like, and the inherent essence of the face recognition technology is that the functions are achieved by utilizing rich information data contained in a face image. Nowadays, with the popularization of face recognition systems, privacy security issues of users are controversial. The face recognition system can be applied to daily life of people owing to face image information provided by each person. It needs to collect a large amount of user image data to improve the recognition rate, and these image data are usually stored in the cloud of the network, and if the face image data are once revealed, the privacy of the user will suffer from unprecedented loss.
In order to solve the privacy protection problem of images, corresponding laws and regulations are issued by some countries and regions to protect the privacy of the public, and even some face data sets are collected. Although these methods have a certain effect on protecting the privacy of the face image, these methods cannot fundamentally solve the privacy problem of the public, and on the contrary, the implementation of these measures can increase the research difficulty of researchers. In fact, tasks such as face detection and pedestrian tracking do not need to use identity information in a face image, so that a reliable anonymous method can not only hide original identity information, but also can use the anonymous image for other computer vision tasks.
In recent years, with the rapid development of deep learning, Generative models represented by Generative Adaptive Networks (GANs) and Variational Auto-Encoders (VAEs) provide a technical base and conditions for the intellectualization of privacy protection. The face image to be anonymized is input into a proper generator model, and corresponding information is added into a proper layer to obtain a corresponding face anonymization image, which is a research problem worthy of further exploration.
In conclusion, anonymization of the face image by using the generation countermeasure network is a direction worthy of research, and the patent is about to solve the difficulties and the key points existing in the current method by switching in from a plurality of key point problems in the task.
The key point of the anonymity of the face of the image is to ensure the anonymity effectiveness of the face, but for most of the current methods, the anonymized face is often very similar to the original image, so that the anonymity effect cannot be achieved, or the quality of the anonymized image is very poor, although the anonymization purpose can be achieved, the usability of the anonymized image is seriously influenced. Specifically, there are two key points in the following aspects:
(1) validity of anonymous images. How to ensure that the anonymous image and the original image have different identity information, in other words, the anonymous image will not appear in the original data set, which is a problem to be solved urgently.
(2) At present, the existing method mainly uses face changing or uses the identity of other people to fuse with the data of the existing method to realize anonymity, but the information of the target face or the target identity can also be revealed after face changing or identity changing. Therefore, how to anonymize the face image of the user without revealing the identity information of other people is a difficult problem at present.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a face generation privacy protection method based on hierarchical k anonymous identity replacement. The invention mainly comprises three points: 1. the invention provides a two-stage generation model for anonymity of human face identity; 2. a traditional k anonymity is improved into a hierarchical k anonymity method; 3. and (3) providing a face replacement model, coding face identity information, feeding the coded information into a generator model to obtain an anonymous image of a corresponding identity, and improving the generation quality of the anonymous image.
A face generation privacy protection method based on hierarchical k anonymous identity replacement comprises the following steps:
step 1: preprocessing a face image data set;
step 2: constructing a hierarchical k anonymous generation countermeasure network structure;
and step 3: constructing a target function with a level k anonymity;
and 4, step 4: constructing a face replacement to generate a confrontation network structure;
and 5: constructing a target function for face replacement;
step 6: training and testing are carried out by using the public data set, and a trained hierarchical k anonymous generation countermeasure network and a human face replacement generation countermeasure network are obtained;
and 7: and generating a countermeasure network anonymously through the trained hierarchy k and generating the countermeasure network by face replacement to finish the image anonymity.
The step 1 comprises the following steps:
1-1, detecting a face area of each image by using a face detector (dlib or MTCNN), and cutting the face area according to face coordinates to obtain a cut image only containing a face so as to avoid the influence of background information on anonymity.
1-2, performing hierarchical clustering on the identity information of the cut face images by adopting a k-means unsupervised clustering mode to obtain a clustered group id.
1-3, processing the uncut image by adopting the existing pre-trained face segmentation network, generating a mask for face segmentation of each picture, and simultaneously splicing the segmented face mask and the uncut image to obtain an uncut background image.
The step 2 comprises the following steps:
the level k anonymous generation countermeasure network comprises a generator, a style and style information extractor, a true and false discriminator and a group id discriminator. Firstly, inputting a group id obtained by common Gaussian noise and image hierarchical clustering into a style information extractor to obtain style information of a corresponding group id; then inputting the style information and the cutting image into a generator to obtain a generated image; secondly, inputting the generated image and the corresponding group id into a true and false discriminator to discriminate the true and false of the image; the generated graph is also input to a group id discriminator to discriminate whether the generated graph has the correct group id. Through the training of the process, the generated picture with vivid visual effect is finally obtained.
2-1 construct the generator. The generator is composed of a plurality of symmetrical Resblock residual blocks. Each residual block has convolution layer, InstanceNorm2d normalization and LeakyReLU activation function, and each residual block receives a 128-dimensional style information, which is injected by AdaIN and expressed as:
Figure BDA0003380549440000051
wherein, thetaiRepresenting an image, μ (θ)i) Representing the mean, σ (θ) of the imagei) Denotes the standard deviation of the image, μ(s)i) Representing the mean, σ(s), of style information input to the generatori) The standard deviation of the style information is represented.
2-2 build a style and style information extractor. The style extractor is composed of a plurality of full connection layers and a ReLU activation function, and finally obtains style information of a corresponding group according to input common Gaussian noise and a corresponding group id.
2-3, constructing an image true and false discriminator. Similar to the structure of the generator, the discriminator is formed by stacking a plurality of Resblock residual blocks, and the difference is that the residual blocks do not need to input style information, and the discriminator takes the cluster group id of the input graph as an index to discriminate true and false graphs.
2-4 construct a group id discriminator. The group id discriminator adopts a network structure of VGG19, a VGG19 network uses a convolution kernel size (3x3) and a maximum pooling size (2x2) with the same size, and the VGG19 comprises 19 hidden layers, wherein 16 convolutional layers and 3 full-connection layers are included. Whether the generated map output from the generator has a correct group id is discriminated by a group id discriminator.
The step 3 comprises the following steps:
the objective function with the level k anonymity comprises an objective function of a generative countermeasure network GAN and a group id discrimination objective function.
Goal function of GAN:
the anonymous image generation is controlled by adopting the idea of the condition GAN, and different anonymous images are obtained by inputting different conditions. The specific operation is that the pattern information of the cut image and the target cluster is input into a generator to obtain a generation graph of the target cluster; meanwhile, the cutting graph, the generated anonymous graph and the corresponding clustering information, target clustering id and gender information are input into an image true and false discriminator to discriminate true and false. In mathematical form can be expressed as:
Figure BDA0003380549440000061
wherein x represents a cut image, s represents the output of the style extractor, y is formed by jointly encoding information of clustering, group id to be mapped and gender information, G represents a generator, and D represents an image true and false discriminator.
Group id discriminates the objective function:
loss of identity information between anonymous images of faces belonging to the same group id is calculated using a jointly trained VGG19 network model. Extracting identity information of images using a VGG19 network, and then calculating L between features belonging to the same group id images1Distance differences, for the same group id, have the same identity information for their corresponding generated graph. FromMathematically it can be expressed as:
Figure BDA0003380549440000062
wherein x is1And x2Representing different cropped images, s being the output of the style extractor, and I representing the group id discriminator.
The step 4 comprises the following steps:
the face replacement generation confrontation network structure comprises a generator, an identity extractor, a mask generator and a true and false discriminator. Firstly, inputting a generated image obtained by anonymously generating a countermeasure network by a level k into an identity extractor to obtain the identity characteristics of the image; then inputting the obtained identity characteristics and the uncut background image into a generator; secondly, inputting an original image which is not cut into a mask generator to obtain a soft mask; and finally, splicing the output of the generator, the output of the mask generator and the uncut original image to obtain a final anonymous image, and inputting the anonymous image into a true and false discriminator to judge true and false. And finally, obtaining an anonymous image with vivid visual effect through the training of the process.
4-1 construct the generator. The generator is formed by symmetrically combining a plurality of Resblock residual blocks. Each residual block has convolution layer, InstanceNorm2d normalization, and LeakyReLU activation functions, and each residual block also receives 512-dimensional identity information that is injected into the generator network by means of AdaIN.
4-2 construct the identity code extractor. The extractor is composed of a plurality of Resblock residual blocks, each residual block is composed of a series of convolution layers, an InstanceNorm2d normalization layer, a LeakyReLU activation function and an average pooling layer, and identity information of an input face image is extracted according to the image.
4-3, constructing a true and false discriminator. The discriminator is formed by stacking a plurality of Resblock residual blocks, the true and false discriminator only needs to input a face image, and finally the probability of the true and false of the image is output, so that a true and false image is discriminated.
4-4 construct the mask generator. The mask generator adopts a U-net structure and comprises 5 downsampling blocks and 5 upsampling blocks, and the middle bottleneck layer comprises 3 Resblock residual blocks. The down-sampling block comprises a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the up-sampling block comprises an Upesample up-sampling layer, a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the Resblock residual block includes two convolutional layers and a ReLU activation function. The mask generator is used to generate a face region mask having soft borders.
The step 5 comprises the following steps:
the target function of the face replacement comprises a target function of the GAN, an identity preserving target function, a reconstruction target function and a mask generating target function.
5-1 GAN. The image generation is controlled by adopting the CGAN idea, so that the id of the image providing the identity characteristic can be transferred to the original image. Inputting an uncut background image in a generator, and simultaneously feeding coded identity information serving as a condition into the next five layers of Resblock residual blocks in the generator; similarly, the original image and the generated anonymous image are input into a discriminator to be discriminated. In mathematical form can be expressed as:
Figure BDA0003380549440000081
wherein x is the original uncut image, id is the coded identity information, and m represents the face mask obtained by segmentation using the pre-trained segmentation network.
5-2 identity preserving objective function. Calculating feature loss using a pre-trained VGG16 network, calculating id identity similarity between generated anonymous images and images providing identity features using the network, inputting the generated anonymous images and images providing identity features into the VGG16 network, calculating L between feature maps1Distance. Meanwhile, the generated anonymous image and the image providing the identity feature are input into an identity code extractor, the cosine similarity of the extracted identity information is calculated,the aim is to make the generated anonymous image have the same identity information as the image providing the identity feature. In mathematical form can be expressed as:
Figure BDA0003380549440000082
where V (-) represents a plurality of feature maps, x, extracted from a VGG16 networktFace image representing target id, cos (-) representing cosine similarity between two vectors, Zid(. -) represents an identity code extractor.
5-3 reconstruct the objective function. When the target identity id information is used as it is, the generated anonymous image is made identical to the original uncut image, so the pixel level L is adopted1Distance to build up losses. In mathematical form can be expressed as:
Figure BDA0003380549440000091
wherein x is the original uncut image, xrecIndicating a reconstructed image obtained using the same id information as itself.
5-4 mask generation of the objective function. Since the mask boundaries obtained by segmenting the network using the pre-trained face are not soft and there are some face regions that are not detected due to the influence of the pre-training performance of the network, the generated mask and the L of the segmented mask are used1The distance is controlled such that the generated mask has similar boundaries as the segmented mask. In mathematical form can be expressed as:
Figure BDA0003380549440000092
wherein, M (-) represents a mask generator, x represents an original image with a background, and M represents a face mask obtained by adopting a pre-trained segmentation network.
The step 6 comprises the following steps:
6-1, preparing a data set, processing the data set by adopting a public face data set according to the preprocessing process in the step 1, and obtaining a cut image only containing a face, a cluster group id, a face segmentation mask and an uncut background image.
6-2, inputting the cut images and the clustering group id into a level k anonymous generation countermeasure network for training, testing by using test data, and generating the countermeasure network anonymously through the level k after training to obtain a generated image corresponding to the training data, wherein the generated image is used for providing identity characteristics for a face replacement generation countermeasure network model.
6-3, inputting the face segmentation mask, the uncut original image, the uncut background image and the generated image obtained by the level k anonymous generation countermeasure network into the face replacement generation countermeasure network for training, and testing through the test data.
The step 7 comprises the following steps:
7-1, carrying out the pretreatment of the step 1 on the image needing anonymity to obtain a cut image, a group id of hierarchical clustering and a face segmentation mask.
7-2, inputting the cut images and the group id of hierarchical clustering into a trained hierarchical k anonymous generation countermeasure network to obtain a generation diagram.
7-3, inputting a generated image, a face segmentation mask, a background image and an uncut image which are obtained by anonymously generating the confrontation network by the level k into the trained face replacement generation confrontation network, and finally obtaining an anonymous image of the uncut image.
The invention has the following benefits:
the method replaces the face region in the image to achieve the effect of face anonymity, and meanwhile, the replaced target face is also generated through the network, so that the privacy of others cannot be invaded, and the method is more effective and more visually friendly compared with the traditional mosaic shielding method. The experimental results clearly prove the high efficiency and the practicability of the proposed method. In conclusion, the proposed method is more efficient and aesthetically pleasing for privacy protection of the person image.
Drawings
FIG. 1 is a flow chart of the steps of an embodiment of the method of the present invention;
FIG. 2 is a diagram of a method embodiment of the invention for generating a countermeasure network architecture graph anonymously at level k;
fig. 3 is a network architecture diagram of a face replacement model according to an embodiment of the method of the present invention.
Detailed Description
The invention is further illustrated by the following figures and examples.
As shown in fig. 1, a face generation privacy protection method based on hierarchical k anonymous identity replacement specifically includes the following steps:
step 1: preprocessing a data set:
1-1, detecting a face area of each image by using a face detector (dlib or MTCNN), and cutting the face area according to face coordinates to obtain a cut image only containing a face so as to avoid the influence of background information on anonymity.
1-2, performing hierarchical clustering on the identity information of the cut face images by adopting a k-means unsupervised clustering mode to obtain a clustered group id.
1-3, processing the uncut image by adopting the existing pre-trained face segmentation network, generating a mask for face segmentation of each picture, and simultaneously splicing the segmented face mask and the uncut image to obtain an uncut background image.
Step 2: constructing a hierarchical k anonymous generation countermeasure network structure:
as shown in fig. 2, the hierarchical k anonymity generation countermeasure network includes a generator, a style and style information extractor, a true and false discriminator, and a group id discriminator. Firstly, inputting a group id obtained by common Gaussian noise and image hierarchical clustering into a style information extractor to obtain style information of a corresponding group id; then inputting the style information and the cutting image into a generator to obtain a generated image; secondly, inputting the generated image and the corresponding group id into a true and false discriminator to discriminate the true and false of the image; the generated graph is also input to a group id discriminator to discriminate whether the generated graph has the correct group id. Through the training of the process, a more vivid generated image with a visual effect is finally obtained, and the generated image generated in the process provides identity characteristics for the identity replacement network in the step 4.
2-1 construct the generator. The generator is composed of a plurality of symmetrical Resblock residual blocks. Respectively adopting 5 resblocks to construct a network structure, respectively injecting 128-dimensional style information in the following 5 Resblock residual blocks in an AdaIN mode, wherein the style information is injected in an AdaIN mode, and the expression is as follows:
Figure BDA0003380549440000121
wherein, thetaiRepresenting an image, μ (θ)i) Representing the mean, σ (θ) of the imagei) Denotes the standard deviation of the image, μ(s)i) Representing the mean, σ(s), of style information input to the generatori) The standard deviation of the style information is represented.
2-2 build a style and style information extractor. The style extractor consists of six full-link layers and six ReLu activation functions, and finally obtains style information of a corresponding group according to input common Gaussian noise and a corresponding group id.
2-3, constructing an image true and false discriminator. The discriminator consists of five Resblock residual blocks, four convolutional layers and four learkyrelu activation functions. The residual block does not need to receive style information, and the discriminator takes the cluster group id to which the input graph belongs as an index to discriminate true and false graphs.
2-4 construct a group id discriminator. The group id discriminator adopts a network structure of VGG19, a VGG19 network uses a convolution kernel size (3x3) and a maximum pooling size (2x2) with the same size, and the VGG19 comprises 19 hidden layers, wherein 16 convolutional layers and 3 full-connection layers are included. Whether the generated map output from the generator has a correct group id is discriminated by a group id discriminator.
And step 3: constructing a target function of hierarchical k anonymity:
the objective function with the level k anonymity comprises an objective function of a generative countermeasure network GAN and a group id discrimination objective function.
Goal function of GAN:
the concept of conditional GAN is used to control the generation of anonymous images, and the invention adopts the concept of k-anonymity, so that different anonymous images are obtained by inputting different conditions. The specific operation is that the pattern information of the cut image and the target cluster is input into a generator to obtain a generation graph of the target cluster; meanwhile, the cutting graph, the generated anonymous graph and the corresponding clustering information, target clustering id and gender information are input into an image true and false discriminator to discriminate true and false. In mathematical form can be expressed as:
Figure BDA0003380549440000131
wherein x represents a cut image, s represents the output of the style extractor, y is formed by jointly encoding information of clustering, group id to be mapped and gender information, G represents a generator, and D represents an image true and false discriminator.
Group id discriminates the objective function:
loss of identity information between anonymous images of faces belonging to the same group id is calculated using a jointly trained VGG19 network model. Extracting identity information of images using a VGG19 network, and then calculating L between features belonging to the same group id images1Distance differences, for the same group id, have the same identity information for their corresponding generated graph. Mathematically expressed as:
Figure BDA0003380549440000132
wherein x is1And x2Representing different cropped images, s being the output of the style extractor, and I representing the group id discriminator.
And 4, step 4: constructing a face replacement to generate a confrontation network structure:
as shown in fig. 3, the face replacement generates a confrontation network structure, which includes a generator, an identity extractor, a mask generator, and a true-false discriminator. Firstly, inputting a generated image obtained by anonymously generating a countermeasure network by a level k into an identity extractor to obtain the identity characteristics of the image; then inputting the obtained identity characteristics and the uncut background image into a generator; secondly, inputting an original image which is not cut into a mask generator to obtain a soft mask; and finally, splicing the output of the generator, the output of the mask generator and the uncut original image to obtain a final anonymous image, and inputting the anonymous image into a true and false discriminator to judge true and false. And finally, obtaining a visually vivid anonymous image through the training of the process.
4-1 construct the generator. The generator is formed by symmetrically combining five Resblock residual blocks in front of and behind the generator. Each residual block has three convolutional layers, two InstanceNorm2d normalization layers, and two LeakyReLU activation functions, and each residual block also accepts 512-dimensional identity information, which is injected into the generator network by means of AdaIN.
4-2 construct the identity code extractor. The extractor consists of five Resblock residual blocks, the residual block consists of three convolution layers, two InstanceNorm2d normalization layers, two LeakyReLU activation functions and one average pooling layer, and identity information of the image is extracted according to an input face image.
4-3, constructing a true and false discriminator. The discriminator is composed of five Resblock residual blocks, four convolution blocks and four LeakyReLU activation functions, the true and false discriminator only needs to input a face image, and the probability of the image true and false is finally output, so that a true and false image is discriminated.
4-4 construct the mask generator. The mask generator adopts a U-net structure and comprises 5 downsampling blocks and 5 upsampling blocks, and the middle bottleneck layer comprises 3 Resblock residual blocks. The downsampling block comprises a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the up-sampling block comprises an Upsample layer, a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the Resblock residual block includes two convolutional layers and a ReLU activation function. The mask generator is used to generate a face region mask having soft borders.
And 5: constructing an objective function of face replacement:
the target function of the face replacement comprises a target function of the GAN, an identity preserving target function, a reconstruction target function and a mask generating target function.
5-1 GAN. The image generation is controlled by adopting the CGAN idea, so that the id providing the identity feature can be transferred to the original image. Inputting an uncut background image in a generator, and simultaneously feeding coded identity information serving as a condition into the next five layers of Resblock residual blocks in the generator; similarly, the original image and the generated anonymous image are input into a discriminator to be discriminated. In mathematical form can be expressed as:
Figure BDA0003380549440000151
wherein x is the original uncut image, id is the coded identity information, and m represents the face mask obtained by segmentation using the pre-trained segmentation network.
5-2 identity preserving objective function. Calculating feature loss using a pre-trained VGG16 network, calculating id identity similarity between generated anonymous images and images providing identity features using the network, inputting the generated anonymous images and images providing identity features into a VGG16 network, calculating L between intermediate layers of feature maps1Distance. Meanwhile, the generated anonymous image and the image providing the identity feature are input into an identity code extractor, and the cosine similarity of the extracted identity information is calculated, so that the generated anonymous image and the image providing the identity feature have the same identity information. In mathematical form can be expressed as:
Figure BDA0003380549440000161
where V (-) represents a plurality of feature maps, x, extracted from a VGG16 networktFace representing object idImage, cos (·,) represents the cosine similarity between two vectors, Zid(. -) represents an identity code extractor.
5-3 reconstruct the objective function. When the target identity id information is used as it is, the generated anonymous image is made identical to the original uncut image, so the pixel level L is adopted1Distance to build up losses. In mathematical form can be expressed as:
Figure BDA0003380549440000162
wherein x is the original uncut image, xrecIndicating a reconstructed image obtained using the same id information as itself.
5-4 mask generation of the objective function. Since the mask boundaries obtained by segmenting the network using the pre-trained face are not soft and there are some face regions that are not detected due to the influence of the pre-training performance of the network, the generated mask and the L of the segmented mask are used1The distance is controlled such that the generated mask has similar boundaries as the segmented mask. In mathematical form can be expressed as:
Figure BDA0003380549440000163
wherein, M (-) represents a mask generator, x represents an original image with a background, and M represents a face mask obtained by adopting a pre-trained segmentation network.
Step 6: training and testing are carried out by using an open data set, and a trained hierarchical k anonymous generation countermeasure network and a face replacement generation countermeasure network are obtained:
6-1, preparing a data set, for example, using a public data set such as VGGFACE2, CelebA and the like, and processing the data set according to the preprocessing process described in step 1, and obtaining a clipped image only including a face, a group id of a cluster, a face segmentation mask and an uncleaved background image.
6-2, inputting the cut images and the clustering group id into a level k anonymous generation countermeasure network for training, testing by using test data, and generating the countermeasure network anonymously through the level k after training to obtain a generated image corresponding to the training data, wherein the generated image is used for providing identity characteristics for a face replacement generation countermeasure network model.
6-3, inputting the face segmentation mask, the uncut original image, the uncut background image and the generated image obtained by the level k anonymous generation countermeasure network into the face replacement generation countermeasure network for training, and testing through the test data.
And 7: generating a confrontation network anonymously through the trained hierarchy k and generating the confrontation network by face replacement to finish the image anonymity:
7-1, carrying out the pretreatment of the step 1 on the image needing anonymity to obtain a cut image, a group id of hierarchical clustering and a face segmentation mask.
7-2, inputting the cut images and the group id of hierarchical clustering into a trained hierarchical k anonymous generation countermeasure network to obtain a generation diagram.
7-3, inputting a generated image, a face segmentation mask, a background image and an uncut image which are obtained by anonymously generating the confrontation network by the level k into the trained face replacement generation confrontation network, and finally obtaining an anonymous image of the uncut image.
The experimental results are as follows:
1. the anonymity rates of the method and the fuzzification, mosaic, NEO and CIAGAN methods are respectively detected. The results are detailed in table 1.
TABLE 1 comparison of anonymity rates between this and other methods
Method Shielding Blurring Pixelization Random noise NEO CIAGAN Method for producing a composite material
Anonymity rate 1.000 0.6989 0.9090 0.2399 0.8433 0.9694 0.9992
2 respectively detecting the identity exchange rate of the method and the fuzzification, mosaic, NEO and CIAGAN methods. The results are detailed in table 2.
TABLE 2 comparison of the identity exchange Rate between this and other methods
Method Shielding Blurring Pixelization Random noise NEO CIAGAN Method for producing a composite material
Rate of identity exchange 0.0000 0.3577 0.1383 0.9970 0.9990 0.0639 0.0524
3. The effective rate of the face recognition of the method and the fuzzification, mosaic, NEO and CIAGAN methods is respectively detected. The results are detailed in table 3.
TABLE 3 comparison of effective rate of face recognition between the method and other methods
Figure BDA0003380549440000181
4. The image quality of the method and the blurring, mosaic, NEO and CIAGAN methods are respectively detected. The results are detailed in table 4.
TABLE 4 results of image quality comparison experiments of this and other methods
Method Shielding Blurring Pixelization Random noise NEO CIAGAN Method for producing a composite material
Image quality 2400.69 434.82 550.97 746.12 53.36 36.03 13.56
。

Claims (8)

1. A face generation privacy protection method based on hierarchical k anonymous identity replacement is characterized by comprising the following steps:
step 1: preprocessing a face image data set;
step 2: constructing a hierarchical k anonymous generation countermeasure network structure;
and step 3: constructing a target function with a level k anonymity;
and 4, step 4: constructing a face replacement to generate a confrontation network structure;
and 5: constructing a target function for face replacement;
step 6: training and testing are carried out by using the public data set, and a trained hierarchical k anonymous generation countermeasure network and a human face replacement generation countermeasure network are obtained;
and 7: and generating a countermeasure network anonymously through the trained hierarchy k and generating the countermeasure network by face replacement to finish the image anonymity.
2. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 1, wherein the specific steps of step 1 are as follows:
1-1, detecting a face area of each image by using a face detector (dlib or MTCNN), and cutting the face area according to face coordinates to obtain a cut image only containing a face so as to avoid the influence of background information on an anonymous effect;
1-2, performing hierarchical clustering on identity information of the cut face image by adopting a k-means unsupervised clustering mode to obtain a clustered group id;
1-3, processing the uncut image by adopting the existing pre-trained face segmentation network, generating a mask for face segmentation of each picture, and simultaneously splicing the segmented face mask and the uncut image to obtain an uncut background image.
3. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 2, wherein the step 2 comprises the following steps:
the level k generates the countermeasure network anonymously, and the countermeasure network comprises a generator, a style and style information extractor, a true and false discriminator and a group id discriminator; firstly, inputting a group id obtained by common Gaussian noise and image hierarchical clustering into a style information extractor to obtain style information of a corresponding group id; then inputting the style information and the cutting image into a generator to obtain a generated image; secondly, inputting the generated image and the corresponding group id into a true and false discriminator to discriminate the true and false of the image; meanwhile, the generated graph is input into a group id discriminator to discriminate whether the generated graph has a correct group id; through the training of the process, a generated image with vivid visual effect is finally obtained;
2-1 constructing a generator; the generator consists of a plurality of symmetrical Resblock residual blocks; each residual block has convolution layer, InstanceNorm2d normalization and LeakyReLU activation function, and each residual block receives a 128-dimensional style information, which is injected by AdaIN and expressed as:
Figure FDA0003380549430000021
wherein, thetaiRepresenting an image, μ (θ)i) Representing the mean, σ (θ) of the imagei) Denotes the standard deviation of the image, μ(s)i) Representing the mean, σ(s), of style information input to the generatori) A standard deviation representing style information;
2-2, constructing a style and style information extractor; the pattern extractor consists of a plurality of full connection layers and a ReLU activation function, and finally obtains the style information of a corresponding group according to the input common Gaussian noise and the corresponding group id;
2-3, constructing an image true and false discriminator; the structure of the generator is similar to that of the generator, the discriminator is formed by stacking a plurality of Resblock residual blocks, the difference is that the residual blocks do not need to input style and style information, and the discriminator takes the cluster group id of the input graph as an index to discriminate true and false graphs;
2-4, constructing a group id discriminator; the group id discriminator adopts a network structure of VGG19, a VGG19 network uses a convolution kernel size (3x3) and a maximum pooling size (2x2) with the same size, and the VGG19 comprises 19 hidden layers, wherein 16 convolutional layers and 3 full-connection layers are included; whether the generated map output from the generator has a correct group id is discriminated by a group id discriminator.
4. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 3, wherein the step 3 comprises the following steps:
the objective function with the anonymous level k comprises an objective function of a generative countermeasure network GAN and a group id discrimination objective function;
goal function of GAN:
the method adopts the concept of conditional GAN to control the generation of anonymous images, and obtains different anonymous images by inputting different conditions; the specific operation is that the pattern information of the cut image and the target cluster is input into a generator to obtain a generation graph of the target cluster; meanwhile, the cutting graph, the generated anonymous graph and the corresponding clustering information, target clustering id and gender information are input into an image true and false discriminator to discriminate true and false; in mathematical form can be expressed as:
Figure FDA0003380549430000031
the system comprises a model style extractor, a model generator and a model true and false discriminator, wherein the model style extractor is used for extracting a model style, and the model style extractor is used for extracting a model style;
group id discriminates the objective function:
calculating the loss of identity information between anonymous face images belonging to the same group id by using a jointly trained VGG19 network model; extracting identity information of images using a VGG19 network, and then calculating L between features belonging to the same group id images1Distance difference, for the same group id, the corresponding generated graph has the same identity information; mathematically expressed as:
Figure FDA0003380549430000041
wherein x is1And x2Representing different cropped images, s being the output of the style extractor, and I representing the group id discriminator.
5. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 4, wherein the step 4 comprises the following steps:
the face replacement generation confrontation network structure comprises a generator, an identity extractor, a mask generator and a true and false discriminator; firstly, inputting a generated image obtained by anonymously generating a countermeasure network by a level k into an identity extractor to obtain the identity characteristics of the image; then inputting the obtained identity characteristics and the uncut background image into a generator; secondly, inputting an original image which is not cut into a mask generator to obtain a soft mask; finally, splicing the output of the generator, the output of the mask generator and the uncut original image to obtain a final anonymous image, and inputting the anonymous image into a true and false discriminator to judge true and false; through the training of the process, an anonymous image with vivid visual effect is finally obtained;
4-1 constructing a generator; the generator is formed by symmetrically combining a plurality of Resblock residual blocks; each residual block has a convolution layer, InstanceNorm2d normalization and LeakyReLU activation function, and each residual block receives 512-dimensional identity information which is injected into the generator network by means of AdaIN;
4-2, constructing an identity code extractor; the extractor consists of a plurality of Resblock residual blocks, the residual blocks consist of a series of convolution layers, an InstanceNorm2d normalization layer, a LeakyReLU activation function and an average pooling layer, and identity information of the image is extracted according to an input face image;
4-3, constructing a true and false discriminator; the discriminator is formed by stacking a plurality of Resblock residual blocks, the true and false discriminator only needs to input a face image, and finally the probability of the true and false of the image is output, so that a true and false image is discriminated;
4-4 constructing a mask generator; the mask generator adopts a U-net structure and comprises 5 downsampling blocks and 5 upsampling blocks, and the middle bottleneck layer comprises 3 Resblock residual blocks; the down-sampling block comprises a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the up-sampling block comprises an Upesample up-sampling layer, a convolution layer, a BatchNorm normalization layer and a ReLU activation function; the Resblock residual block comprises two convolutional layers and a ReLU activation function; the mask generator is used to generate a face region mask having soft borders.
6. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 5, wherein the specific steps of step 5 are as follows:
the target function of the face replacement comprises a target function of the GAN, an identity maintaining target function, a reconstruction target function and a mask generation target function;
5-1 GAN; the image generation is controlled by adopting the CGAN idea, so that the id of the image providing the identity characteristic can be transferred to the original image; inputting an uncut background image in a generator, and simultaneously feeding coded identity information serving as a condition into the next five layers of Resblock residual blocks in the generator; in the same way, the original image and the generated anonymous image are respectively input into a discriminator to be discriminated; in mathematical form can be expressed as:
Figure FDA0003380549430000061
wherein x is an original uncut image, id is coded identity information, and m represents a face mask obtained by adopting pre-trained segmentation network segmentation;
5-2 identity preserving objective function; calculating feature loss using a pre-trained VGG16 network, calculating id identity similarity between generated anonymous images and images providing identity features using the network, inputting the generated anonymous images and images providing identity features into the VGG16 network, calculating L between feature maps1A distance; meanwhile, the generated anonymous image and the image providing the identity feature are input into an identity code extractor, and the cosine similarity of the extracted identity information is calculated, so that the generated anonymous image and the image providing the identity feature have the same identity information; in mathematical form can be expressed as:
Figure FDA0003380549430000062
where V (-) represents a plurality of feature maps, x, extracted from a VGG16 networktFace image representing target id, coS (·,) representing cosine similarity between two vectors, Zid() represents an identity code extractor;
5-3 reconstructing an objective function; when the target identity id information is used as it is, the generated anonymous image is made identical to the original uncut image, so the pixel level L is adopted1Distance to build up losses; in mathematical form can be expressed as:
Figure FDA0003380549430000063
wherein x is the original uncut image, xrecRepresenting a reconstructed image obtained using id information identical to itself;
5-4, generating an objective function by a mask; since the mask boundaries obtained by segmenting the network using the pre-trained face are not soft and there are some face regions that are not detected due to the influence of the pre-training performance of the network, the generated mask and the L of the segmented mask are used1Controlling the distance so that the generated mask and the segmented mask have similar boundaries; in mathematical form can be expressed as:
Figure FDA0003380549430000071
wherein, M (-) represents a mask generator, x represents an original image with a background, and M represents a face mask obtained by adopting a pre-trained segmentation network.
7. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 6, wherein the step 6 comprises the following steps:
6-1, preparing a data set, processing by adopting a public face data set according to the preprocessing process in the step 1, and obtaining a cut image only containing a face, a cluster group id, a face segmentation mask and an uncut background image;
6-2, inputting the cut images and the clustering group id into a level k anonymous generation countermeasure network for training, testing by using test data, generating the countermeasure network anonymously through the level k after the training is finished to obtain a generated image corresponding to the training data, and providing identity characteristics for a face replacement generation countermeasure network model;
6-3, inputting the face segmentation mask, the uncut original image, the uncut background image and the generated image obtained by the level k anonymous generation countermeasure network into the face replacement generation countermeasure network for training, and testing through the test data.
8. The method for generating privacy protection based on hierarchical k anonymous identity replacement according to claim 7, wherein the step 7 comprises the following steps:
7-1, carrying out the pretreatment of the step 1 on the image to be anonymized to obtain a cut image, a hierarchical cluster group id and a face segmentation mask;
7-2, inputting the cut images and the group id of hierarchical clustering into a trained hierarchical k anonymous generation countermeasure network to obtain a generation diagram;
7-3, inputting a generated image, a face segmentation mask, a background image and an uncut image which are obtained by anonymously generating the confrontation network by the level k into the trained face replacement generation confrontation network, and finally obtaining an anonymous image of the uncut image.
CN202111431904.6A 2021-11-29 2021-11-29 Face generation privacy protection method based on hierarchical k anonymous identity replacement Pending CN114139198A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111431904.6A CN114139198A (en) 2021-11-29 2021-11-29 Face generation privacy protection method based on hierarchical k anonymous identity replacement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111431904.6A CN114139198A (en) 2021-11-29 2021-11-29 Face generation privacy protection method based on hierarchical k anonymous identity replacement

Publications (1)

Publication Number Publication Date
CN114139198A true CN114139198A (en) 2022-03-04

Family

ID=80389138

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111431904.6A Pending CN114139198A (en) 2021-11-29 2021-11-29 Face generation privacy protection method based on hierarchical k anonymous identity replacement

Country Status (1)

Country Link
CN (1) CN114139198A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023168903A1 (en) * 2022-03-10 2023-09-14 腾讯科技（深圳）有限公司 Model training method and apparatus, identity anonymization method and apparatus, device, storage medium, and program product
CN117195286A (en) * 2023-09-04 2023-12-08 北京超然聚力网络科技有限公司 User privacy protection method and system based on big data

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023168903A1 (en) * 2022-03-10 2023-09-14 腾讯科技（深圳）有限公司 Model training method and apparatus, identity anonymization method and apparatus, device, storage medium, and program product
CN117195286A (en) * 2023-09-04 2023-12-08 北京超然聚力网络科技有限公司 User privacy protection method and system based on big data
CN117195286B (en) * 2023-09-04 2024-05-07 河南中信科大数据科技有限公司 User privacy protection method and system based on big data

Similar Documents

Publication Publication Date Title
Asghar et al. Copy-move and splicing image forgery detection and localization techniques: a review
CN111242837B (en) Face anonymity privacy protection method based on generation countermeasure network
CN109815850A (en) Iris segmentation and localization method, system, device based on deep learning
CN114139198A (en) Face generation privacy protection method based on hierarchical k anonymous identity replacement
Yang et al. Masked relation learning for deepfake detection
CN111241989A (en) Image recognition method and device and electronic equipment
Ding et al. Real-time estimation for the parameters of Gaussian filtering via deep learning
Yu et al. Robust median filtering forensics by CNN-based multiple residuals learning
Baek et al. Generative adversarial ensemble learning for face forensics
CN111797702A (en) Face counterfeit video detection method based on spatial local binary pattern and optical flow gradient
CN114169002A (en) Key point differential privacy driven face image privacy protection method
Zhou et al. Adaptive digital watermarking scheme based on support vector machines and optimized genetic algorithm
Gao et al. TBNet: A two-stream boundary-aware network for generic image manipulation localization
CN115424314A (en) Recognizable face anonymization processing method and system
Zhu et al. Supplement and suppression: Both boundary and nonboundary are helpful for salient object detection
Shi et al. PR‐NET: progressively‐refined neural network for image manipulation localization
Gu et al. FBI-Net: Frequency-based image forgery localization via multitask learning with self-attention
Meng et al. High-capacity steganography using object addition-based cover enhancement for secure communication in networks
Lu et al. An interpretable image tampering detection approach based on cooperative game
Deng et al. New finding and unified framework for fake image detection
CN113011307A (en) Face recognition identity authentication method based on deep residual error network
CN111737688A (en) Attack defense system based on user portrait
Gan et al. Highly accurate end-to-end image steganalysis based on auxiliary information and attention mechanism
CN112668401B (en) Face privacy protection method and device based on feature decoupling
CN114937300A (en) Method and system for identifying shielded face

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination