CN116402916B - Face image restoration method and device, computer equipment and storage medium - Google Patents
Face image restoration method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN116402916B CN116402916B CN202310672537.1A CN202310672537A CN116402916B CN 116402916 B CN116402916 B CN 116402916B CN 202310672537 A CN202310672537 A CN 202310672537A CN 116402916 B CN116402916 B CN 116402916B
- Authority
- CN
- China
- Prior art keywords
- face
- model
- coding
- vector
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 239000013598 vector Substances 0.000 claims abstract description 252
- 238000012549 training Methods 0.000 claims abstract description 69
- 238000010801 machine learning Methods 0.000 claims abstract description 9
- 230000006870 function Effects 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 18
- 238000005457 optimization Methods 0.000 claims description 10
- 230000009467 reduction Effects 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 9
- 238000013473 artificial intelligence Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000012795 verification Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 230000005540 biological transmission Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000012935 Averaging Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007667 floating Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012804 iterative process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0475—Generative networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/094—Adversarial learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Molecular Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The embodiment of the application provides a face image restoration method, a device, computer equipment and a storage medium, and relates to the technical field of artificial intelligence, wherein the method comprises the following steps: when the face coding model is a black box, a training data set is constructed according to the face image and the face coding vector obtained through the face coding model, a training machine learning component is trained by adopting the training data set to obtain a face inverse coding model, and a first face coding vector to be restored is input into the face inverse coding model to obtain a restored face image corresponding to the first face coding vector to be restored; when the face coding model is a white box, optimizing hidden vectors according to the face coding vector to be restored, model related parameters of the face coding model and model related parameters of the image generation model, and inputting the optimized hidden vectors into the image generation model to obtain a restored face image corresponding to the face coding vector to be restored. The scheme can conveniently and accurately restore the face coding vector to the face image.
Description
Technical Field
The present application relates to the field of artificial intelligence technologies, and in particular, to a method and apparatus for restoring a face image, a computer device, and a storage medium.
Background
In an AI face comparison system, a face image (such as a face certificate) is encoded by an encoder based on a face recognition model to extract a face feature vector, and a database for face comparison is constructed based on the face feature vector. In the contrast stage, after a face is identified, the face image captured is encoded by an encoder of the same face recognition model to extract a face feature vector of the face image captured, and the face feature vector is compared with the existing face feature vector in a face database based on the face feature vector so as to search and locate a specific person.
However, at present, a technical scheme of restoring the face feature vector to the face image is not realized.
Disclosure of Invention
In view of this, the embodiment of the application provides a method for restoring a face image, so as to solve the technical problem that a face feature vector cannot be restored to the face image in the prior art. The method comprises the following steps:
judging whether the face coding model is a black box or a white box according to the related known information of the face coding model;
when the face coding model is a black box, a training data set is constructed according to a face image and a face coding vector obtained by the face image through the face coding model, a machine learning component is trained by adopting the training data set to obtain a face inverse coding model, and a first face coding vector to be restored is input into the face inverse coding model to obtain a restored face image corresponding to the first face coding vector to be restored; and/or the number of the groups of groups,
when the face coding model is a white box, setting hidden vectors, optimizing the hidden vectors according to the face coding vector to be restored, model related parameters of the face coding model and model related parameters of an image generation model to obtain optimized hidden vectors, and inputting the optimized hidden vectors into the image generation model to obtain a restored face image corresponding to the face coding vector to be restored.
The embodiment of the application also provides a device for restoring the face image, which aims to solve the technical problem that the face feature vector cannot be restored into the face image in the prior art. The device comprises:
the judging module is used for judging whether the face coding model is a black box or a white box according to the related known information of the face coding model;
the first face reduction module is used for constructing a training data set according to the face image and the face coding vector obtained by the face coding model when the face coding model is a black box, training a machine learning component by adopting the training data set to obtain a face inverse coding model, and inputting a first face coding vector to be reduced into the face inverse coding model to obtain a reduced face image corresponding to the first face coding vector to be reduced; and/or the number of the groups of groups,
and the second face reduction module is used for setting hidden vectors when the face coding model is a white box, optimizing the hidden vectors according to the face coding vector to be reduced, the model related parameters of the face coding model and the model related parameters of the image generation model to obtain optimized hidden vectors, and inputting the optimized hidden vectors into the image generation model to obtain a reduced face image corresponding to the face coding vector to be reduced.
The embodiment of the application also provides computer equipment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor realizes the arbitrary face image restoration method when executing the computer program so as to solve the technical problem that the face feature vector cannot be restored into the face image in the prior art.
The embodiment of the application also provides a computer readable storage medium which stores a computer program for executing the method for restoring any face image, so as to solve the technical problem that the face feature vector cannot be restored to the face image in the prior art.
Compared with the prior art, the beneficial effects that above-mentioned at least one technical scheme that this description embodiment adopted can reach include at least: judging whether the face coding model is a black box or a white box according to the related known information of the face coding model, and further constructing a training data set through a face image and a face coding vector obtained by the face coding model when the face coding model is the black box, so as to train a face inverse coding model, and further restoring the first face coding vector to be restored into the face image based on the face inverse coding model; when the face coding model is a white box, optimizing the hidden vector according to the face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model to obtain an optimized hidden vector, and generating a restored face image corresponding to the face coding vector to be restored based on the optimized hidden vector through the image generation model. The method realizes that the face coding vector can be restored to the face image under the condition that the face coding model is a white box or a black box; meanwhile, when the face coding model is a black box, the face inverse coding model is obtained by training a training data set constructed by face coding vectors obtained by the face coding model based on the face image and the face image, so that the face inverse coding model learns the relevant information of the face coding model for coding the face image into the face coding vectors, and the face coding vectors output by the face coding model can be quickly and accurately restored into the face image based on the face inverse coding model; when the face coding model is a white box, the hidden vector is optimized according to the face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model, so that the optimized hidden vector is more similar or identical to the face coding vector to be restored, and the face coding vector output by the face coding model can be quickly, conveniently and accurately restored to a face image through the image generation model based on the optimized hidden vector.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a face image restoration method provided in an embodiment of the present application;
fig. 2 is a schematic diagram of a principle of restoring a face coding vector to a face image according to an embodiment of the present application;
FIG. 3 is a schematic diagram of a training face inverse coding model according to an embodiment of the present application;
fig. 4 is a schematic diagram of a principle of face image restoration based on a face inverse coding model according to an embodiment of the present application;
fig. 5 is a schematic diagram of a model architecture of DCGAN according to an embodiment of the present application;
FIG. 6 is a schematic diagram of a principle of optimizing hidden vectors and restoring a face image based on the optimized hidden vectors according to an embodiment of the present application;
FIG. 7 is a schematic diagram of an image generation model for generating a face image based on hidden vectors according to an embodiment of the present application;
FIG. 8 is a block diagram of a computer device according to an embodiment of the present application;
fig. 9 is a block diagram of a face image restoration device according to an embodiment of the present application.
Detailed Description
Embodiments of the present application will be described in detail below with reference to the accompanying drawings.
Other advantages and effects of the present application will become apparent to those skilled in the art from the following disclosure, which describes the embodiments of the present application with reference to specific examples. It will be apparent that the described embodiments are only some, but not all, embodiments of the application. The application may be practiced or carried out in other embodiments that depart from the specific details, and the details of the present description may be modified or varied from the spirit and scope of the present application. It should be noted that the following embodiments and features in the embodiments may be combined with each other without conflict. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
In an embodiment of the present application, a method for restoring a face image is provided, as shown in fig. 1, where the method includes:
step S101: judging whether the face coding model is a black box or a white box according to the related known information of the face coding model;
step S102: when the face coding model is a black box, a training data set is constructed according to a face image and a face coding vector obtained by the face image through the face coding model, a machine learning component is trained by adopting the training data set to obtain a face inverse coding model, and a first face coding vector to be restored is input into the face inverse coding model to obtain a restored face image corresponding to the first face coding vector to be restored; and/or the number of the groups of groups,
step S103: when the face coding model is a white box, setting hidden vectors, optimizing the hidden vectors according to the face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model to obtain optimized hidden vectors, and inputting the optimized hidden vectors into the image generation model to obtain a restored face image corresponding to the face coding vector to be restored.
As can be seen from the flow shown in fig. 1, in the embodiment of the present application, it is realized that the face coding vector can be restored to the face image under the condition that the face coding model is a white box or a black box; meanwhile, when the face coding model is a black box, the face inverse coding model is obtained by training a training data set constructed by face coding vectors obtained by the face coding model based on the face image and the face image, so that the face inverse coding model learns the relevant information of the face coding model for coding the face image into the face coding vectors, and the face coding vectors output by the face coding model can be quickly and accurately restored into the face image based on the face inverse coding model; when the face coding model is a white box, the hidden vector is optimized according to the face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model, so that the optimized hidden vector is more similar or identical to the face coding vector to be restored, and the face coding vector output by the face coding model can be quickly, conveniently and accurately restored to a face image through the image generation model based on the optimized hidden vector.
In specific implementation, the inventor finds that no technical scheme for reducing the face coding vector to the face image exists in the prior art, but in actual life, the requirement for reducing the face coding vector to the face image exists. To meet this requirement, the present inventors have found that, for each face recognition model encoder (i.e., the face encoding model described above), for any face encoding vector of a face image generated by the encoder, a function f (x) for mapping the face encoding vector back to the original face image needs to be found, as shown in fig. 2 (when the face image is replaced with a sketch in the drawing of the present application, the restoration result is a specific, real face image in the implementation), by which the restoration of the face encoding vector to the face image can be achieved. The inventor of the application finds through a great deal of researches that the function can be a function in the forms of a model, a vector and the like, so that the method for restoring the face image is provided, and when the face coding model is a black box, the face coding vector is restored to the face image by training the face inverse coding model; when the face coding model is a white box, the face coding vector is restored to the face image by optimizing the hidden vector and generating the model based on the image, so that the face image can be restored conveniently, quickly and accurately.
In the specific implementation, in the process of judging whether the face coding model is a black box or a white box, whether the model related parameters such as the architecture, the weight and the like of the face coding model can be determined can be judged according to the related known information of the face coding model, and if so, the face coding model can be judged to be the white box; if not, the face coding model can be judged to be a black box.
In the specific implementation, when the face coding model is a black box, in order to realize convenient and accurate restoration of the face image, training different face inverse coding models aiming at each different face coding model is provided, and then the face coding vector output by the face coding model is restored to the face image based on the corresponding face inverse coding model.
In a specific implementation, since details such as model related information of the face coding model cannot be obtained when the face coding model is a black box, in order to improve accuracy of restoring the face image by the face inverse coding model, as shown in fig. 3, a training data set of large-scale "face image-face coding vector" image coding pairs is constructed by using an encoder SDK (i.e., the face coding model), and then a machine learning component is trained based on the training data set, so as to obtain the face inverse coding model.
In specific implementation, as shown in fig. 4, the process of restoring the face image based on the face inverse coding model generates the face image restored by the face inverse coding model for any face coding vector input to the face inverse coding model (the face coding vector needs to be generated by the face coding model used in the process of constructing the training data set of the face inverse coding model).
In the specific implementation, in order to further improve the accuracy of the face inverse coding model in restoring the face image, in the process of constructing the training data set, it is proposed to construct the training data set by averaging the face image, for example, obtain the face coding vector of each face image by the face coding model; calculating an average face image for a plurality of face images of the same person; and constructing the training data set according to the average face image of each person and the face coding vectors of a plurality of face images of each person, namely, the image coding pair in the training data set is the face coding vector of one average face image of the same person corresponding to a plurality of face images of the same person.
In the implementation, in the process of calculating the average face image for the face images of the same person, the pixels of the face images may be averaged to obtain an average face image.
In particular, in constructing the training dataset, the dataset may be constructed using face images in an existing dataset (for example, an MS1Mv2 open source dataset may be used, which contains 5,822,653 face images of 85,742 persons), and the face encoding vector of each face image may be generated by an existing face encoding model (for example, an open source ArcFace model is used as a face encoding model, which may map the face image to 512-dimensional float32 vectors), so as to form image encoding pairs in the dataset, and the dataset may be segmented into the training dataset and the test dataset by person.
In a specific implementation, in order to obtain a face inverse coding model with higher performance in the process of training the face inverse coding model, in this embodiment, it is proposed to train a deep neural network to obtain the face inverse coding model, for example, use a face coding vector in the training data set as input data of a DCGAN (deep convolution generation countermeasure network) model, use a face image in the training data set as a label of the face coding vector, and train the DCGAN model to obtain the face inverse coding model.
In particular, as shown in fig. 5, the exemplary model architecture of DCGAN is a convolution network composed of transposed convolution blocks, which uses a multidimensional vector as an input data set, uses a color image as output data, and the vector dimension of the input data can be determined according to specific requirements, for example, a 100-dimensional vector (100 z as shown in fig. 5) is used as an input, and can output a 64x64x3 color image (G (z) as shown in fig. 5).
In specific implementation, in order to further improve the performance of the face inverse coding model, the adjustment is performed on the basis of the model architecture of the DCGAN shown in fig. 5, so as to adapt to the output of the encoder and realize the face inverse coding function.
The model architecture of DCGAN has repeated blocks of the following structure:
transpose the convolution block;
batch normalization (Batch Normalization) layer;
a ReLU nonlinear layer;
wherein the last layer converts the output data to the range of 0,1 by a sigmoid function, which is the same as the floating point representation of the RGB image.
Specifically, for example, the model network of DCGAN uses 512-dimensional face coding vectors output by the face coding model as input data, firstly converts the input data into 8192-dimensional vectors through a full connection layer and a batch normalization layer, and converts reshape into 512×4×4 tensors, then sequentially converts the tensors into 256×8×8, 128×16×16, 64×32×32, 32×64×64 tensors through four consecutive transpose convolution modules, and finally converts the tensors into a format of an RGB image of 3×128×128 through a transpose convolution layer and a sigmoid function.
In the specific implementation, in order to further improve the accuracy of the face inverse coding model in recovering the face image and minimize the difference between the pixels of the recovered image and the real image during the training of the face inverse coding model, in this embodiment, the training of the face inverse coding model is regarded as a regression problem, and the distance loss at the pixel level between the images is used as a loss function to encourage the model predictive image to approach the pixel value of the real image. Specifically, the model is trained using Adam optimizers and combinations of loss functions, while trying to add gaussian noise to the input vector to enhance robustness and adding residual connections between transposed convolution blocks to avoid gradient vanishing. For example, gaussian noise data is added to input data of the DCGAN model, and/or residual connection is added between every two transposed convolution blocks in the DCGAN model, the DCGAN model is trained, and a training result index of the DCGAN model is evaluated by adopting an L1 loss function, so that the face inverse coding model is obtained.
In the implementation, different gaussian noise data can be added to the input data of the DCGAN model, and in order to further improve the robustness of the face inverse coding model, it is proposed to add gaussian noise data with a mean value of 0 and a variance of 0.001 in each dimension to the input data of the DCGAN model.
In the implementation, residual error connection with different layers can be added between every two transposed convolution blocks in the DCGAN model, and in order to further improve the reduction effect of the face inverse coding model, a layer of residual error connection is added between every two transposed convolution blocks in the DCGAN model.
In specific implementation, in order to verify the influence of adding different Gaussian noise data, adopting different loss functions and adding residual error connection of different layers on the reduction effect of the face inverse coding model, the following groups of verification experiments are performed, and the experimental configuration of each group of verification experiments is as follows:
experiment group one: l1 loss function, no Gaussian noise, no residual connection.
Experimental group two: and adding 512-dimensional Gaussian noise with the mean value of 0 and the variance of 0.005 into the L1+L2 loss function, and adding a layer of residual error connection between every two transposed convolution blocks.
Experimental group three: l1 loss function, adding 512-dimensional Gaussian noise with mean value of 0 and variance of 0.001, and adding a layer of residual error connection between every two transposed convolution blocks.
Experimental group four: l2 loss function, no Gaussian noise, and two layers of residual error connection are added between every two transposed convolution blocks.
Experimental group five: l1 loss function, no Gaussian noise, and a layer of residual error connection is added between every two transposed convolution blocks.
The loss of the trained face inverse coding model on the test set is used as an index for evaluating the training effect, and the result shows that the loss of the face inverse coding model on the test set obtained by the training of the experimental group III is minimum, so that the training of the face inverse coding model is carried out by adopting the configuration of the experimental group III. After training, the face inverse coding model can take the face coding vector output by the face coding model as input data, and accurately convert the face coding vector back to the photo of the person represented by the face coding vector (i.e. restore to a face image).
In specific implementation, by verifying the restoration effect of the face inverse coding model, the verification result shows that the face inverse coding model has a certain generalization capability, can restore the face characteristics of the original face image, and the restored face image can be identified as the same person as the original face image, which means that the face inverse coding model can accurately restore the face image.
In the specific implementation, when the face coding model is a white box, the related information of the face coding model such as the structure, the weight and the like can be known, so that the training model is not needed to restore the face image, and only the face coding vector of one face image is needed to be restored in a targeted way. There are many technical problems faced in the process of restoring the face image, for example: 1. the face features in the face coding vector are highly abstract, and detail information is lost; 2. the solution space is a face image, the dimension is high, the optimization is difficult, the reduction result is a reversely pushed face image, the detail information of the reversely pushed face image is more, and the five-sense organ feature needs to be more similar to the target face image. In order to improve the efficiency and accuracy of restoring the face image, it is proposed to complete the restoration of the face image by means of an image generation model and hidden vectors, the hidden vectors representing hidden features of the face are used as optimization objects, the hidden feature dimensions of the face are low, the face is easy to optimize, and one hidden vector is optimized by each second face coding vector to be restored, model related parameters of the face coding model and model related parameters of the image generation model respectively, so that the optimized hidden vector is more similar or more similar to the second face coding vector to be restored, and further the restored face image corresponding to the second face coding vector to be restored can be generated by the image generation model based on the optimized hidden vector.
In specific implementation, as shown in fig. 6, for each face coding vector to be restored, the restoration of the face image is realized based on the image generation model and the hidden vector by the following steps:
first, randomly initializing a hidden vector, and generating a face image (i.e., a face image in the iteration of fig. 6) based on the hidden vector by using an image generation model; and then coding the generated face image through a known face recognition model coder (namely the face coding model), obtaining a face coding vector, calculating loss of the obtained face coding vector and an original face coding vector (namely the face coding vector to be restored generated by the face coding model of the original face image), reversely transmitting model-related parameters of the loss and the face coding model and model-related parameters of the image generation model to the hidden vector, updating the parameters of the hidden vector, repeatedly iterating the process for a plurality of epochs (as shown in fig. 6, iterating for 50-100 times to optimize the hidden vector, wherein the iteration times can be determined according to specific iteration effects or specific precision requirements), inputting the optimized hidden vector into the image generation model, generating a restored face image corresponding to the original face coding vector, and thus finishing the face restoration operation.
In specific implementation, the process of iteratively optimizing the hidden vector is realized by the following steps:
taking the hidden vector as a current hidden vector, circularly executing the following steps to perform an iterative optimization process, and taking the current hidden vector after performing the iterative optimization process for preset times as the optimized hidden vector:
generating a current face image of the current hidden vector through the image generation model;
generating a current face coding vector of the current face image through the face coding model;
calculating the loss between the current face coding vector and the second face coding vector to be restored;
and reversely transmitting the loss, the model-related parameters of the face coding model and the model-related parameters of the image generation model to the current hidden vector, updating the parameters of the current hidden vector, and taking the current hidden vector after updating the parameters as a new current hidden vector.
In specific implementation, the process of iteratively optimizing the hidden vector can be implemented by the following steps:
it is known that a face recognition model outputs face coding vectors to a face imageThe face code vector ∈ ->Encoding vectors for the faces to be restored;
initializing a hidden vectorAs shown in fig. 7, the model is generated based on hidden vector +.>Generating a face image;
the generated face image is recognized through the faceOther model encoders obtain encoded vectorsEncoding vectorsDimension and face coding vector +.>The same;
calculating a coding vectorCoding vector +.>Is a loss, such as a mean square error, and back-propagates the loss back to the hidden vector via the face recognition model and the image generation model>Updating hidden vector +.>To->；
Repeating the iterative process for several epochs (for example, iterating 50-100 rounds) to obtain final optimized hidden vectorsThe finally obtained hidden vector +.>The face image obtained by the image generation model (i.e., the generator in fig. 7) is taken as a restored face image.
In the specific implementation, through carrying out effect verification on the method for restoring the face image through the image generation model and the hidden vector, the same Arcface model as the black box scheme is adopted to encode the original face image to obtain the face encoding vector, the face encoding vector is restored to the face image through the method for restoring the face image through the image generation model and the hidden vector, and the restored face image result shows that the optimized hidden vector is more similar to the face encoding vector to be restored due to the fact that the detailed information such as the model related parameters of the face encoding model can be acquired in the white box scene, so that the effect of restoring the face image can be better based on the optimized hidden vector, namely the restored face image is highly similar to the original face image, and the difference is smaller. Meanwhile, compared with the verification result of the black box scheme and the white box scheme, the face image restored based on the image generation model and the hidden vector under the white box scene has better effect compared with the face image restored based on the face inverse coding model under the black box scene.
In this embodiment, a computer device is provided, as shown in fig. 8, including a memory 801, a processor 802, and a computer program stored in the memory and capable of running on the processor, where the processor implements any of the above-mentioned face image restoration methods when executing the computer program.
In particular, the computer device may be a computer terminal, a server or similar computing means.
In the present embodiment, there is provided a computer-readable storage medium storing a computer program for executing the above arbitrary face image restoration method.
In particular, computer-readable storage media, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Disks (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable storage media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.
Based on the same inventive concept, the embodiment of the application also provides a device for restoring the face image, as described in the following embodiment. Because the principle of the face image restoration device for solving the problem is similar to that of the face image restoration method, the implementation of the face image restoration device can refer to the implementation of the face image restoration method, and the repetition is omitted. As used below, the term "unit" or "module" may be a combination of software and/or hardware that implements the intended function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
Fig. 9 is a block diagram of a face image restoration apparatus according to an embodiment of the present application, as shown in fig. 9, including:
the judging module 901 is configured to judge whether the face coding model is a black box or a white box according to related known information of the face coding model;
the first face restoration module 902 is configured to construct a training data set according to a face image and a face coding vector obtained by the face image through the face coding model when the face coding model is a black box, train a machine learning component to obtain a face inverse coding model by adopting the training data set, and input a face coding vector to be restored into the face inverse coding model to obtain a restored face image corresponding to the face coding vector to be restored; and/or the number of the groups of groups,
the second face restoration module 903 is configured to set a hidden vector when the face coding model is a white box, optimize the hidden vector according to a face coding vector to be restored, a model related parameter of the face coding model, and a model related parameter of an image generation model, obtain an optimized hidden vector, and input the optimized hidden vector into the image generation model to obtain a restored face image corresponding to the face coding vector to be restored.
In one embodiment, a first face reduction module includes:
the image processing module is used for obtaining a face coding vector of each face image through the face coding model; calculating an average face image for a plurality of face images of the same person; and constructing the training data set according to the average face image of each person and face coding vectors of a plurality of face images of each person.
In one embodiment, a first face reduction module includes:
and the model training module is used for taking the face coding vector in the training data set as input data of a DCGAN model, taking the face image in the training data set as a label of the face coding vector, and training the DCGAN model to obtain the face inverse coding model.
In one embodiment, the model training module is further configured to add gaussian noise data to input data of the DCGAN model, and/or add a residual connection between every two transposed convolution blocks in the DCGAN model, train the DCGAN model, and evaluate a training result index of the DCGAN model by using an L1 loss function to obtain the face inverse coding model.
In one embodiment, the model training module is further configured to add gaussian noise data with a mean value of 0 and a variance of 0.001 in each dimension to the input data of the DCGAN model.
In one embodiment, the model training module is further configured to add a layer of residual connection between every two transposed convolution blocks in the DCGAN model.
In one embodiment, the second face restoration module is configured to perform an iterative optimization process by taking the hidden vector as a current hidden vector, and loop-executing the following steps, where the current hidden vector after performing the iterative optimization process for a preset number of times is taken as the optimized hidden vector: generating a current face image of the current hidden vector through the image generation model; generating a current face coding vector of the current face image through the face coding model; calculating the loss between the current face coding vector and the second face coding vector to be restored; and reversely transmitting the loss, the model-related parameters of the face coding model and the model-related parameters of the image generation model to the current hidden vector, updating the parameters of the current hidden vector, and taking the current hidden vector after updating the parameters as a new current hidden vector.
The embodiment of the application realizes the following technical effects: the method realizes that the face coding vector can be restored to the face image under the condition that the face coding model is a white box or a black box; meanwhile, when the face coding model is a black box, the face inverse coding model is obtained by training a training data set constructed by face coding vectors obtained by the face coding model based on the face image and the face image, so that the face inverse coding model learns the relevant information of the face coding model for coding the face image into the face coding vectors, and the face coding vectors output by the face coding model can be quickly and accurately restored into the face image based on the face inverse coding model; when the face coding model is a white box, the hidden vector is optimized according to the face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model, so that the optimized hidden vector is more similar or identical to the face coding vector to be restored, and the face coding vector output by the face coding model can be quickly, conveniently and accurately restored to a face image through the image generation model based on the optimized hidden vector.
It will be apparent to those skilled in the art that the modules or steps of the embodiments of the application described above may be implemented in a general purpose computing device, they may be concentrated on a single computing device, or distributed across a network of computing devices, they may alternatively be implemented in program code executable by computing devices, so that they may be stored in a storage device for execution by computing devices, and in some cases, the steps shown or described may be performed in a different order than what is shown or described, or they may be separately fabricated into individual integrated circuit modules, or a plurality of modules or steps in them may be fabricated into a single integrated circuit module. Thus, embodiments of the application are not limited to any specific combination of hardware and software.
The above description is only of the preferred embodiments of the present application and is not intended to limit the present application, and various modifications and variations can be made to the embodiments of the present application by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (8)
1. A method for restoring a face image, comprising:
judging whether the face coding model is a black box or a white box according to the related known information of the face coding model;
when the face coding model is a black box, a training data set is constructed according to a face image and a face coding vector obtained by the face image through the face coding model, a machine learning component is trained by adopting the training data set to obtain a face inverse coding model, and a first face coding vector to be restored is input into the face inverse coding model to obtain a restored face image corresponding to the first face coding vector to be restored;
setting hidden vectors when the face coding model is a white box, optimizing the hidden vectors according to a second face coding vector to be restored, model related parameters of the face coding model and model related parameters of an image generation model to obtain optimized hidden vectors, and inputting the optimized hidden vectors into the image generation model to obtain a restored face image corresponding to the second face coding vector to be restored, wherein the image generation model belongs to a generated countermeasure network architecture;
judging whether the face coding model is a black box or a white box according to the related known information of the face coding model, comprising:
judging whether the related parameters of the face coding model are determined according to the related known information of the face coding model, if so, judging that the face coding model is a white box; if not, judging the face coding model as a black box, wherein the related parameters comprise, but are not limited to, the architecture and/or the weight of the face coding model;
training a machine learning component by adopting the training data set to obtain a face inverse coding model, comprising:
taking the face coding vector in the training data set as input data of a DCGAN model, and taking a face image in the training data set as a label of the face coding vector to train the DCGAN model to obtain the face inverse coding model;
optimizing the hidden vector according to a second face coding vector to be restored, the model related parameters of the face coding model and the model related parameters of the image generation model to obtain an optimized hidden vector, and comprising the following steps:
taking the hidden vector as a current hidden vector, circularly executing the following steps to perform an iterative optimization process, and taking the current hidden vector after performing the iterative optimization process for preset times as the optimized hidden vector:
generating a current face image of the current hidden vector through the image generation model;
generating a current face coding vector of the current face image through the face coding model;
calculating the loss between the current face coding vector and the second face coding vector to be restored;
and reversely transmitting the loss, the model-related parameters of the face coding model and the model-related parameters of the image generation model to the current hidden vector, updating the parameters of the current hidden vector, and taking the current hidden vector after updating the parameters as a new current hidden vector.
2. The method for restoring a face image according to claim 1, wherein constructing a training data set from a face image and a face encoding vector of the face image obtained by the face encoding model comprises:
obtaining a face coding vector of each face image through the face coding model;
calculating an average face image for a plurality of face images of the same person;
and constructing the training data set according to the average face image of each person and face coding vectors of a plurality of face images of each person.
3. The method of face image restoration according to claim 1, wherein training the DCGAN model to obtain the face inverse coding model includes:
and adding Gaussian noise data into the input data of the DCGAN model, and/or adding residual connection between every two transposed convolution blocks in the DCGAN model, training the DCGAN model, and evaluating training result indexes of the DCGAN model by adopting an L1 loss function to obtain the face inverse coding model.
4. A face image restoration method according to claim 3, wherein adding gaussian noise data to the input data of the DCGAN model includes:
and adding Gaussian noise data with the mean value of each dimension of 0 and the variance of 0.001 into the input data of the DCGAN model.
5. A method of face image restoration as claimed in claim 3 wherein adding a residual join between every two transposed convolution blocks in the DCGAN model comprises:
and adding a layer of residual error connection between every two transposed convolution blocks in the DCGAN model.
6. A face image restoration apparatus, comprising:
the judging module is used for judging whether the face coding model is a black box or a white box according to the related known information of the face coding model;
the first face reduction module is used for constructing a training data set according to the face image and the face coding vector obtained by the face coding model when the face coding model is a black box, training a machine learning component by adopting the training data set to obtain a face inverse coding model, and inputting a first face coding vector to be reduced into the face inverse coding model to obtain a reduced face image corresponding to the first face coding vector to be reduced;
the second face reduction module is used for setting hidden vectors when the face coding model is a white box, optimizing the hidden vectors according to the face coding vector to be reduced, model related parameters of the face coding model and model related parameters of an image generation model to obtain optimized hidden vectors, inputting the optimized hidden vectors into the image generation model to obtain a reduced face image corresponding to the face coding vector to be reduced, wherein the image generation model belongs to a generated countermeasure network architecture;
the judging module is used for judging whether the related parameters of the face coding model are determined according to the related known information of the face coding model, if yes, judging that the face coding model is a white box; if not, judging the face coding model as a black box, wherein the related parameters comprise, but are not limited to, the architecture and/or the weight of the face coding model;
a first face reduction module comprising:
the model training module is used for taking the face coding vector in the training data set as input data of a DCGAN model, taking a face image in the training data set as a label of the face coding vector, and training the DCGAN model to obtain the face inverse coding model;
the second face restoration module is used for taking the hidden vector as a current hidden vector, circularly executing the following steps to perform an iterative optimization process, and taking the current hidden vector after performing the iterative optimization process for preset times as the optimized hidden vector: generating a current face image of the current hidden vector through the image generation model; generating a current face coding vector of the current face image through the face coding model; calculating the loss between the current face coding vector and the second face coding vector to be restored; and reversely transmitting the loss, the model-related parameters of the face coding model and the model-related parameters of the image generation model to the current hidden vector, updating the parameters of the current hidden vector, and taking the current hidden vector after updating the parameters as a new current hidden vector.
7. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the method of restoration of a face image according to any one of claims 1 to 5 when the computer program is executed.
8. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that executes the face image restoration method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310672537.1A CN116402916B (en) | 2023-06-08 | 2023-06-08 | Face image restoration method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310672537.1A CN116402916B (en) | 2023-06-08 | 2023-06-08 | Face image restoration method and device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116402916A CN116402916A (en) | 2023-07-07 |
| CN116402916B true CN116402916B (en) | 2023-09-05 |
Family
ID=87018428
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310672537.1A Active CN116402916B (en) | 2023-06-08 | 2023-06-08 | Face image restoration method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116402916B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114332119A (en) * | 2021-12-24 | 2022-04-12 | 深圳数联天下智能科技有限公司 | Method and related device for generating face attribute change image |
| CN114494713A (en) * | 2020-11-13 | 2022-05-13 | 三星电子株式会社 | Method and apparatus for image recognition |
| CN115795406A (en) * | 2022-12-19 | 2023-03-14 | 重庆邮电大学 | Reversible face anonymization processing system |
| CN115908109A (en) * | 2022-11-02 | 2023-04-04 | 阿里巴巴（中国）有限公司 | Facial image stylized model training method, equipment and storage medium |
| CN115936958A (en) * | 2022-04-11 | 2023-04-07 | 南京大学 | Data privacy protection method based on counterstudy |
| CN116168127A (en) * | 2021-11-23 | 2023-05-26 | 广州视源电子科技股份有限公司 | Image processing method, device, computer storage medium and electronic equipment |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11720994B2 (en) * | 2021-05-14 | 2023-08-08 | Lemon Inc. | High-resolution portrait stylization frameworks using a hierarchical variational encoder |
-
2023
- 2023-06-08 CN CN202310672537.1A patent/CN116402916B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114494713A (en) * | 2020-11-13 | 2022-05-13 | 三星电子株式会社 | Method and apparatus for image recognition |
| CN116168127A (en) * | 2021-11-23 | 2023-05-26 | 广州视源电子科技股份有限公司 | Image processing method, device, computer storage medium and electronic equipment |
| CN114332119A (en) * | 2021-12-24 | 2022-04-12 | 深圳数联天下智能科技有限公司 | Method and related device for generating face attribute change image |
| CN115936958A (en) * | 2022-04-11 | 2023-04-07 | 南京大学 | Data privacy protection method based on counterstudy |
| CN115908109A (en) * | 2022-11-02 | 2023-04-04 | 阿里巴巴（中国）有限公司 | Facial image stylized model training method, equipment and storage medium |
| CN115795406A (en) * | 2022-12-19 | 2023-03-14 | 重庆邮电大学 | Reversible face anonymization processing system |
Non-Patent Citations (1)
| Title |
|---|
| Application of an Improved DCGAN for Image Generation;Bingqi Liu 等;Mobile Information Systems;第1-14页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116402916A (en) | 2023-07-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111079532B (en) | Video content description method based on text self-encoder | |
| CN108108854B (en) | Urban road network link prediction method, system and storage medium | |
| CN113240580A (en) | Lightweight image super-resolution reconstruction method based on multi-dimensional knowledge distillation | |
| CN112801881B (en) | High-resolution hyperspectral calculation imaging method, system and medium | |
| CN110071798B (en) | Equivalent key obtaining method and device and computer readable storage medium | |
| KR20220065773A (en) | Converting data samples to normal data | |
| Yang et al. | Towards automatic embedding cost learning for JPEG steganography | |
| KR102605692B1 (en) | Method and system for detecting anomalies in an image to be detected, and method for training restoration model there of | |
| CN114782291B (en) | Training method and device of image generator, electronic equipment and readable storage medium | |
| CN111694974A (en) | Depth hash vehicle image retrieval method integrating attention mechanism | |
| Huai et al. | Zerobn: Learning compact neural networks for latency-critical edge systems | |
| Vašata et al. | Image inpainting using Wasserstein generative adversarial imputation network | |
| CN111814693A (en) | Marine ship identification method based on deep learning | |
| CN113723294B (en) | Data processing method and device and object recognition method and device | |
| Fang et al. | Gifd: A generative gradient inversion method with feature domain optimization | |
| JP6935868B2 (en) | Image recognition device, image recognition method, and program | |
| CN114372505A (en) | Unsupervised network alignment method and system | |
| CN116402916B (en) | Face image restoration method and device, computer equipment and storage medium | |
| CN113591892A (en) | Training data processing method and device | |
| CN112926543A (en) | Image generation method, three-dimensional model generation method, image generation device, three-dimensional model generation device, electronic device, and medium | |
| CN112329918A (en) | Anti-regularization network embedding method based on attention mechanism | |
| CN109523478B (en) | Image descreening method and storage medium | |
| CN115131605A (en) | Structure perception graph comparison learning method based on self-adaptive sub-graph | |
| CN115578574A (en) | Three-dimensional point cloud completion method based on deep learning and topology perception | |
| CN115409997A (en) | Data labeling method, device, equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |