CN113989902A - Method, device and storage medium for identifying shielded face based on feature reconstruction - Google Patents
Method, device and storage medium for identifying shielded face based on feature reconstruction Download PDFInfo
- Publication number
- CN113989902A CN113989902A CN202111344585.5A CN202111344585A CN113989902A CN 113989902 A CN113989902 A CN 113989902A CN 202111344585 A CN202111344585 A CN 202111344585A CN 113989902 A CN113989902 A CN 113989902A
- Authority
- CN
- China
- Prior art keywords
- face
- image
- feature
- reconstruction
- shielding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 238000003860 storage Methods 0.000 title claims abstract description 15
- 238000013507 mapping Methods 0.000 claims abstract description 31
- 230000006870 function Effects 0.000 claims abstract description 19
- 230000000694 effects Effects 0.000 claims abstract description 7
- 238000004422 calculation algorithm Methods 0.000 claims description 19
- 230000015572 biosynthetic process Effects 0.000 claims description 12
- 238000003786 synthesis reaction Methods 0.000 claims description 12
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000012545 processing Methods 0.000 claims description 6
- 239000000126 substance Substances 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004590 computer program Methods 0.000 claims description 5
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 238000003475 lamination Methods 0.000 claims description 4
- 238000013461 design Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 238000012549 training Methods 0.000 description 7
- 238000012795 verification Methods 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 3
- 230000001815 facial effect Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000011084 recovery Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 208000032443 Masked facies Diseases 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000017105 transposition Effects 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Probability & Statistics with Applications (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a method, a device and a storage medium for identifying an occluded face based on feature reconstruction, wherein the method comprises the following steps: respectively inputting the non-shielding face image and the shielding face image into a face feature reconstruction model, reconstructing the non-shielding face image and the shielding face image, and acquiring a mapping relation between the shielding face image and the non-shielding face image; inputting the image characteristics of the shielded human face into a designed small network to obtain a weighted mapping relation, and then outputting the weighted image characteristics as image characteristics to be repaired; and constructing a reconstruction loss function and a style content loss function for optimizing the reconstruction effect. The device comprises: a processor and a memory. The method can better recover the human face shielding part, and cannot influence the characteristics of the non-shielding part.
Description
Technical Field
The invention relates to the field of face recognition, in particular to a method, a device and a storage medium for recognizing an occluded face based on feature reconstruction.
Background
In the field of face recognition, even with the most advanced generic face recognition models, its performance is significantly degraded by occlusion. To solve this problem, related researchers have proposed many methods of occlusion face recognition, which can be summarized into two types: the first method comprises the following steps: the occluded face portion is restored. And the second method comprises the following steps: removing the feature damaged by the occlusion.
In a first class of methods, a representative work is Sparse Representation-based Classification, which recovers occluded face portions using a linear combination of training images. Subsequently, the method is improved by designing the distribution of sparse constraint terms or characterizing structural information. Subsequently, there are also relevant researchers who introduce deep learning to recover occluded faces for recognition. In the prior art, an LSTM (Long-Short Term Memory RNN), which is a cyclic neural network of Long-Short Term Memory model, is proposed to recover the face region blocked in the field and identify the recovered face image. However, the first category of methods generally does not recover the facial portion well, while preserving strong identity information remains very challenging.
The second category of early studies was based on shallow models with hand-made features. Therefore, when occluded, it is very simple to delete the damaged feature. However, the accuracy of these methods is limited by the shallow structure. Researchers have introduced depth models to optimize this problem by designing complex algorithms to remove the damaged depth features. But since the convolutional neural network in depth learning causes the spatial mapping between the input image and the depth features to be opaque, it is difficult to identify the corrupted features even if the occlusion position in the input image is provided. To address this problem, researchers have addressed this problem by adding a mask branch in the middle layer of the convolutional neural network model, which is expected to assign a lower weight to hidden cells that are corrupted by occlusion. But the middle part of the network contains too much extraneous information and no additional supervision is provided for learning guidance, so that it is difficult to reliably identify a broken unit.
Currently, the relatively best approach is to use a binary mask to clean corrupted features from a higher level network where the features are more discriminative. Specifically, the face image is divided into blocks and a dictionary containing all the blocks is learned to map each occlusion block to a corresponding feature mask, and then in the testing phase, it first detects the occluded blocks and then retrieves the corresponding binary mask to apply to the test features. This two-stage approach must rely on external occlusion detectors, resulting in a large network model. Furthermore, to learn the lexicon, the deep face model must be trained separately, making training inefficient and time consuming.
Disclosure of Invention
The invention provides a method, a device and a storage medium for identifying an occluded face based on feature reconstruction, which can better recover a face occluded part without influencing the feature of an unoccluded part, and are described in detail as follows:
in a first aspect, a method for identifying an occluded face based on feature reconstruction includes:
respectively inputting the non-shielding face image and the shielding face image into a face feature reconstruction model, reconstructing the non-shielding face image and the shielding face image, and acquiring a mapping relation between the shielding face image and the non-shielding face image;
inputting the image characteristics of the shielded human face into a designed small network to obtain a weighted mapping relation, and then outputting the weighted image characteristics as image characteristics to be repaired;
and constructing a reconstruction loss function and a style content loss function for optimizing the reconstruction effect.
In one embodiment, the face feature reconstruction model is:
wherein Z isi＝Ei(Ii) Representing hidden vector features, IiRepresenting the ith faceImage, EiRepresentation feature extractor, GiA representation of the image generator is provided,
representing the reconstructed image.
Wherein the face feature reconstruction model comprises two branches, the weights of the two branches are shared,
reconstructing a new face image by the first branch; the second branch reconstructs the occluded face image.
In one embodiment, the designed small network is:
first local feature Z1Sending into a volume of lamination layer to respectively generate two new mapping characteristics Z2And Z3，{Z2,Z3}∈RC ×H×WIs a reaction of Z2,Z3Resetting to C × N, wherein N ═ H × W is the number of pixels;
Z2and Z3Matrix multiplication is carried out, and the information Z of interest is calculated through a softmax layer5∈RN×NR is a feature space;
wherein the content of the first and second substances,
and
is Z1A new mapping characteristic is generated and the new mapping characteristic,
the features obtained for the calculation of the softmax layer.
Further, the method further comprises:
local feature Z1Input to the convolutional layer to generate a new feature map Z4∈RC×H×WAnd reset to RC×NTo Z is paired with4And Z5Performing matrix multiplication operation and setting the operation result as RC×H×WMultiplying the operation result by a proportional parameter lambda, and carrying out element-based summation operation on the characteristics to obtain a final output Z6∈RC×H×W。
In one embodiment, when the second branch, the method further comprises:
and designing an occlusion data synthesis algorithm for recovering and occluding partial areas of the human face to acquire occluded human face images.
Wherein, the occlusion data synthesis algorithm is as follows:
inputting normal image datasets Images and dataset attribute position label text label.
Extracting a label value label in the text; obtaining a specific coordinate value coordinate through a label value label;
changing the pixel value in the coordinate value range; storing the processed image; and circularly processing each image until all the images are processed.
In a second aspect, an occlusion face recognition apparatus based on feature reconstruction, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform the method steps of any of the first aspects.
In a third aspect, a computer-readable storage medium storing a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of any one of the first aspect.
The technical scheme provided by the invention has the beneficial effects that:
(1) the method can better recover the shielded human face characteristics without influencing the unshielded human face characteristics;
(2) the two branch parameters of the core model designed in the method are shared, so that the parameter quantity of the model can be reduced;
(3) the small network designed in the method ensures the semantic consistency of the shielding characteristics and other characteristics of the human face, better generates shielding local information and is suitable for being migrated and used in the shielding human face recognition field;
(4) the loss function adopted by the design of the method is more suitable for processing the problem of shielded face recognition, the convergence of the task can be accelerated, and the performance is effectively improved;
(5) experiments prove that the method is suitable for application and popularization in the field of shielded face recognition.
Drawings
FIG. 1 is a diagram of a face reconstruction model;
FIG. 2 is a structural design diagram of the inside of a small network for face reconstruction;
FIG. 3 is a structural design diagram of a pre-training model of a face reconstruction model;
FIG. 4 is a Bottleneck structural design diagram inside a pre-training model of a face reconstruction model;
FIG. 5 is a flow chart of occlusion data synthesis for an occlusion face recognition method based on feature reconstruction;
FIG. 6 is an exemplary diagram of an occlusion data synthesis for an occlusion face recognition method based on feature reconstruction;
FIG. 7 is an exemplary diagram of an experimental result of a method for identifying an occluded face based on feature reconstruction;
fig. 8 is a schematic structural diagram of an occlusion face recognition device based on feature reconstruction.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention are described in further detail below.
In order to solve the technical problem in the background art, the embodiment of the invention provides a feature reconstruction-based method for completing the identification of the occluded face, and on one hand, the embodiment of the invention recovers the occluded face part. On the other hand, after recovery, embodiments of the present invention will remove poorly recovered features, i.e., damaged features. More importantly, the algorithm model provided by the embodiment of the invention can solve the problem of face recognition of partial occlusion to a certain extent, and experiments prove the effectiveness of the algorithm.
Example 1
The embodiment of the invention provides an occlusion face recognition method based on feature reconstruction, and with reference to the figures 1-3, the method comprises the following steps:
101: respectively inputting the non-shielding face image and the shielding face image into a face feature Reconstruction model FRM (face Reconstruction model), reconstructing the non-shielding face image and the shielding face image, and acquiring a mapping relation between the shielding face image and the non-shielding face image;
wherein, the mapping relation of FRM is defined as f, the input parameters are respectively an occluded Face image and a non-occluded Face image, and are expressed as FaceoccAnd FacecleanThe method aims to: the FRM is made to learn the mapping relationship f, which can be expressed in a formalization as follows:
FRM＝f(Faceocc,Faceclean) (1)
102: inputting the image characteristics of the shielded human face into a designed small network to obtain a weighted mapping relation, and then outputting the weighted image characteristics as image characteristics to be repaired;
wherein the weighted mapping relationship is represented as f', and the parameter is from E in FIG. 12And E1，E2Feature extractor for representing occluded face images, E1Representing a non-shielding face image feature extractor, A representing a small network, and formally expressing a mapping relation f' as follows:
f′＝A(E2(Faceocc),E1(Faceclean)) (2)
103: and constructing a reconstruction loss function and a style content loss function for optimizing the reconstruction effect.
The face feature reconstruction model comprises two branches, the weights of the two branches are shared, and the first branch reconstructs a new face image; the second branch reconstructs the occluded face image.
When the second branch is reached, before step 101, the method for identifying the occlusion face further includes the following steps:
and designing an occlusion data synthesis algorithm for recovering and occluding partial areas of the human face to acquire occluded human face images.
Wherein, the occlusion data synthesis algorithm is as follows:
inputting normal image datasets Images and dataset attribute position label text label.
Extracting a label value label in the text; obtaining a specific coordinate value coordinate through a label value label;
changing the pixel value in the coordinate value range; storing the processed image; and circularly processing each image until all the images are processed.
Further, the small network in the step 102 is: first local feature Z1Sending into a volume of lamination layer to respectively generate two new mapping characteristics Z2And Z3，{Z2,Z3}∈RC×H×WIs a reaction of Z2,Z3Resetting to C × N, wherein N ═ H × W is the number of pixels;
Z2and Z3Matrix multiplication is carried out, and the information Z of interest is calculated through a softmax layer5∈RN×NR is a feature space;
wherein the content of the first and second substances,
and
is Z1A new mapping characteristic is generated and the new mapping characteristic,
the features obtained for the calculation of the softmax layer.
In summary, the embodiment of the present invention can better recover the human face shielding part through the steps 101 to 103, and does not affect the characteristics of the non-shielding part, thereby meeting various requirements in practical applications.
Example 2
The protocol of example 1 is further described below in conjunction with specific examples, experimental data, and as described in detail below:
firstly, designing a human face feature reconstruction model
The face feature reconstruction model of the embodiment of the present invention is designed based on VAE (Variational auto encoder) as a whole, and the embodiment of the present invention innovatively designs two branches, the first branch is a reconstructed normal face model, as shown in fig. 1, the input is a normal face image, the face feature is extracted through a pre-trained face recognition model ResNet (residual error network), and the model structure is as shown in fig. 3:
wherein, the STAGE0 is for preprocessing INPUT, the STAGE1 (STAGE one) includes 3 bottlenecks (Bottleneck layer), and the rest STAGEs 2, 3, and 4 include 4, 6, and 3 bottlenecks, respectively. The bottleeck structure is shown in fig. 4, where C in the structure diagram represents the number of channels of the input map, W represents the width of the input map, CONV represents the convolutional layer, BN is Batch Normalization, and RELU refers to the RELU activation function. The facial features are then input into the generative model, thereby reconstructing a new facial image. The generation model is realized through a series of two-dimensional convolution transposition layers, and each transposition layer is paired with a two-dimensional batch standard layer and an activation function, so that the generation model is used for mapping a hidden vector obtained from a pre-training model to a data space, and a new face image is generated.
In addition, the second branch is used for reconstructing an occlusion face model, and unlike the first branch, the second branch is used for inputting an occlusion face image and outputting the occlusion face image as a reconstructed occlusion face image.
In the testing stage, the shielded face image is input, and the normal face image is output through the face characteristic reconstruction model designed by the embodiment of the invention.
The embodiment of the invention defines the human face characteristic reconstruction process as follows:
wherein Z isi＝Ei(Ii) Representing hidden vector features, IiRepresenting the ith human face image, EiRepresentation feature extractor, GiA representation of the image generator is provided,
representing the reconstructed image.
Namely, the formula (1) can be used for reconstructing a normal face model by the first branch to generate a new face image; and the method can also be used for reconstructing an occlusion face model by the second branch to generate a new occlusion image. When the face image is used for reconstructing a normal face model, a normal face image is input; when used in the second branch to reconstruct the occlusion face model, the input is the occlusion face image. The face feature reconstruction model of the embodiment of the invention comprises two branches, so that the embodiment of the invention shares the weight of the two branches, thereby reducing the parameter quantity of the face feature reconstruction model and accelerating the convergence speed of the face feature reconstruction model. For example: and if the convolution kernel is m × m, the total number of parameters is m × C when weight sharing is performed, wherein C is the number of channels, but weight sharing is not performed, the number of total parameters is W × H × C, wherein W is the width of the image, H is the length, and C is the number of channels, so that the total parameters are changed into times of the parameters for weight sharing.
Second, design of rebuilding small network
In the embodiment of the invention, the purpose of designing the small network is to enable the human face feature reconstruction model to learn the shielded local features, so that the local features are smoothly recovered. Therefore, the focus of the designed small network is on the recovery of occlusion features, and the invention embodiment introduces an Attention mechanism to help achieve this function. As shown in fig. 1, the input of the small network is a Z1 vector, that is, the local feature is taken, and in order to learn a better local feature, the embodiment of the present invention inputs the local feature into the small network, as shown in fig. 2, the core is calculated by performing a dot product first, then performing a softmax weight extraction operation, and then obtaining a weighted feature, so as to serve as an output, where the small network specifically is:
given a local feature Z1∈RC×H×WThe local feature is first sent to a convolution layer to generate two new mapping features Z2And Z3Wherein { Z2,Z3}∈RC×H×WThen Z is2,Z3The reset is C × N, where N ═ H × W is the number of pixels. Finally, the embodiment of the present invention is directed to Z2And Z3Matrix multiplication is carried out, and the information Z of interest is calculated through a softmax layer5∈RN×NR is a feature space; this process is defined by the embodiments of the present invention as:
wherein the content of the first and second substances,
and
is Z1A new mapping characteristic is generated and the new mapping characteristic,
for the features obtained through softmax layer calculation, the formula also shows that the more similar the two location features, the more correlation will likely occur.
In the design of a small network, in addition to the above design, as shown in fig. 2, the embodiment of the present invention further provides a feature Z1Input to the convolutional layer to generate a new feature map Z4∈RC×H×WAnd reset to RC×NThen add Z to it5Performing matrix multiplication operation and setting the operation result as RC×H×WFinally, the embodiment of the invention multiplies the characteristic by a proportional parameter lambda, and carries out summation operation on the characteristic according to elements to obtain the final productOutput Z6∈RC×H×WIn the embodiment of the present invention, the process is defined as:
wherein the content of the first and second substances,
is a local feature of the image and is,
is composed of
And generating a new feature map through the convolutional layer.
The formula also shows that the result feature of each position is the feature weighted sum of all the positions and the original features, so that the small network designed by the embodiment of the invention can help to obtain the correlation between the local information of the face mask and other information and keep the consistency of semantics.
Design of model loss function
In order to optimize the reconstruction effect, the embodiment of the invention designs two loss functions, namely reconstruction loss and style content loss. Specifically, the reconstitution loss is expressed as:
Lrecon＝||f(Irecon)-Ii||2 (7)
wherein, f (I)recon) Representing a reconstruction feature, IiRepresenting the input features. And the method is used for constraining the characteristics of the reconstructed image so that the content of the reconstructed face characteristics is closer to the input face characteristics.
The stylistic content loss is expressed as:
Loss＝αLosscontent+βLossstyle (8)
the loss of the stylistic content is to further constrain the stylistic and reconstruction characteristics of the reconstructed map so that the reconstructed map is more effective. Where α and β are hyper-parameters, the style loss is:
the content loss is:
the embodiment of the invention synthesizes the learning target of the model by using the loss function, namely the smaller the total loss is, the better the learning effect of the model is.
In conclusion, after the model trained by the embodiment of the invention is used for preprocessing the occluded face image, the occluded face recognition precision is improved.
Fourth, design occlusion data synthesis algorithm
In order to make the generalization capability of the model stronger, the embodiment of the invention designs an occlusion data synthesis algorithm to help complete the preparation of the data set. The method comprises the following steps of blocking a partial region of a human face, and recovering the partial region of the blocked human face through a designed algorithm, so as to prove the effectiveness of the model provided by the embodiment of the invention, wherein the synthetic data algorithm is divided into six steps in total, the flow is shown in fig. 5, an example is shown in fig. 6, and the algorithm is as follows:
the first step is as follows: inputting normal image datasets Images and dataset attribute position label text label.
The second step is that: extracting a label value label in the text;
the third step: obtaining a specific coordinate value coordinate through a label value label;
fourthly, changing the pixel value within the coordinate value range;
the fifth step: storing the processed image;
and a sixth step: and circularly processing each image until all the images are processed.
In other words, in practical application, before the second branch inputs the masked face image, the masked face image is processed through the first to sixth steps and then input into the second branch.
The data set attribute position tag text label is a markup file for each attribute position of a human face in a data set, and specific contents of the file are shown in table 1:
TABLE 1
| Img_num | nose_x | nose_y | leftmouth_x | leftmouth_y | rightmouth_x | rightmouth_y |
| 000001.jpg | 196 | 249 | 194 | 271 | 266 | 260 |
In table 1, the first column indicates the number of the image, and the following columns indicate the coordinate positions of the attributes in the image.
In summary, the embodiment of the present invention can better recover the human face shielding part through the above steps, and does not affect the characteristics of the non-shielding part, thereby meeting various requirements in practical applications.
Example 3
The following experiments were performed to verify the feasibility of the protocols of examples 1 and 2, as described in detail below:
as shown in fig. 7, the feasibility verification results of examples 1 and 2 are given. After the model is trained, inputting an untrained shielding face image, selecting mask shielding at the position, then carrying out face recognition, recovering the shielding face by the model, carrying out feature comparison with the image in a face recognition library, accurately finding the image in the library for final face verification, and outputting a result corresponding to the second output result in the figure 7; or the non-occluded face after occlusion recovery can be used for verification, corresponding to the first output result in fig. 7. And in the face verification stage, after the comparison is correct, the face is marked out by a rectangular frame, and corresponding face identity information is displayed.
In addition, for better verification, after training a good model by using the CelebA data set, the embodiment of the present invention selects verification of accuracy of face identification for occlusion on the LFW data set, and the result is shown in table 2 below:
TABLE 2
| Data set type | Accuracy of identification |
| Non-shielding face | About 99.78% |
| With occluded faces, but without using embodiment algorithms | About 82.34 percent |
| With occluded faces, using embodiment Algorithm | About 91.12% |
From the results in table 2, it can be seen that the accuracy of the identification algorithm for the face without occlusion can reach about 99%, and the accuracy is reduced to about 80% after occlusion occurs, whereas the accuracy can be improved by about 10% by repairing the occlusion feature after the method designed by the embodiment of the present invention is used. Theoretically, it is speculated that if the training data is increased, the promotion is more obvious, and the requirements in practical application are completely met.
Example 4
An occlusion face recognition apparatus based on feature reconstruction, referring to fig. 8, the apparatus comprising: a processor 1 and a memory 2, the memory 2 having stored therein program instructions, the processor 1 calling the program instructions stored in the memory 2 to cause the apparatus to perform the method steps described above:
respectively inputting the face features and the shielded face images into a face feature reconstruction model, reconstructing a non-shielded face image and a shielded face image, and acquiring a mapping relation between the shielded face image and the non-shielded face image;
inputting the image characteristics of the shielded human face into a designed small network to obtain a weighted mapping relation, and then outputting the weighted image characteristics as image characteristics to be repaired;
and constructing a reconstruction loss function and a style content loss function for optimizing the reconstruction effect.
The face feature reconstruction model comprises two branches, the weights of the two branches are shared, and the first branch reconstructs a new face image; the second branch reconstructs the occluded face image.
When the second branch, the processor is further configured to: and designing an occlusion data synthesis algorithm for recovering and occluding partial areas of the human face to acquire occluded human face images.
Wherein, the occlusion data synthesis algorithm is as follows:
inputting normal image datasets Images and dataset attribute position label text label.
Extracting a label value label in the text; obtaining a specific coordinate value coordinate through a label value label;
changing the pixel value in the coordinate value range; storing the processed image; and circularly processing each image until all the images are processed.
Further, the small network is: first local feature Z1Sending into a volume of lamination layer to respectively generate two new mapping characteristics Z2And Z3，{Z2,Z3}∈RC×H×WIs a reaction of Z2,Z3Resetting to C × N, wherein N ═ H × W is the number of pixels;
Z2and Z3Matrix multiplication is carried out, and the information Z of interest is calculated through a softmax layer5∈RN×NR is a feature space;
wherein the content of the first and second substances,
and
is Z1A new mapping characteristic is generated and the new mapping characteristic,
the features obtained for the calculation of the softmax layer.
In summary, the embodiment of the present invention can better recover the human face shielding part through the processor and the memory, and does not affect the characteristics of the non-shielding part, thereby meeting various requirements in practical applications.
Example 5
Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, where the storage medium includes a stored program, and when the program runs, the apparatus on which the storage medium is located is controlled to execute the method steps in the foregoing embodiments.
The computer readable storage medium includes, but is not limited to, flash memory, hard disk, solid state disk, and the like.
It should be noted that the descriptions of the readable storage medium in the above embodiments correspond to the descriptions of the method in the embodiments, and the descriptions of the embodiments of the present invention are not repeated here.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. The procedures or functions according to the embodiments of the invention are brought about in whole or in part when the computer program instructions are loaded and executed on a computer.
The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored on or transmitted over a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device, such as a server, a data center, etc., that incorporates one or more of the available media. The usable medium may be a magnetic medium or a semiconductor medium, etc.
In the embodiment of the present invention, except for the specific description of the model of each device, the model of other devices is not limited, as long as the device can perform the above functions.
Those skilled in the art will appreciate that the drawings are only schematic illustrations of preferred embodiments, and the above-described embodiments of the present invention are merely provided for description and do not represent the merits of the embodiments.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (9)
1. An occlusion face recognition method based on feature reconstruction, the method comprising:
respectively inputting the non-shielding face image and the shielding face image into a face feature reconstruction model, reconstructing the non-shielding face image and the shielding face image, and acquiring a mapping relation between the shielding face image and the non-shielding face image;
inputting the image characteristics of the shielded human face into a designed small network to obtain a weighted mapping relation, and then outputting the weighted image characteristics as image characteristics to be repaired;
and constructing a reconstruction loss function and a style content loss function for optimizing the reconstruction effect.
2. The method for recognizing the shielding face based on the feature reconstruction as claimed in claim 1, wherein the face feature reconstruction model is:
wherein Z isi＝Ei(Ii) Representing hidden vector features, IiRepresenting the ith human face image, EiRepresentation feature extractor, GiA representation of the image generator is provided,
representing the reconstructed image.
3. The method for identifying the occlusion face based on the feature reconstruction as claimed in claim 1 or 2, wherein the face feature reconstruction model comprises two branches, the weights of the two branches are shared,
reconstructing a new face image by the first branch; the second branch reconstructs the occluded face image.
4. The method for recognizing the shielding face based on the characteristic reconstruction as claimed in claim 1, wherein the designed small network is as follows:
firstly local feature z1Sending into a volume of lamination layer to respectively generate two new mapping characteristics Z2And Z3，{Z2，Z3}∈RC×H×WIs a reaction of Z2，Z3Resetting to C × N, wherein N ═ H × W is the number of pixels;
Z2and z3Matrix multiplication is carried out, and the information Z of interest is calculated through a softmax layer5∈RN×NR is a feature space;
wherein the content of the first and second substances,
and
is Z1A new mapping characteristic is generated and the new mapping characteristic,
the features obtained for the calculation of the softmax layer.
5. The method for recognizing the shielding face based on the characteristic reconstruction as claimed in claim 1, wherein the method further comprises:
local feature Z1Input to the convolutional layer to generate a new feature map Z4∈RC×H×WAnd reset to RC×NTo Z is paired with4And Z5Performing matrix multiplication operation and setting the operation result as RC×H×WMultiplying the operation result by a proportional parameter lambda, and carrying out element-based summation operation on the characteristics to obtain a final output Z6∈RC×H×W。
6. The method for recognizing the shielding face based on the characteristic reconstruction as claimed in claim 2, wherein when the second branch is selected, the method further comprises:
and designing an occlusion data synthesis algorithm for recovering and occluding partial areas of the human face to acquire occluded human face images.
7. The method according to claim 6, wherein the occlusion data synthesis algorithm is:
inputting normal image datasets Images and dataset attribute position label text label.
Extracting a label value label in the text; obtaining a specific coordinate value coordinate through a label value label;
changing the pixel value in the coordinate value range; storing the processed image; and circularly processing each image until all the images are processed.
8. An occlusion face recognition apparatus based on feature reconstruction, the apparatus comprising: a processor and a memory, the memory having stored therein program instructions, the processor calling upon the program instructions stored in the memory to cause the apparatus to perform the method steps of any of claims 1-7.
9. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program comprising program instructions which, when executed by a processor, cause the processor to carry out the method steps of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111344585.5A CN113989902A (en) | 2021-11-15 | 2021-11-15 | Method, device and storage medium for identifying shielded face based on feature reconstruction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111344585.5A CN113989902A (en) | 2021-11-15 | 2021-11-15 | Method, device and storage medium for identifying shielded face based on feature reconstruction |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113989902A true CN113989902A (en) | 2022-01-28 |
Family
ID=79748385
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111344585.5A Pending CN113989902A (en) | 2021-11-15 | 2021-11-15 | Method, device and storage medium for identifying shielded face based on feature reconstruction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113989902A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109377452A (en) * | 2018-08-31 | 2019-02-22 | 西安电子科技大学 | Facial image restorative procedure based on VAE and production confrontation network |
| CN109886167A (en) * | 2019-02-01 | 2019-06-14 | 中国科学院信息工程研究所 | One kind blocking face identification method and device |
| CN111127308A (en) * | 2019-12-08 | 2020-05-08 | 复旦大学 | Mirror image feature rearrangement repairing method for single sample face recognition under local shielding |
| CN112949565A (en) * | 2021-03-25 | 2021-06-11 | 重庆邮电大学 | Single-sample partially-shielded face recognition method and system based on attention mechanism |
| CN112990052A (en) * | 2021-03-28 | 2021-06-18 | 南京理工大学 | Partially-shielded face recognition method and device based on face restoration |
| CN113378980A (en) * | 2021-07-02 | 2021-09-10 | 西安电子科技大学 | Mask face shading recovery method based on self-adaptive context attention mechanism |
-
2021
- 2021-11-15 CN CN202111344585.5A patent/CN113989902A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109377452A (en) * | 2018-08-31 | 2019-02-22 | 西安电子科技大学 | Facial image restorative procedure based on VAE and production confrontation network |
| CN109886167A (en) * | 2019-02-01 | 2019-06-14 | 中国科学院信息工程研究所 | One kind blocking face identification method and device |
| CN111127308A (en) * | 2019-12-08 | 2020-05-08 | 复旦大学 | Mirror image feature rearrangement repairing method for single sample face recognition under local shielding |
| CN112949565A (en) * | 2021-03-25 | 2021-06-11 | 重庆邮电大学 | Single-sample partially-shielded face recognition method and system based on attention mechanism |
| CN112990052A (en) * | 2021-03-28 | 2021-06-18 | 南京理工大学 | Partially-shielded face recognition method and device based on face restoration |
| CN113378980A (en) * | 2021-07-02 | 2021-09-10 | 西安电子科技大学 | Mask face shading recovery method based on self-adaptive context attention mechanism |
Non-Patent Citations (3)
| Title |
|---|
| LINGXUE SONG, DIHONG GONG, ZHIFENG LI, CHANGSONG LIU, WEI LIU；: "Occlusion Robust Face Recognition Based on Mask LearningWith Pairwise Differential Siamese Network", 《PROCEEDINGS OF THE IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV)》, 17 August 2019 (2019-08-17), pages 773 - 782 * |
| RUOQI SUN, CHEN HUANG, HENGLIANG ZHU & LIZHUANG MA；: "Mask-aware photorealistic facial attribute manipulation", 《 COMPUTATIONAL VISUAL MEDIA》, vol. 7, 28 April 2021 (2021-04-28), pages 363 - 374 * |
| XIAOWEI LIU, KENLI LI & KEQIN LI；: "Attentive Semantic and Perceptual Faces Completion Using Self-attention Generative Adversarial Networks", 《NEURAL PROCESSING LETTERS》, vol. 51, 27 July 2019 (2019-07-27), pages 211 - 229, XP037048822, DOI: 10.1007/s11063-019-10080-2 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108509915B (en) | Method and device for generating face recognition model | |
| Feng et al. | Residual learning for salient object detection | |
| EP3963516B1 (en) | Teaching gan (generative adversarial networks) to generate per-pixel annotation | |
| JP2023549579A (en) | Temporal Bottleneck Attention Architecture for Video Behavior Recognition | |
| US20240161251A1 (en) | Image denoising method and apparatus based on wavelet high-frequency channel synthesis | |
| CN111325695B (en) | Low-dose image enhancement method and system based on multi-dose grade and storage medium | |
| US11410016B2 (en) | Selective performance of deterministic computations for neural networks | |
| CN114694224A (en) | Customer service question and answer method, customer service question and answer device, customer service question and answer equipment, storage medium and computer program product | |
| Amini et al. | A new framework to train autoencoders through non-smooth regularization | |
| US20220101122A1 (en) | Energy-based variational autoencoders | |
| Singh et al. | Generative scatternet hybrid deep learning (g-shdl) network with structural priors for semantic image segmentation | |
| Kroviakov et al. | Sparse convolutional neural network for skull reconstruction | |
| Lyu et al. | FETNet: Feature erasing and transferring network for scene text removal | |
| CN116977624A (en) | Target identification method, system, electronic equipment and medium based on YOLOv7 model | |
| CN113989902A (en) | Method, device and storage medium for identifying shielded face based on feature reconstruction | |
| Bhattacharjya et al. | A genetic algorithm for intelligent imaging from quantum-limited data | |
| CN116977343A (en) | Image processing method, apparatus, device, storage medium, and program product | |
| US20220101145A1 (en) | Training energy-based variational autoencoders | |
| Li et al. | Feature pre-inpainting enhanced transformer for video inpainting | |
| Wang et al. | Mdisn: Learning multiscale deformed implicit fields from single images | |
| Laradji et al. | SSR: Semi-supervised Soft Rasterizer for single-view 2D to 3D Reconstruction | |
| Yang et al. | Scene sketch semantic segmentation with hierarchical Transformer | |
| CN113496228A (en) | Human body semantic segmentation method based on Res2Net, TransUNet and cooperative attention | |
| Gao et al. | CIGNet: Category-and-Intrinsic-Geometry Guided Network for 3D coarse-to-fine reconstruction | |
| CN113222100A (en) | Training method and device of neural network model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |