CN111402401A - Method for acquiring 3D face data, face recognition method and device - Google Patents

Method for acquiring 3D face data, face recognition method and device Download PDF

Info

Publication number
CN111402401A
CN111402401A CN202010174487.0A CN202010174487A CN111402401A CN 111402401 A CN111402401 A CN 111402401A CN 202010174487 A CN202010174487 A CN 202010174487A CN 111402401 A CN111402401 A CN 111402401A
Authority
CN
China
Prior art keywords
face
face data
data
model
recognized
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202010174487.0A
Other languages
Chinese (zh)
Other versions
CN111402401B (en
Inventor
张彦博
李骊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing HJIMI Technology Co Ltd
Original Assignee
Beijing HJIMI Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing HJIMI Technology Co Ltd filed Critical Beijing HJIMI Technology Co Ltd
Priority to CN202010174487.0A priority Critical patent/CN111402401B/en
Publication of CN111402401A publication Critical patent/CN111402401A/en
Application granted granted Critical
Publication of CN111402401B publication Critical patent/CN111402401B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G06V40/165—Detection; Localisation; Normalisation using facial parts and geometric relationships

Abstract

The embodiment of the application discloses a method for acquiring 3D face data, a face recognition method and a face recognition device, and particularly, 2D face data is acquired firstly, and then 3D face data corresponding to the 2D face data is constructed by using a 3D face deformation model. That is, when a large amount of 3D face data needs to be acquired, different 2D face data can be converted by using the 3D face deformation model, thereby generating a large amount of 3D face data. Namely, according to the method provided by the embodiment of the application, the 3D face deformation model is utilized to construct the 3D face data of a large level, so that the problem of lack of the 3D data in the 3D face recognition is solved, and the face recognition precision is improved to a certain extent.

Description

Method for acquiring 3D face data, face recognition method and device
Technical Field
The present application relates to the field of image processing technologies, and in particular, to a method for acquiring 3D face data, a face recognition method, and an apparatus.
Background
With the progress of science and technology, the 2D face recognition technology is mature day by day and is widely applied in life, but under the influence of some illumination, posture or expression, the 2D face recognition capability is reduced, even misjudgment occurs, and the loss of substances or properties of people is caused.
In order to make up for the defects of 2D face recognition, 3D face recognition is carried out at present based on deep learning, 2D face recognition only has information on a plane, and a 3D face not only contains information on the plane, but also contains information in the depth direction, can reflect more feature information of the face, such as features of the concave-convex degree, curvature and the like of the face, and can overcome the defects of 2D face recognition to a certain extent. However, due to the limitation of the application environment during the training process, the acquired 3D data is deficient, so that the generalization capability of the recognition model cannot be improved, and the accuracy of the recognition is affected.
Disclosure of Invention
In view of this, embodiments of the present application provide a method for acquiring 3D face data, a face recognition method and a face recognition device, which generate a face recognition model by constructing abundant and diverse 3D face data, so as to improve the accuracy of face recognition.
In order to solve the above problem, the technical solution provided by the embodiment of the present application is as follows:
in a first aspect of an embodiment of the present application, a method for acquiring 3D face data is provided, including:
acquiring 2D face data;
and converting the 2D face data according to the 3D face deformation model to generate 3D face data.
In one possible implementation, the acquiring 2D face data includes:
for a 2D face image, obtaining each key point of the face to form 2D face data.
In a possible implementation manner, the converting the 2D face data according to the 3D face deformation model to generate 3D face data includes:
acquiring a 3D key point set and a first feature set corresponding to the 3D face deformation model;
acquiring a weight set corresponding to the first feature set according to each key point in the 2D face data, the 3D key point set and the first feature set;
obtaining 3D face data corresponding to the 2D face data by using the weight set, a second feature set corresponding to the 3D face deformation model and a standard vector set, wherein the weight number in the weight set is consistent with the feature number in the second feature set; the feature dimension in the second feature set is larger than that in the first feature set, and the first feature set is obtained by performing dimension reduction conversion on the second feature set.
In one possible implementation manner, the first feature set includes a first ID feature set and a first expression feature set, and the weight set includes an ID weight set and an expression weight set; the second feature set includes a second ID feature set and a second expressive feature set.
In one possible implementation, the method further includes:
and mapping the 3D face data to a plane by using a transformation matrix to obtain target face data.
In one possible implementation, the method further includes: (ii) a
And training model parameters of an initial deep learning model according to the target face data and the label corresponding to the target face data to generate a face recognition model.
In a possible implementation manner, the mapping the 3D face data to a plane by using a transformation matrix to obtain target face data includes:
carrying out normalization processing on the 3D face data to obtain processed 3D face data;
and mapping the processed 3D face data to a plane by using a transformation matrix to obtain target face data.
In one possible implementation, the initial deep learning model is a Mobilenet model or a ShuffleNet model.
In a second aspect of the embodiments of the present application, a face recognition method is provided, where the method includes:
acquiring a face image to be recognized, and extracting 2D face data to be recognized from the face image to be recognized;
converting the 2D face data to be recognized according to the 3D face deformation model to generate 3D face data to be recognized;
mapping the 3D face data to be recognized to a plane by using a transformation matrix to obtain target face data to be recognized;
and inputting the target face data to be recognized into a face recognition model generated by pre-training to obtain a recognition result, wherein the face recognition model is generated by training according to the generation method of the face recognition model of the first aspect.
In a third aspect of the embodiments of the present application, there is provided an apparatus for acquiring 3D face data, the apparatus including:
a first acquisition unit for acquiring 2D face data;
and the first generation unit is used for converting the 2D face data according to the 3D face deformation model to generate 3D face data.
In a fourth aspect of the embodiments of the present application, there is provided a face recognition apparatus, including:
the second acquisition unit is used for acquiring a face image to be recognized and extracting 2D face data to be recognized from the face image to be recognized;
the third generation unit is used for converting the 2D face data to be recognized according to the 3D face deformation model to generate 3D face data to be recognized;
the second mapping unit is used for mapping the 3D face data to be recognized to a plane by using a transformation matrix to obtain target face data to be recognized;
and the recognition unit is used for inputting the target face data to be recognized into a face recognition model generated by pre-training to obtain a recognition result, wherein the face recognition model is generated by training according to the generation method of the face recognition model in the first aspect.
Therefore, the embodiment of the application has the following beneficial effects:
according to the embodiment of the application, the 2D face data is firstly obtained, and then the 3D face data corresponding to the 2D face data is constructed by using the 3D face deformation model. That is, when a large amount of 3D face data needs to be acquired, different 2D face data can be converted by using the 3D face deformation model, thereby generating a large amount of 3D face data. Namely, according to the method provided by the embodiment of the application, the 3D face deformation model is utilized to construct the 3D face data of a large level, so that the problem of lack of the 3D data in the 3D face recognition is solved, and the face recognition precision is improved to a certain extent.
In addition, after a large number of 3D face data sets are obtained by the above method, the 3D face data sets may also be used to train a face recognition model, specifically, a transformation matrix is used to map the 3D face data sets to a plane to obtain a target face data set. And finally, training the initial deep learning model by using the face data in the target face data set and the labels corresponding to the face data to obtain a face recognition model. Namely, by the model generation method provided by the embodiment of the application, the model training is performed by using the 3D face data with more face features, so that the trained face recognition model can extract more comprehensive and effective face features, and the accuracy of face recognition is further improved.
Drawings
Fig. 1 is a flowchart of a method for acquiring 3D face data according to an embodiment of the present disclosure;
fig. 2a is a schematic view of a face detection provided in an embodiment of the present application;
FIG. 2b is a schematic diagram of a key point extraction method and system provided by an embodiment of the present application;
fig. 3 is a flowchart of a face recognition model generation method according to an embodiment of the present application;
fig. 4a is a schematic plane diagram illustrating mapping of face data of an x coordinate in 3D face data according to an embodiment of the present application;
fig. 4b is a schematic plane diagram illustrating that the face data of the y coordinate in the 3D face data is mapped according to the embodiment of the present application;
fig. 4c is a schematic plane diagram illustrating mapping of z-coordinate face data in 3D face data according to an embodiment of the present application;
fig. 4D is a schematic plane diagram illustrating mapping of 3D face data according to an embodiment of the present disclosure;
fig. 5 is a flowchart of a face recognition method according to an embodiment of the present application;
fig. 6 is a frame diagram of a face recognition model generation provided in an embodiment of the present application;
fig. 7 is a structural diagram of an apparatus for acquiring 3D face data according to an embodiment of the present disclosure;
fig. 8 is a structural diagram of a face recognition device according to an embodiment of the present application.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present application more comprehensible, embodiments accompanying the drawings are described in detail below.
In order to facilitate understanding of the technical solutions provided in the embodiments of the present application, the following description will first describe the techniques related to the embodiments of the present application.
The face recognition technology is based on the face characteristics of people, firstly, whether a face exists in an input face image or video stream is judged, and if the face exists, the position and the size of each face and the position information of each main facial organ are further given. And then, further extracting the identity characteristics implied in each face according to the position information, and comparing the identity characteristics with the known faces so as to identify the identity of each face.
The existing face recognition technology is developed based on 2D images, but the 2D images contain limited information, and under certain specific conditions, such as dark illumination, large posture and exaggerated expression, the 2D face recognition technology cannot play good performance and cannot meet the expectations of people. The 3D face not only contains information on a plane, but also contains information in a depth direction, and can reflect more feature information of the face, such as features of the concave-convex degree, curvature, and the like of the face. However, 3D face recognition based on deep learning faces the lack of 3D data, and how to acquire a large amount of 3D data is a technical problem to be solved urgently.
Based on this, the embodiment of the present application provides a method for obtaining 3D face data, which includes obtaining 2D face data, and then converting the 2D face data by using a 3D face deformation model to obtain corresponding 3D face data. When 3D face recognition based on deep learning is carried out, the generated 3D face data can be used for training to obtain a face recognition model, and therefore the accuracy of face recognition is improved.
Referring to fig. 1, which is a flowchart of a method for acquiring 3D face data according to an embodiment of the present application, as shown in fig. 1, the method may include:
s101: 2D face data is acquired.
In this embodiment, to construct 3D face data, first, 2D face data is obtained, where the 2D face data is data extracted from a 2D face image.
In a specific implementation, since the CASIA-WebFace dataset includes 50 million face images with different poses and environments, it can provide various 2D face images, and therefore, a required 2D face image can be extracted from the dataset. And acquiring each key point in the face of each obtained face image so as to form a group of 2D face data.
The method for acquiring each key point in the face from each face image can be acquired in the following manner: firstly, detecting the position of a human face in a human face image by using a human face detector, and marking by using a human face frame, as shown in fig. 2 a; then, a key point detector is used to detect key points of the face in the face frame, so as to obtain each key point in the current face image, as shown in fig. 2 b. In practical application, the face detector and the key point detector are trained in advance, the number of key points detected by the key point detector is related to the type of a subsequently used 3D face deformation model, and the number of key points corresponding to different types of 3D face deformation models is different. For example, when the 3D Face deformation Model is a Basel Face Model (BFM), the number of keypoints detected by the keypoint detector is 68.
S102: and converting the 2D face data according to the 3D face deformation model to generate 3D face data.
In this embodiment, after the 2D face data is acquired, the 3D face deformation model is used to convert the 2D face data, and the 3D face data is acquired. That is, in the present embodiment, different 3D face data are obtained by using the 3D face deformation model and changing deformation parameters in the deformation model, so that a large amount of 3D face data are generated. The number of deformation parameters (weights) corresponding to different types of 3D face deformation models is different, the specific working principle is similar, and for convenience of understanding, the present embodiment will be described by taking a BFM model as an example, where the BFM model includes standard 3D face features and 2D face features, and the deformation parameters can be calculated according to the standard 3D face features and the obtained 2D face data, so as to obtain a model in which the 2D face data is converted into 3D face data, and then generate the 3D face data by using the deformation parameters and the 2D face features.
That is, the process of converting the 2D face data according to the 3D face deformation model and generating the 3D face data includes:
1) and aiming at any group of 2D face data, acquiring a 3D key point set and a first feature set corresponding to the 3D face deformation model.
In this embodiment, for each group of 2D face data, a 3D key point set and a first feature set corresponding to the 3D face deformation model are obtained, so as to calculate a deformation parameter (weight) of the 3D face deformation model by using the 3D key point set, the first feature set, and the group of 2D face data. Specifically, the features in the first feature set may include a first ID feature set and a first expression feature set, where the ID features in the first ID feature set are used to represent ID information of a human face, and the expression features in the first expression feature set are used to represent expression information of the human face. The 3D key point set includes 3D key points of a standard face, and specifically, the number of key points in the 3D key point set, the number of ID features in the first ID feature set, and the number of expression features in the first expression feature set are all consistent with the number of key points included in the set of 2D face data.
For example, the 3D face deformation model isA BFM model comprising 68 3D key points of a standard face, i.e. a set s of 3D key points0＝{L0,L1,...,L68}. The first ID feature set in the first feature set is
Figure BDA0002410307560000081
(s denotes that the feature is an ID feature, M denotes the number of ID features), the first expression feature set is
Figure BDA0002410307560000082
(e indicates that the feature is an expressive feature and N indicates the number of expressive features).
2) And acquiring a weight set corresponding to the first feature set according to each key point in the 2D face data, the 3D key point set and the first feature set.
When each piece of information (a 3D key point set and a first feature set) of the 3D face deformation model is obtained, a weight set (a deformation parameter set) corresponding to the first feature set is calculated by using the information and the set of 2D face data. It should be noted that, when the first feature set includes both the first ID feature set and the first expression feature set, the calculated weight set includes both the ID weight set and the expression weight set. For a specific calculation, see the following formula:
firstly, a transformation matrix R from a 3D key point to a 2D key point is calculated by using a formula (1), and the standard 3D key point (taking a BFM model as an example) is converted into homogeneous coordinates:
Figure BDA0002410307560000083
the conversion of real-time 2D key points (taking BFM model as an example) into homogeneous coordinates is:
Figure BDA0002410307560000084
the following transformation equation is constructed:
RP＝p (1)
the transformation matrix R can be obtained by solving the transformation equation.
After the transformation matrix R is obtained through calculation, a weight set, namely a deformation parameter set, is obtained through calculation by using a formula (2):
Figure BDA0002410307560000085
wherein s is0Representing a 3D key point set corresponding to the 3D face deformation model, S0Representing a set of 2D face data, the set of 2D face data comprising a plurality of keypoints, fi sIndicating ID feature, fi eRepresenting an expressive feature, ui sRepresenting ID feature weight, vi eRepresenting the expressive feature weights.
After the equation set is established, the linear equation set can be solved through QR decomposition to obtain an ID weight set and an expression weight set. Taking BFM model as an example, the obtained ID weight set is
Figure BDA0002410307560000091
Expression weight set
Figure BDA0002410307560000092
That is, the deformation parameters are obtained by solving a system of linear equations.
It should be noted that, for any 3D face deformation model, the deformation parameters corresponding to the 3D face deformation model can be obtained through formulas (1) and (2). In addition, the deformation parameters are related to the currently used 2D face data, and when different 2D face data are subjected to deformation parameter calculation using the same 3D face deformation model, the obtained deformation parameters may be different.
3) And obtaining the 3D face data by using the weight set, the second feature set corresponding to the 3D face deformation model and the standard vector set.
And after the weight set is obtained through the formula, obtaining a second feature set and a standard vector set corresponding to the 3D face deformation model, and obtaining 3D face data by using the weight set, the second feature set and the standard vector set. The number of the weights in the weight set is consistent with the number of the features in the second feature set, the feature dimension in the second feature set is larger than that in the first feature set, and the first feature set is obtained by performing dimension reduction conversion on the second feature set. Namely, the second feature set is an original feature set, the first feature set is a feature set obtained by dimension reduction and conversion of the original feature set, and the operation complexity is reduced and the operation efficiency is improved through dimension reduction operation. The standard vector set represents feature vectors corresponding to standard 3D face models (front face, neutral expression).
It will be understood that when the first feature set includes the first ID feature set and the second expression feature set, the second feature set will also include the second ID feature set and the second expression feature set, and the standard vector set is a data set of a standard face corresponding to the 3D face deformation model.
In a specific implementation, the 3D face data can be obtained by calculation using the following formula:
Figure BDA0002410307560000101
wherein F represents a group of 3D face data, S represents a standard vector set, Fi sRepresenting a second ID feature, Fi eRepresenting a second expressive feature. Taking the BFM model as an example, the second ID feature set is
Figure BDA0002410307560000102
The second expression feature set is
Figure BDA0002410307560000103
It can be understood that for each acquired face image, the corresponding transformation matrix and 3D face data can be obtained through the above equations (1) - (3).
4) And 3D face data corresponding to each group of 2D face data respectively form a 3D face data set.
And after the 3D face data corresponding to each group of 2D face data is respectively obtained through the formula, all the obtained 3D face data are combined to obtain a 3D face data set.
As can be seen from the above description, the 2D face data is converted into the 3D face data by using the 3D face deformation model, thereby increasing the number of 3D face data.
It should be noted that the face 3D data obtained by the present embodiment may be used for not only the subsequent training of the face recognition model, but also as test data to test the accuracy of the generated face recognition model.
In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, a face recognition model generation process provided by the embodiments of the present application will be described below with reference to the accompanying drawings.
Referring to fig. 3, which is a flowchart of a method for generating a face recognition model according to an embodiment of the present application, as shown in fig. 3, the method may include:
s301: a 2D face data set is obtained.
In this embodiment, to construct a 3D face data set, a 2D face data set is first obtained, where the 2D face data set includes multiple sets of 2D face data, and each set of 2D face data is data extracted from one 2D face image.
For specific implementation of acquiring 2D face data, reference may be made to corresponding contents in S101.
S302: and converting each group of 2D face data in the 2D face data set according to the 3D face deformation model to generate a 3D face data set.
In this embodiment, after the 2D face data set is obtained, each group of 2D face data in the 2D face data set is converted by using the 3D face deformation model to obtain a 3D face data set, where the 3D face data set includes a plurality of groups of 3D face data.
See S102 for a specific implementation of converting each set of 2D face data in the set of 2D face data into 3D face data.
S303: and mapping each group of 3D face data in the 3D face data set to a plane by using the transformation matrix to obtain a target face data set.
Because the deep learning network model can only learn and train plane images, after a 3D face data set is obtained, the 3D face data set is mapped to a 2D plane by using a transformation matrix to obtain a target face data set. It should be noted that the principle of mapping by using the transformation matrix is to map the data of 3 planes in a group of 3D face data to 2D planes respectively to obtain 3 plane maps, and still retain 3D feature information, and then fuse the 3 plane maps to obtain target face data. That is, the target face data set includes a plurality of pieces of face image data. The transformation matrix corresponds to each group of 3D face data one to one, that is, the transformation matrices corresponding to different groups of 3D face data may be different.
For example, a set of 3D face data is { x }1,x2,...,xn,y1,y2,...,yn,z1,z2,...znAnd mapping all the x, y and z coordinates to a plane by using a transformation matrix to obtain three plane mapping maps, as shown in fig. 4a-4c, and then combining the three plane mapping maps to obtain a target training map, as shown in fig. 4 d. Where 4a, 4b, 4c are the result of mapping vertices x, y, z in the 3D face data to a 2D plane, respectively.
In practical application, for convenience of network training and improvement of efficiency of subsequent model training, aiming at any group of 3D face data, before the 3D face data is mapped to a 2D plane, normalization processing is carried out on the 3D face data to obtain processed 3D face data; mapping the processed 3D face data to a plane by using a transformation matrix to obtain target face data; each target face data constitutes a target face data set. Specifically, the 3D face data may be normalized to between [ -11 ].
S104: and training the model parameters of the initial deep learning model according to the target face data in the target face data set and the labels corresponding to the target face data to generate a face recognition model.
Through the mapping, a target face data set required by a training network model can be obtained, then, model parameters of the initial deep learning model are trained by using each group of target face data in the target face data set and labels corresponding to the target group of face data, and a face recognition model is generated. In specific implementation, to ensure the real-time performance of training, the deep learning model usually adopted is a lightweight model, such as a Mobilenet model or a ShuffleNet model.
It can be known from the description that a 2D face data set is obtained first, and a large number of 3D face data sets are constructed using a face deformation model. Then, the 3D face data set is mapped to a plane by using the transformation matrix, and a target face data set is obtained. And finally, training the initial deep learning model by using the face data in the target face data set and the labels corresponding to the face data to obtain a face recognition model. Namely, according to the model generation method provided by the embodiment of the application, on one hand, 3D face data of a large level is constructed by using a 3D face deformation model, so that the problem of lack of 3D data in 3D face recognition is solved, and the accuracy of the face recognition is improved to a certain extent; on the other hand, more comprehensive and more effective facial features of the human face can be extracted by utilizing the deep learning method, and the accuracy of the human face recognition is further improved.
Based on the embodiment, the face recognition model can be generated, and in practical application, the face recognition model can be used for face recognition.
Referring to fig. 5, which is a flowchart of a face recognition method provided in the embodiment of the present application, as shown in fig. 5, the method may include:
s501: and acquiring a face image to be recognized, and extracting 2D face data to be recognized from the face image to be recognized.
S502: and converting the 2D face data to be recognized according to the 3D face deformation model to generate the 3D face data to be recognized.
In practical application, after a face image to be recognized is obtained, firstly, 2D face data to be recognized is extracted from the face image to be recognized, and the 2D face data to be recognized is converted according to a 3D face deformation model to generate 3D face data to be recognized. It should be noted that the 3D face deformation model used in this embodiment is consistent with the 3D face deformation model used for training the face recognition model, for example, the BFM model used in training is also used in this embodiment to convert the 2D face data to be recognized.
Specifically, in the process of extracting the 2D face data to be recognized from the face image to be recognized, the face position in the face image to be recognized may be detected by using a pre-trained face detector, and then the key point of the face may be detected by using a key point detector, so as to obtain the 2D face data to be recognized. And then, converting the 2D face data to be recognized by using the 3D face deformation model, wherein the process of obtaining the 3D face data to be recognized can refer to the above formulas (1) - (3), and the description of this embodiment is omitted here.
It should be emphasized that the face image to be recognized in this embodiment is any one of the face images acquired during training.
S503: and mapping the 3D face data to be recognized to a plane by using the transformation matrix to obtain the target face data to be recognized.
Specifically, the specific implementation of S503 can be referred to the corresponding content of S103.
S504: and inputting the target face data to be recognized into a face recognition model generated by pre-training to obtain a recognition result.
After the target data to be recognized is obtained, the target data to be recognized is input into the face recognition model generated in the above embodiment, so as to obtain a recognition result. The face recognition model is generated by training the generation method of the face recognition model.
Therefore, in practical application, a face image to be recognized is obtained, 2D face data to be recognized is extracted from the face image, the 2D face data to be recognized is converted into 3D face data through the 3D face deformation model, the 3D face data is mapped to a plane through the transformation matrix to obtain target data to be recognized, the target data to be recognized is input into the generated face recognition model, and accuracy of face recognition is improved through obtaining more characteristic information.
For the convenience of understanding the overall implementation process of the embodiment of the present application, reference is made to a frame diagram of generating a face recognition model shown in fig. 6, which is illustrated by taking a face image as an example. Firstly, a face image is obtained, and a face detector is used for determining the face position from the face image. Then, each key point is extracted from the position of the human face by using a key point detector, thereby constituting 2D face data. And then, calculating a deformation parameter (weight set) by using the BFM model and the 2D face data, and determining the 3D face data by using the deformation parameter and a 3D standard vector set in the BFM model. After the 3D face data are obtained, the 3D face data are subjected to normalization processing, the processed 3D face data are mapped to a 2D plane, target face data used for training are obtained, and then the target face data and a label corresponding to the target face data are used for training to obtain a face recognition model.
That is, in the application, an optimized linear equation set can be established according to key points on the face image and key points on a BFM standard face by means of a BFM face deformation model and the face image, an ID weight parameter and an expression weight parameter of the current face can be obtained by solving the equation, and a 3D data set of the current face image is obtained by performing linear weighting by using the weight parameter, the ID feature in the BFM and the expression feature. That is, a larger level of 3D data set can be constructed by this method; further, the preconditioning mechanism of the present application; obtaining a 3D-to-2D conversion matrix by means of a BFM (bidirectional Forwarding model), a face detection technology and a key point detection technology, normalizing key points of the 3D face model, mapping a 3D data set to a 2D plane by applying the conversion matrix, and combining the 3D data set into a 3-channel face image as a face data training network model.
Therefore, compared with the traditional method, the deep learning-based method has more comprehensive and effective feature extraction; 3D data is the most deficient of 3D face recognition based on deep learning, and a 3D data set with a larger level is constructed, so that the face recognition precision can be improved to a certain degree; the method directly normalizes the coordinates of the 3D face respectively and maps the coordinates to the 2D plane, so that the major 3D information is reserved, and the complexity of operation is reduced; in addition, the application can adopt a lightweight network model, and can ensure real-time identification.
Based on the above method embodiments, the present application embodiment provides a face recognition model generation apparatus and a face recognition apparatus, which will be described below with reference to the accompanying drawings.
Referring to fig. 7, which is a block diagram of an apparatus for acquiring 3D face data according to an embodiment of the present disclosure, as shown in fig. 7, the apparatus may include:
a first obtaining unit 701, configured to obtain 2D face data;
a first generating unit 702, configured to convert the 2D face data according to a 3D face deformation model to generate 3D face data.
In a possible implementation manner, the first obtaining unit is specifically configured to obtain, for a 2D face image, each key point of a face to constitute 2D face data.
In one possible implementation manner, the first generating unit includes:
the first obtaining subunit is used for obtaining a 3D key point set and a first feature set corresponding to the 3D face deformation model;
a second obtaining subunit, configured to obtain, according to each keypoint in the 2D face data, the 3D keypoint set, and the first feature set, a weight set corresponding to the first feature set;
a third obtaining subunit, configured to obtain, by using the weight set, a second feature set corresponding to the 3D face deformation model, and a standard vector set, 3D face data corresponding to the 2D face data, where the number of weights in the weight set is consistent with the number of features in the second feature set; the feature dimension in the second feature set is larger than that in the first feature set, and the first feature set is obtained by performing dimension reduction conversion on the second feature set.
In one possible implementation manner, the first feature set includes a first ID feature set and a first expression feature set, and the weight set includes an ID weight set and an expression weight set; the second feature set includes a second ID feature set and a second expressive feature set.
In one possible implementation, the apparatus further includes:
and the first mapping unit is used for mapping the 3D face data to a plane by using a transformation matrix to obtain target face data.
In one possible implementation, the apparatus further includes:
and the second generation unit is used for training the model parameters of the initial deep learning model according to the target face data and the label corresponding to the target face data to generate a face recognition model.
In a possible implementation manner, the first mapping unit includes:
the processing subunit is used for carrying out normalization processing on the 3D face data to obtain processed 3D face data;
and the mapping subunit is used for mapping the processed 3D face data to a plane by using a transformation matrix to obtain target face data.
In one possible implementation, the initial deep learning model is a Mobilenet model or a ShuffleNet model.
It should be noted that, implementation of each unit in this embodiment may refer to the foregoing method embodiment, and this example is not described herein again.
Referring to fig. 8, which is a structural diagram of a face recognition apparatus provided in an embodiment of the present application, the apparatus includes:
a second obtaining unit 801, configured to obtain a face image to be recognized, and extract 2D data to be recognized from the face image to be recognized;
a third generating unit 802, configured to convert the to-be-recognized 2D data according to the 3D face deformation model, and generate to-be-recognized 3D data;
a second mapping unit 803, configured to map the to-be-recognized 3D face data to a plane by using a transformation matrix, to obtain target to-be-recognized data;
the recognition unit 804 is configured to input the target data to be recognized into a face recognition model generated by pre-training to obtain a recognition result, where the face recognition model is generated by training using the above-mentioned generation method of the face recognition model.
It should be noted that, implementation of each unit in this embodiment may refer to the above method embodiment, and this embodiment is not described herein again.
In addition, an embodiment of the present application further provides an apparatus for acquiring 3D face data, including: a processor and a memory; the memory for storing instructions or computer programs; the processor, configured to execute the instructions or the computer program, performs the method of fig. 1.
The embodiment of the present application further provides an apparatus for face recognition, which includes: a processor and a memory; the memory for storing instructions or computer programs; the processor, configured to execute the instructions or the computer program, executes the method of fig. 5.
The present application provides a computer-readable storage medium, which is characterized by comprising instructions or a computer program, and when the instructions or the computer program is executed on a computer, the computer is caused to execute the method described in the above fig. 1, fig. 2 and fig. 5.
Therefore, by the device, the 2D face data can be firstly obtained, and then the 3D face data corresponding to the 2D face data is constructed by utilizing the 3D face deformation model. That is, when a large amount of 3D face data needs to be acquired, different 2D face data can be converted by using the 3D face deformation model, thereby generating a large amount of 3D face data. Namely, according to the method provided by the embodiment of the application, the 3D face deformation model is utilized to construct the 3D face data of a large level, so that the problem of lack of the 3D data in the 3D face recognition is solved, and the face recognition precision is improved to a certain extent.
In addition, after a large number of 3D face data sets are obtained by the above method, the 3D face data sets may also be used to train a face recognition model, specifically, a transformation matrix is used to map the 3D face data sets to a plane to obtain a target face data set. And finally, training the initial deep learning model by using the face data in the target face data set and the labels corresponding to the face data to obtain a face recognition model. Namely, by the model generation method provided by the embodiment of the application, the model training is performed by using the 3D face data with more face features, so that the trained face recognition model can extract more comprehensive and effective face features, and the accuracy of face recognition is further improved.
It should be noted that, in the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments may be referred to each other. For the system or the device disclosed by the embodiment, the description is simple because the system or the device corresponds to the method disclosed by the embodiment, and the relevant points can be referred to the method part for description.
It should be understood that in the present application, "at least one" means one or more, "a plurality" means two or more. "and/or" for describing an association relationship of associated objects, indicating that there may be three relationships, e.g., "a and/or B" may indicate: only A, only B and both A and B are present, wherein A and B may be singular or plural. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship. "at least one of the following" or similar expressions refer to any combination of these items, including any combination of single item(s) or plural items. For example, at least one (one) of a, b, or c, may represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, c may be single or plural.
It is further noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims (11)

1. A method for acquiring 3D face data, the method comprising:
acquiring 2D face data;
and converting the 2D face data according to the 3D face deformation model to generate 3D face data.
2. The method of claim 1, wherein the obtaining 2D face data comprises:
for a 2D face image, obtaining each key point of the face to form 2D face data.
3. The method of claim 1, wherein the converting the 2D face data according to the 3D face deformation model to generate 3D face data comprises:
acquiring a 3D key point set and a first feature set corresponding to the 3D face deformation model;
acquiring a weight set corresponding to the first feature set according to each key point in the 2D face data, the 3D key point set and the first feature set;
obtaining 3D face data corresponding to the 2D face data by using the weight set, a second feature set corresponding to the 3D face deformation model and a standard vector set, wherein the weight number in the weight set is consistent with the feature number in the second feature set; the feature dimension in the second feature set is larger than that in the first feature set, and the first feature set is obtained by performing dimension reduction conversion on the second feature set.
4. The method of claim 3, wherein the first feature set comprises a first ID feature set and a first expression feature set, and wherein the weight set comprises an ID weight set and an expression weight set; the second feature set includes a second ID feature set and a second expressive feature set.
5. The method according to any one of claims 1-4, further comprising:
and mapping the 3D face data to a plane by using a transformation matrix to obtain target face data.
6. The method of claim 5, further comprising:
and training model parameters of an initial deep learning model according to the target face data and the label corresponding to the target face data to generate a face recognition model.
7. The method according to claim 5 or 6, wherein the mapping the 3D face data to a plane by using a transformation matrix to obtain target face data comprises:
carrying out normalization processing on the 3D face data to obtain processed 3D face data;
and mapping the processed 3D face data to a plane by using a transformation matrix to obtain target face data.
8. The method of claim 6, wherein the initial deep learning model is a Mobilene model or a ShuffleNet model.
9. A face recognition method, comprising:
acquiring a face image to be recognized, and extracting 2D face data to be recognized from the face image to be recognized;
converting the 2D face data to be recognized according to the 3D face deformation model to generate 3D face data to be recognized;
mapping the 3D face data to be recognized to a plane by using a transformation matrix to obtain target face data to be recognized;
inputting the target face data to be recognized into a face recognition model generated by pre-training to obtain a recognition result, wherein the face recognition model is generated by training according to the generation method of the face recognition model of any one of claims 6 to 8.
10. An apparatus for acquiring 3D face data, the apparatus comprising:
a first acquisition unit for acquiring 2D face data;
and the first generation unit is used for converting the 2D face data according to the 3D face deformation model to generate 3D face data.
11. An apparatus for face recognition, the apparatus comprising:
the second acquisition unit is used for acquiring a face image to be recognized and extracting 2D face data to be recognized from the face image to be recognized;
the third generation unit is used for converting the 2D face data to be recognized according to the 3D face deformation model to generate 3D face data to be recognized;
the second mapping unit is used for mapping the 3D face data to be recognized to a plane by using a transformation matrix to obtain target face data to be recognized;
a recognition unit, configured to input the target face data to be recognized into a face recognition model generated by pre-training to obtain a recognition result, where the face recognition model is generated by training according to the generation method of the face recognition model according to any one of claims 6 to 8.
CN202010174487.0A 2020-03-13 2020-03-13 Method for acquiring 3D face data, face recognition method and device Active CN111402401B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010174487.0A CN111402401B (en) 2020-03-13 2020-03-13 Method for acquiring 3D face data, face recognition method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010174487.0A CN111402401B (en) 2020-03-13 2020-03-13 Method for acquiring 3D face data, face recognition method and device

Publications (2)

Publication Number Publication Date
CN111402401A true CN111402401A (en) 2020-07-10
CN111402401B CN111402401B (en) 2023-08-18

Family

ID=71413920

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010174487.0A Active CN111402401B (en) 2020-03-13 2020-03-13 Method for acquiring 3D face data, face recognition method and device

Country Status (1)

Country Link
CN (1) CN111402401B (en)

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2008200301A1 (en) * 2008-01-22 2009-08-06 The University Of Western Australia Image recognition
CN102999942A (en) * 2012-12-13 2013-03-27 清华大学 Three-dimensional face reconstruction method
US20140355843A1 (en) * 2011-12-21 2014-12-04 Feipeng Da 3d face recognition method based on intermediate frequency information in geometric image
WO2016188318A1 (en) * 2015-05-22 2016-12-01 腾讯科技（深圳）有限公司 3d human face reconstruction method, apparatus and server
CN106780713A (en) * 2016-11-11 2017-05-31 吴怀宇 A kind of three-dimensional face modeling method and system based on single width photo
CN108363995A (en) * 2018-03-19 2018-08-03 百度在线网络技术（北京）有限公司 Method and apparatus for generating data
CN108491881A (en) * 2018-03-23 2018-09-04 百度在线网络技术（北京）有限公司 Method and apparatus for generating detection model
CN108549873A (en) * 2018-04-19 2018-09-18 北京华捷艾米科技有限公司 Three-dimensional face identification method and three-dimensional face recognition system
CN109712080A (en) * 2018-10-12 2019-05-03 迈格威科技有限公司 Image processing method, image processing apparatus and storage medium
CN109829969A (en) * 2018-12-27 2019-05-31 北京奇艺世纪科技有限公司 A kind of data capture method, device and storage medium
CN110619676A (en) * 2019-09-18 2019-12-27 东北大学 End-to-end three-dimensional face reconstruction method based on neural network
WO2020034785A1 (en) * 2018-08-16 2020-02-20 Oppo广东移动通信有限公司 Method and device for processing three-dimensional model

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2008200301A1 (en) * 2008-01-22 2009-08-06 The University Of Western Australia Image recognition
US20140355843A1 (en) * 2011-12-21 2014-12-04 Feipeng Da 3d face recognition method based on intermediate frequency information in geometric image
CN102999942A (en) * 2012-12-13 2013-03-27 清华大学 Three-dimensional face reconstruction method
WO2016188318A1 (en) * 2015-05-22 2016-12-01 腾讯科技（深圳）有限公司 3d human face reconstruction method, apparatus and server
CN106780713A (en) * 2016-11-11 2017-05-31 吴怀宇 A kind of three-dimensional face modeling method and system based on single width photo
CN108363995A (en) * 2018-03-19 2018-08-03 百度在线网络技术（北京）有限公司 Method and apparatus for generating data
CN108491881A (en) * 2018-03-23 2018-09-04 百度在线网络技术（北京）有限公司 Method and apparatus for generating detection model
CN108549873A (en) * 2018-04-19 2018-09-18 北京华捷艾米科技有限公司 Three-dimensional face identification method and three-dimensional face recognition system
WO2020034785A1 (en) * 2018-08-16 2020-02-20 Oppo广东移动通信有限公司 Method and device for processing three-dimensional model
CN109712080A (en) * 2018-10-12 2019-05-03 迈格威科技有限公司 Image processing method, image processing apparatus and storage medium
CN109829969A (en) * 2018-12-27 2019-05-31 北京奇艺世纪科技有限公司 A kind of data capture method, device and storage medium
CN110619676A (en) * 2019-09-18 2019-12-27 东北大学 End-to-end three-dimensional face reconstruction method based on neural network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张怡: "《基于二维人脸图像的三维建模研究》" *

Also Published As

Publication number Publication date
CN111402401B (en) 2023-08-18

Similar Documents

Publication Publication Date Title
KR102266529B1 (en) Method, apparatus, device and readable storage medium for image-based data processing
WO2017024963A1 (en) Image recognition method, measure learning method and image source recognition method and device
WO2018196718A1 (en) Image disambiguation method and device, storage medium, and electronic device
CN109214366A (en) Localized target recognition methods, apparatus and system again
CN108898269A (en) Electric power image-context impact evaluation method based on measurement
CN113887487A (en) Facial expression recognition method and device based on CNN-Transformer
US20230298307A1 (en) System for three-dimensional geometric guided student-teacher feature matching (3dg-stfm)
CN112949469A (en) Image recognition method, system and equipment for face tampered image characteristic distribution
KR20190061538A (en) Method and apparatus of recognizing motion pattern base on combination of multi-model
CN107330363B (en) Rapid internet billboard detection method
TWI731919B (en) Image recognition method and device and metric learning method and device
Zhou et al. MTCNet: Multi-task collaboration network for rotation-invariance face detection
Wang et al. From low-cost depth sensors to cad: Cross-domain 3d shape retrieval via regression tree fields
CN111723688B (en) Human body action recognition result evaluation method and device and electronic equipment
CN112668662B (en) Outdoor mountain forest environment target detection method based on improved YOLOv3 network
CN113033587A (en) Image recognition result evaluation method and device, electronic equipment and storage medium
CN110909678A (en) Face recognition method and system based on width learning network feature extraction
CN111402401B (en) Method for acquiring 3D face data, face recognition method and device
CN109472023A (en) Entity association degree measuring method and system based on entity and text combined embedding and storage medium
CN111428670B (en) Face detection method, face detection device, storage medium and equipment
CN114299590A (en) Training method of face completion model, face completion method and system
CN111126617B (en) Method, device and equipment for selecting fusion model weight parameters
CN114005169B (en) Face key point detection method and device, electronic equipment and storage medium
US11531829B2 (en) Automatic image annotation
US20230127555A1 (en) Grading apparatus and method based on digital data

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant