CN106503669B - Training and recognition method and system based on multitask deep learning network - Google Patents
Training and recognition method and system based on multitask deep learning network Download PDFInfo
- Publication number
- CN106503669B CN106503669B CN201610952920.2A CN201610952920A CN106503669B CN 106503669 B CN106503669 B CN 106503669B CN 201610952920 A CN201610952920 A CN 201610952920A CN 106503669 B CN106503669 B CN 106503669B
- Authority
- CN
- China
- Prior art keywords
- task
- face
- deep learning
- learning network
- recognition task
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/164—Detection; Localisation; Normalisation using holistic features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
- G06V40/169—Holistic features and representations, i.e. based on the facial image taken as a whole
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
Abstract
The invention provides a training method, an identification method and a system based on a multitask deep learning network, wherein the training method comprises the following steps: acquiring a face area of a face image in a training set; carrying out key point detection on the face area to obtain key feature point positions; affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image; inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model; the identification method comprises the following steps: carrying out affine transformation on the face image to be recognized according to the key feature position of the face image to be recognized to obtain an aligned face image; inputting the aligned face images into a trained multi-task deep learning network model for feature extraction to obtain feature information; and respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result. Therefore, the efficiency of multi-task deep learning network training and recognition can be improved.
Description
Technical Field
The invention relates to the technical field of face recognition, in particular to a training method, a recognition method and a system based on a multi-task deep learning network.
background
the face recognition technology is a technology for performing identity verification by using physiological characteristics or behavior characteristics which are owned by human beings and can uniquely mark the identity of the human beings based on a biological characteristic recognition mode. With the increasingly wide application of the human-computer interaction technology, the human face recognition technology has very important significance in the field of human-computer interaction. As one of the main research methods in the field of pattern recognition and machine learning, a large number of face recognition algorithms have been proposed.
At present, in a face recognition mode and various attribute recognition modes thereof, a deep learning network is usually trained independently according to different tasks to obtain respective deep learning network models, and then the deep learning network models obtained through training are independently recognized. However, the existing single-task deep learning network has low training and recognition efficiency, thereby causing the overall performance of the network to be reduced.
disclosure of Invention
in view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a training method, a recognition method and a system based on a multitask deep learning network, which can improve the efficiency and recognition rate of the training and recognition of the multitask deep learning network.
to achieve the above and other related objects, an embodiment of the present invention provides a training method based on a multitask deep learning network, including:
acquiring a face area of a face image in a training set;
carrying out key point detection on the face area to obtain key feature point positions of the face area;
Affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image;
Inputting the aligned face images into a multitask deep learning neural network for training to obtain a multitask deep learning neural network model;
Wherein, the structure of multitask deep learning neural network adopts the GoogleNet structure, multitask deep learning neural network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task sharing multitask deep learning neural network's convolution layer and first all-link layer, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task is the triplet function the loss function of age identification task is the softmax function the loss function of gender identification task is the softmax function.
preferably, the inputting the aligned face images into a multitask deep learning neural network for training to obtain a multitask deep learning neural network model includes:
inputting the aligned face image into a first layer convolution of a multitask deep learning neural network to complete convolution operation;
Inputting the obtained operation result to a second layer of convolution of the multi-task deep learning neural network to complete convolution operation until the obtained operation result is input to an Nth layer of convolution of the multi-task deep learning neural network to complete convolution operation, and then linking two full link layers to obtain a final training result;
and determining the multitask deep learning neural network model according to the training result.
Preferably, the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task are added according to a weight proportion to obtain a total loss function of the multitask deep learning neural network.
the embodiment of the invention also provides an identification method based on the multitask deep learning network, which comprises the following steps:
Acquiring a face area of a face image to be recognized;
Carrying out key point detection on the face area to obtain key feature point positions of the face area;
carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image;
inputting the aligned face images into a trained multitask deep learning neural network model for feature extraction to obtain feature information of the face images to be recognized;
respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result;
wherein, the structure of multitask deep learning neural network adopts the GoogleNet structure, multitask deep learning neural network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task sharing multitask deep learning neural network's convolution layer and first all-link layer, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task is the triplet function the loss function of age identification task is the softmax function the loss function of gender identification task is the softmax function.
preferably, the matching the feature information of the facial image to be recognized with the feature information corresponding to each facial image in the registered set respectively to obtain the recognition result includes:
Determining similarity values of the facial images to be recognized and each facial image in the registered set respectively by calculating Euclidean distances between the characteristic information of the facial images to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
and determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
According to the method, the embodiment of the invention provides a training system based on a multitask deep learning network, which comprises the following steps: the system comprises a face region acquisition module, a key point detection module, a face alignment module and a training module; wherein the content of the first and second substances,
the face region acquisition module is used for acquiring a face region of a face image in a training set;
the key point detection module is used for detecting key points of the face area to obtain key feature point positions of the face area;
the face alignment module is used for carrying out affine transformation on the face image according to the key feature position to obtain an aligned face image;
The training module is used for inputting the aligned face images into the multitask deep learning neural network for training to obtain a multitask deep learning neural network model;
wherein, the structure of multitask deep learning neural network adopts the GoogleNet structure, multitask deep learning neural network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task sharing multitask deep learning neural network's convolution layer and first all-link layer, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task is the triplet function the loss function of age identification task is the softmax function the loss function of gender identification task is the softmax function.
preferably, the training module is specifically configured to:
inputting the aligned face image into a first layer convolution of a multitask deep learning neural network to complete convolution operation;
Inputting the obtained operation result to a second layer of convolution of the multi-task deep learning neural network to complete convolution operation until the obtained operation result is input to an Nth layer of convolution of the multi-task deep learning neural network to complete convolution operation, and then linking two full link layers to obtain a final training result;
and obtaining the multitask deep learning neural network model according to the training result.
preferably, the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task are added according to a weight proportion to obtain a total loss function of the multitask deep learning neural network.
according to the method, the embodiment of the invention provides an identification system based on a multitask deep learning network, which comprises the following steps: the system comprises a face region acquisition module, a key point detection module, a face alignment module, a feature extraction module and a matching identification module; wherein the content of the first and second substances,
the face region acquisition module is used for acquiring a face region of a face image to be recognized;
the key point detection module is used for detecting key points of the face area to obtain key feature point positions of the face area;
the face alignment module is used for carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image;
The feature extraction module is used for inputting the aligned face images into the trained multitask deep learning neural network model for feature extraction to obtain feature information of the face images to be recognized;
The matching identification module is used for matching the characteristic information of the facial image to be identified with the characteristic information corresponding to each facial image in the registered set respectively to obtain an identification result;
wherein, the structure of multitask deep learning neural network adopts the GoogleNet structure, multitask deep learning neural network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task sharing multitask deep learning neural network's convolution layer and first all-link layer, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task is the triplet function the loss function of age identification task is the softmax function the loss function of gender identification task is the softmax function.
preferably, the matching identification module is specifically configured to:
Determining similarity values of the facial images to be recognized and each facial image in the registered set respectively by calculating Euclidean distances between the characteristic information of the facial images to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
And determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
The invention provides a training method, an identification method and a system based on a multitask deep learning network, wherein the training method comprises the following steps: acquiring a face area of a face image in a training set; carrying out key point detection on the face area to obtain key feature point positions of the face area; affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image; inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model; the identification method comprises the following steps: acquiring a face area of a face image to be recognized; carrying out key point detection on the face area to obtain key feature point positions of the face area; carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image; inputting the aligned face images into a trained multi-task deep learning network model for feature extraction to obtain feature information of the face images to be recognized; respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result; the structure of the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with a full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function. Therefore, in the embodiment of the invention, the face recognition task, the gender recognition task and the age recognition task are arranged in the multi-task deep learning network, the correlation among the face recognition task, the gender recognition task and the age recognition task is established in the multi-task deep learning network, the multi-task deep learning network is used for training and learning common characteristics to obtain a multi-task deep learning network model, and then the trained multi-task deep learning network is used for extracting and recognizing the characteristics, so that the efficiency and the recognition rate of the overall multi-task deep learning network training and recognition can be improved, and the recognition rate of a single task can be improved.
Drawings
FIG. 1 is a schematic flow chart of a training method based on a multitask deep learning network according to the present invention;
FIG. 2 is a flowchart illustrating a recognition method based on a multitask deep learning network according to the present invention;
FIG. 3 is a schematic diagram illustrating the structure of the training system based on the multitask deep learning network according to the present invention;
FIG. 4 is a schematic diagram showing the structure of the recognition system based on the multitask deep learning network according to the present invention.
Detailed Description
in the embodiment of the invention, firstly, a face area of a face image in a training set is obtained; carrying out key point detection on the face area to obtain key feature point positions of the face area; affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image; inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model; and then, performing feature extraction and recognition on the face image to be recognized according to the trained multitask deep learning network model.
the invention is described in further detail below with reference to the figures and the embodiments.
the embodiment of the invention provides a training method based on a multitask deep learning network, which comprises the following steps of:
step S100: and acquiring the face area of the face image in the training set.
in the step, firstly, images containing human faces are collected, and human face regions and key feature points in the human face images are calibrated according to a preset rule to generate a training set. Specifically, a face region and key feature points in a face image are calibrated according to preset rules of a training set for an image containing a face, which is acquired by a user through various ways, and the position and scale information of the calibrated face region and the coordinate information of the key feature points are uploaded to a PC (personal computer) and stored in a corresponding document by a server.
in this step, a face detection algorithm may be used to obtain the face region of the face image in the training set, the face detection algorithm may be an AdaBoost algorithm or a deep learning face detection algorithm, and how to obtain the face region of the face image in the training set by using the AdaBoost algorithm or the deep learning face detection algorithm belongs to the prior art, and repeated parts are not described again. The face detection algorithm is not particularly limited herein.
In this step, the face image in the training set is one of bmp, jpg, tiff, gif, pcx, tga, exif, fpx, svg, psd, cdr, pcd, dxf, ufo, eps, ai, and raw in any format, and is a non-compressed image.
step S101: and carrying out key point detection on the face area to obtain the key feature point position of the face area.
In this step, how to perform the key point detection on the face region adopts the existing key point detection algorithm, and repeated parts are not described again.
Step S102: and carrying out affine transformation on the face image according to the key feature position to obtain an aligned face image.
in this step, how to obtain an aligned face image by affine transformation of the face image according to the key feature position belongs to the prior art, and repeated parts are not described again.
step S103: and inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model.
In this step, the structure of multitask deep learning network adopts the GoogleNet structure, multitask deep learning network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task sharing the convolution layer and the first all-link layer of multitask deep learning network, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task be the triplet function the loss function of age identification task be the softmax function the loss function of gender identification task is the softmax function.
specifically, the triplet loss function of the face recognition task is as follows:
wherein, gamma is a triple of the face image in the training setthe collection of (a) and (b),is a matrix of images of a human face,Is another face image matrix belonging to the same class as the face image,is a face image matrix which does not belong to the same category as the face image, and alpha is a matrix in a tripleSum matrixSimilarity value and matrix ofSum matrixI is a positive integer.
Specifically, the softmax loss function of the gender identification task is as follows:
L＝-(1-g)·log(1-p0)-g·log(p1)
Wherein, if the gender is female, g is set to 0; if the gender is male, setting g to 1; p0For the calculated probability of gender being female, P, from the multitask deep learning network1And the sex calculated from the multitask deep learning network is male.
specifically, the softmax loss function of the age identification task is as follows:
L＝-(g0)·log(p0)-g1·log(p1)-…gn·log(pn)
Wherein, PnFor each age probability, g, calculated from the multitask deep learning networknFor each age weight coefficient, n is a positive integer.
specifically, the loss function of each task is distributed according to a weight proportion, and the total loss function of the multitask deep learning network obtained by adding the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task according to the weight proportion is as follows:
wherein L isallFor the total loss function, L, of the multitask deep learning networkna loss function, λ, for the nth task in said multi-task deep learning networknThe weighting scaling factor in the overall loss for the nth task.
here, the weight ratio of the loss function of each of the face recognition task, the age recognition task, and the gender recognition task is set according to actual conditions and requirements, and the weight ratio of the loss function of each of the face recognition task, the age recognition task, and the gender recognition task is not particularly limited.
In this step, the multitask deep learning network needs to be trained, and the multitask deep learning network is specifically trained in the following manner:
inputting the aligned face images into a first layer of convolution of a multitask deep learning network to complete convolution operation;
inputting the obtained operation result to the second layer of convolution of the multi-task deep learning network to complete convolution operation until the obtained operation result is input to the Nth layer of convolution of the multi-task deep learning network to complete convolution operation, and then linking two full link layers to obtain a final training result;
and determining the multi-task deep learning network model according to the training result.
it should be noted that the convolution process of the multitask deep learning network belongs to the prior art, and repeated parts are not described again.
the embodiment of the invention provides an identification method based on a multitask deep learning network, which comprises the following steps of:
step S200: and acquiring a face area of the face image to be recognized.
in this step, the same face detection algorithm as that in step S100 is used to obtain the face region of the face image to be recognized, and the repeated parts are not described again.
step S201: and carrying out key point detection on the face area to obtain the key feature point position of the face area.
in this step, the same key point detection algorithm as that in step S101 is used to perform key point detection on the face region of the face image to be recognized, so as to obtain the key feature point position of the face region.
Step S202: and carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image.
in this step, as in step S102, how to obtain an aligned face image by affine transformation of the face image according to the key feature position belongs to the prior art, and repeated parts are not described again.
Step S203: and inputting the aligned face images into the trained multi-task deep learning network model for feature extraction to obtain feature information of the face images to be recognized.
In this step, the multitask deep learning network model is the network model that obtains for training through step S100 ~ S102, wherein, the structure of multitask deep learning network adopts GoogleNet structure, multitask deep learning network includes face identification task, age identification task, gender identification task, the face identification task the age identification task reaches gender identification task shares the convolution layer and the first all-link layer of multitask deep learning network, the loss function of the specific all-link layer of every task of first all-link layer connection and every task, the loss function of face identification task is the triplet function, the loss function of age identification task is the softmax function, the loss function of recognition task is the softmax function.
in this step, the aligned face images are input into the trained deep learning network model for feature extraction, and the output of the first full link layer of the multi-task deep learning network model is used as the feature information of the face images to be recognized.
Step S204: and respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result.
Specifically, firstly, determining similarity values between the facial image to be recognized and each facial image in the registered set by calculating Euclidean distances between the characteristic information of the facial image to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
And then determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
Here, how to calculate the euclidean distance between the feature information of the face image to be recognized and the feature information corresponding to each face image in the registration set belongs to the prior art, and repeated parts are not described again.
Here, the similarity threshold may be preset according to actual situations and requirements, and is not specifically limited herein.
how to determine the recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value is explained in detail below:
If any similarity value is larger than or equal to the preset similarity threshold, matching is successful, and the identification result is the identification success and the number of the corresponding category.
And if all the similarity values are smaller than the preset similarity threshold value, the matching fails and the recognition result is output as the recognition failure.
in order to implement the method, the embodiment of the invention also provides a training system based on the fusion of key feature points of the multitask deep learning network and an identification system based on the fusion of key feature points of the multitask deep learning network.
An embodiment of the present invention provides a training system based on a multitask deep learning network, as shown in fig. 3, the system includes: a face region acquisition module 300, a key point detection module 301, a face alignment module 302 and a training module 303; wherein the content of the first and second substances,
the face region acquiring module 300 is configured to acquire a face region of a face image in a training set;
The key point detection module 301 is configured to perform key point detection on the face region to obtain key feature point positions of the face region;
the face alignment module 302 is configured to perform affine transformation on the face image according to the key feature position to obtain an aligned face image;
The training module 303 is configured to input the aligned face images into a multitask deep learning network for training to obtain a multitask deep learning network model;
the structure of the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with a full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function.
In a specific implementation, the training module 303 is specifically configured to:
Inputting the aligned face images into a first layer of convolution of a multitask deep learning network to complete convolution operation;
Inputting the obtained operation result to the second layer of convolution of the multi-task deep learning network to complete convolution operation until the obtained operation result is input to the Nth layer of convolution of the multi-task deep learning network to complete convolution operation, and then linking two full link layers to obtain a final training result;
And obtaining the multi-task deep learning network model according to the training result.
In specific implementation, the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task are added according to a weight proportion to obtain a total loss function of the multi-task deep learning network.
the above division manner of the functional modules is only one preferred implementation manner given in the embodiment of the present invention, and the division manner of the functional modules does not limit the present invention. For convenience of description, the parts of the system described above are separately described as functionally divided into various modules or units. Of course, the functionality of the various modules or units may be implemented in the same one or more pieces of software or hardware in practicing the invention.
The embodiment of the invention provides an identification system based on a multitask deep learning network, as shown in fig. 4, the system comprises: a face region acquisition module 400, a key point detection module 401, a face alignment module 402, a feature extraction module 403 and a matching identification module 404; wherein the content of the first and second substances,
The face region acquiring module 400 is configured to acquire a face region of a face image to be recognized;
The key point detection module 401 is configured to perform key point detection on the face region to obtain key feature point positions of the face region;
the face alignment module 402 is configured to perform affine transformation on a face image to be recognized according to the key feature position to obtain an aligned face image;
the feature extraction module 403 is configured to input the aligned face images into a trained multitask deep learning network model for feature extraction, so as to obtain feature information of the face images to be recognized;
The matching identification module 404 is configured to match the feature information of the facial image to be identified with the feature information corresponding to each facial image in the registered set, so as to obtain an identification result;
The structure of the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with a full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function.
In a specific implementation, the matching identification module 404 is specifically configured to:
Determining similarity values of the facial images to be recognized and each facial image in the registered set respectively by calculating Euclidean distances between the characteristic information of the facial images to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
And determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
The above division manner of the functional modules is only one preferred implementation manner given in the embodiment of the present invention, and the division manner of the functional modules does not limit the present invention. For convenience of description, the parts of the system described above are separately described as functionally divided into various modules or units. Of course, the functionality of the various modules or units may be implemented in the same one or more pieces of software or hardware in practicing the invention.
In summary, in the training process, the embodiment of the present invention first obtains the face area of the face image in the training set; carrying out key point detection on the face area to obtain key feature point positions of the face area; affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image; inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model; in the identification process, acquiring a face area of a face image to be identified; carrying out key point detection on the face area to obtain key feature point positions of the face area; carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image; inputting the aligned face images into a trained multi-task deep learning network model for feature extraction to obtain feature information of the face images to be recognized; respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result; the structure of the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with a full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function. Therefore, the embodiment of the invention obtains a multitask deep learning network model through the multitask deep learning network training, the model can extract the fusion characteristic information including face recognition, gender recognition and age recognition, and the multitask recognition is carried out according to the characteristic information, so that the multitask recognition of the face and the attribute characteristics is realized, the efficiency and the recognition rate of the whole multitask deep learning network training and recognition can be improved, and the recognition rate of a single task can be improved.
the foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Accordingly, it is intended that all equivalent modifications or changes which can be made by those skilled in the art without departing from the spirit and technical spirit of the present invention be covered by the claims of the present invention.
Claims (8)
1. a training method based on a multitask deep learning network is characterized by comprising the following steps:
acquiring a face area of a face image in a training set;
carrying out key point detection on the face area to obtain key feature point positions of the face area;
Affine transformation is carried out on the face image according to the key feature position to obtain an aligned face image;
inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model;
the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with the full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function; and adding the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task according to a weight proportion to obtain a total loss function of the multi-task deep learning network.
2. the method of claim 1, wherein the inputting the aligned face images into a multitask deep learning network for training to obtain a multitask deep learning network model comprises:
Inputting the aligned face images into a first layer of convolution of a multitask deep learning network to complete convolution operation;
inputting the obtained operation result to the second layer of convolution of the multi-task deep learning network to complete convolution operation until the obtained operation result is input to the Nth layer of convolution of the multi-task deep learning network to complete convolution operation, and then linking two full link layers to obtain a final training result;
and determining the multi-task deep learning network model according to the training result.
3. a recognition method based on a multitask deep learning network is characterized by comprising the following steps:
Acquiring a face area of a face image to be recognized;
carrying out key point detection on the face area to obtain key feature point positions of the face area;
Carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image;
Inputting the aligned face images into a trained multi-task deep learning network model for feature extraction to obtain feature information of the face images to be recognized;
respectively matching the characteristic information of the facial image to be recognized with the characteristic information corresponding to each facial image in the registered set to obtain a recognition result;
The multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with the full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function; and adding the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task according to a weight proportion to obtain a total loss function of the multi-task deep learning network.
4. The method according to claim 3, wherein the matching the feature information of the facial image to be recognized with the feature information corresponding to each facial image in the registered set respectively to obtain the recognition result comprises:
Determining similarity values of the facial images to be recognized and each facial image in the registered set respectively by calculating Euclidean distances between the characteristic information of the facial images to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
and determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
5. a training system based on a multitask deep learning network, the system comprising: the system comprises a face region acquisition module, a key point detection module, a face alignment module and a training module; wherein the content of the first and second substances,
the face region acquisition module is used for acquiring a face region of a face image in a training set;
The key point detection module is used for detecting key points of the face area to obtain key feature point positions of the face area;
the face alignment module is used for carrying out affine transformation on the face image according to the key feature position to obtain an aligned face image;
The training module is used for inputting the aligned face images into a multi-task deep learning network for training to obtain a multi-task deep learning network model;
the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with the full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function; and adding the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task according to a weight proportion to obtain a total loss function of the multi-task deep learning network.
6. the system of claim 5, wherein the training module is specifically configured to:
Inputting the aligned face images into a first layer of convolution of a multitask deep learning network to complete convolution operation;
Inputting the obtained operation result to the second layer of convolution of the multi-task deep learning network to complete convolution operation until the obtained operation result is input to the Nth layer of convolution of the multi-task deep learning network to complete convolution operation, and then linking two full link layers to obtain a final training result;
and obtaining the multi-task deep learning network model according to the training result.
7. A recognition system based on a multitask deep learning network, the system comprising: the system comprises a face region acquisition module, a key point detection module, a face alignment module, a feature extraction module and a matching identification module; wherein the content of the first and second substances,
the face region acquisition module is used for acquiring a face region of a face image to be recognized;
The key point detection module is used for detecting key points of the face area to obtain key feature point positions of the face area;
The face alignment module is used for carrying out affine transformation on the face image to be recognized according to the key feature position to obtain an aligned face image;
The feature extraction module is used for inputting the aligned face images into the trained multi-task deep learning network model for feature extraction to obtain feature information of the face images to be recognized;
The matching identification module is used for matching the characteristic information of the facial image to be identified with the characteristic information corresponding to each facial image in the registered set respectively to obtain an identification result;
the multitask deep learning network adopts a GoogleNet structure, the multitask deep learning network comprises a face recognition task, an age recognition task and a gender recognition task, the face recognition task, the age recognition task and the gender recognition task share a convolution layer and a first full link layer of the multitask deep learning network, the first full link layer is connected with the full link layer specific to each task and a loss function of each task, the loss function of the face recognition task is a triplet function, the loss function of the age recognition task is a softmax function, and the loss function of the gender recognition task is a softmax function; and adding the loss functions of each task of the face recognition task, the age recognition task and the gender recognition task according to a weight proportion to obtain a total loss function of the multi-task deep learning network.
8. the system of claim 7, wherein the match identification module is specifically configured to:
Determining similarity values of the facial images to be recognized and each facial image in the registered set respectively by calculating Euclidean distances between the characteristic information of the facial images to be recognized and the characteristic information corresponding to each facial image in the registered set respectively;
And determining a recognition result according to the similarity value between the facial image to be recognized and each facial image in the registered set and a preset similarity threshold value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610952920.2A CN106503669B (en) | 2016-11-02 | 2016-11-02 | Training and recognition method and system based on multitask deep learning network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610952920.2A CN106503669B (en) | 2016-11-02 | 2016-11-02 | Training and recognition method and system based on multitask deep learning network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106503669A CN106503669A (en) | 2017-03-15 |
| CN106503669B true CN106503669B (en) | 2019-12-10 |
Family
ID=58321285
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610952920.2A Active CN106503669B (en) | 2016-11-02 | 2016-11-02 | Training and recognition method and system based on multitask deep learning network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106503669B (en) |
Families Citing this family (56)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563281A (en) * | 2017-07-24 | 2018-01-09 | 南京邮电大学 | A kind of construction site personal security hidden danger monitoring method based on deep learning |
| CN108875489A (en) * | 2017-09-30 | 2018-11-23 | 北京旷视科技有限公司 | Method for detecting human face, device, system, storage medium and capture machine |
| CN107679513B (en) * | 2017-10-20 | 2021-07-13 | 北京达佳互联信息技术有限公司 | Image processing method and device and server |
| CN107832700A (en) * | 2017-11-03 | 2018-03-23 | 全悉科技（北京）有限公司 | A kind of face identification method and system |
| CN107886062B (en) * | 2017-11-03 | 2019-05-10 | 北京达佳互联信息技术有限公司 | Image processing method, system and server |
| CN108052862B (en) * | 2017-11-09 | 2019-12-06 | 北京达佳互联信息技术有限公司 | Age estimation method and device |
| CN107977633B (en) * | 2017-12-06 | 2019-04-09 | 平安科技（深圳）有限公司 | Age recognition methods, device and the storage medium of facial image |
| CN107844784A (en) * | 2017-12-08 | 2018-03-27 | 广东美的智能机器人有限公司 | Face identification method, device, computer equipment and readable storage medium storing program for executing |
| CN107895160A (en) * | 2017-12-21 | 2018-04-10 | 曙光信息产业(北京)有限公司 | Human face detection and tracing device and method |
| CN110197099B (en) * | 2018-02-26 | 2022-10-11 | 腾讯科技（深圳）有限公司 | Method and device for cross-age face recognition and model training thereof |
| CN108734145B (en) * | 2018-02-27 | 2022-03-29 | 北京紫睛科技有限公司 | Face recognition method based on density self-adaptive face representation model |
| WO2019202073A1 (en) * | 2018-04-18 | 2019-10-24 | Deepmind Technologies Limited | Neural networks for scalable continual learning in domains with sequentially learned tasks |
| EA202092529A1 (en) * | 2018-04-23 | 2021-02-02 | Общество с ограниченной ответственностью "ЦРТ-инновации" | METHOD FOR TRAINING A NEURAL NETWORK FOR HUMAN FACE RECOGNITION |
| CN108665065B (en) * | 2018-04-25 | 2020-08-04 | 清华大学 | Method, device and equipment for processing task data and storage medium |
| CN110263603B (en) * | 2018-05-14 | 2021-08-06 | 桂林远望智能通信科技有限公司 | Face recognition method and device based on central loss and residual error visual simulation network |
| CN109086660A (en) * | 2018-06-14 | 2018-12-25 | 深圳市博威创盛科技有限公司 | Training method, equipment and the storage medium of multi-task learning depth network |
| CN109033953A (en) * | 2018-06-14 | 2018-12-18 | 深圳市博威创盛科技有限公司 | Training method, equipment and the storage medium of multi-task learning depth network |
| CN108960167B (en) * | 2018-07-11 | 2023-08-18 | 腾讯科技（深圳）有限公司 | Hairstyle identification method, device, computer readable storage medium and computer equipment |
| CN108986094B (en) * | 2018-07-20 | 2021-08-17 | 南京开为网络科技有限公司 | Automatic data updating method for training image library for face recognition |
| CN109190514B (en) * | 2018-08-14 | 2021-10-01 | 电子科技大学 | Face attribute recognition method and system based on bidirectional long-short term memory network |
| CN109117801A (en) * | 2018-08-20 | 2019-01-01 | 深圳壹账通智能科技有限公司 | Method, apparatus, terminal and the computer readable storage medium of recognition of face |
| CN109359528A (en) * | 2018-09-11 | 2019-02-19 | 开放智能机器（上海）有限公司 | A kind of recognition of face neural network |
| CN109522829B (en) * | 2018-11-02 | 2021-10-26 | 南京邮电大学 | Smart phone face-brushing conference registration method based on deep learning |
| CN109522872A (en) * | 2018-12-04 | 2019-03-26 | 西安电子科技大学 | A kind of face identification method, device, computer equipment and storage medium |
| CN109712127B (en) * | 2018-12-21 | 2023-07-28 | 云南电网有限责任公司电力科学研究院 | Power transmission line fault detection method for machine inspection video stream |
| CN109784219A (en) * | 2018-12-28 | 2019-05-21 | 广州海昇计算机科技有限公司 | A kind of face identification method, system and device based on concentration cooperated learning |
| CN109543663B (en) * | 2018-12-28 | 2021-04-27 | 北京旷视科技有限公司 | Method, device and system for identifying identity of dog and storage medium |
| EP3674974A1 (en) * | 2018-12-31 | 2020-07-01 | Samsung Electronics Co., Ltd. | Apparatus and method with user verification |
| CN109766872B (en) * | 2019-01-31 | 2021-07-09 | 广州视源电子科技股份有限公司 | Image recognition method and device |
| CN111723613A (en) * | 2019-03-20 | 2020-09-29 | 广州慧睿思通信息科技有限公司 | Face image data processing method, device, equipment and storage medium |
| CN110147721B (en) * | 2019-04-11 | 2023-04-18 | 创新先进技术有限公司 | Three-dimensional face recognition method, model training method and device |
| CN110033332A (en) * | 2019-04-23 | 2019-07-19 | 杭州智趣智能信息技术有限公司 | A kind of face identification method, system and electronic equipment and storage medium |
| CN110059645A (en) * | 2019-04-23 | 2019-07-26 | 杭州智趣智能信息技术有限公司 | A kind of face identification method, system and electronic equipment and storage medium |
| CN110070047B (en) * | 2019-04-23 | 2021-03-26 | 杭州智趣智能信息技术有限公司 | Face comparison method and system, electronic equipment and storage medium |
| CN110084174A (en) * | 2019-04-23 | 2019-08-02 | 杭州智趣智能信息技术有限公司 | A kind of face identification method, system and electronic equipment and storage medium |
| CN110163269A (en) * | 2019-05-09 | 2019-08-23 | 北京迈格威科技有限公司 | Model generating method, device and computer equipment based on deep learning |
| CN110276403B (en) * | 2019-06-25 | 2022-02-18 | 北京百度网讯科技有限公司 | Model building method and device |
| CN112149449A (en) * | 2019-06-26 | 2020-12-29 | 北京华捷艾米科技有限公司 | Face attribute recognition method and system based on deep learning |
| EP4012578A4 (en) | 2019-08-15 | 2022-10-05 | Huawei Technologies Co., Ltd. | Face retrieval method and device |
| CN110728255B (en) * | 2019-10-22 | 2022-12-16 | Oppo广东移动通信有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN110826525B (en) * | 2019-11-18 | 2023-05-26 | 天津高创安邦技术有限公司 | Face recognition method and system |
| CN112836549A (en) * | 2019-11-22 | 2021-05-25 | 虹软科技股份有限公司 | User information detection method and system and electronic equipment |
| CN110879993B (en) * | 2019-11-29 | 2023-03-14 | 北京市商汤科技开发有限公司 | Neural network training method, and execution method and device of face recognition task |
| CN111027428B (en) * | 2019-11-29 | 2024-03-08 | 北京奇艺世纪科技有限公司 | Training method and device for multitasking model and electronic equipment |
| CN111339869A (en) * | 2020-02-18 | 2020-06-26 | 北京拙河科技有限公司 | Face recognition method, face recognition device, computer readable storage medium and equipment |
| CN111460939A (en) * | 2020-03-20 | 2020-07-28 | 深圳市优必选科技股份有限公司 | Deblurring face recognition method and system and inspection robot |
| CN111428671A (en) * | 2020-03-31 | 2020-07-17 | 杭州博雅鸿图视频技术有限公司 | Face structured information identification method, system, device and storage medium |
| CN111598000A (en) * | 2020-05-18 | 2020-08-28 | 中移（杭州）信息技术有限公司 | Face recognition method, device, server and readable storage medium based on multiple tasks |
| CN112115854A (en) * | 2020-09-17 | 2020-12-22 | 广东光速智能设备有限公司 | Offline lightweight face recognition system applied to construction site gate |
| CN112241723B (en) * | 2020-10-27 | 2021-06-25 | 新疆爱华盈通信息技术有限公司 | Sex and age identification method, system, electronic device and storage medium |
| CN112308149B (en) * | 2020-11-02 | 2023-10-24 | 平安科技（深圳）有限公司 | Optimization method and device for image information identification based on machine learning |
| CN112488053B (en) * | 2020-12-17 | 2023-10-13 | 深圳市优必选科技股份有限公司 | Face recognition method, device, robot and storage medium |
| CN112597888B (en) * | 2020-12-22 | 2024-03-08 | 西北工业大学 | Online education scene student attention recognition method aiming at CPU operation optimization |
| CN112949571A (en) * | 2021-03-26 | 2021-06-11 | 北京金山云网络技术有限公司 | Method for identifying age, and training method and device of age identification model |
| CN113239885A (en) * | 2021-06-04 | 2021-08-10 | 新大陆数字技术股份有限公司 | Face detection and recognition method and system |
| CN117079337B (en) * | 2023-10-17 | 2024-02-06 | 成都信息工程大学 | High-precision face attribute feature recognition device and method |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138973A (en) * | 2015-08-11 | 2015-12-09 | 北京天诚盛业科技有限公司 | Face authentication method and device |
| CN105404877A (en) * | 2015-12-08 | 2016-03-16 | 商汤集团有限公司 | Human face attribute prediction method and apparatus based on deep study and multi-task study |
| CN105912611A (en) * | 2016-04-05 | 2016-08-31 | 中国科学技术大学 | CNN based quick image search method |
| CN105912990A (en) * | 2016-04-05 | 2016-08-31 | 深圳先进技术研究院 | Face detection method and face detection device |
| CN105913025A (en) * | 2016-04-12 | 2016-08-31 | 湖北工业大学 | Deep learning face identification method based on multiple-characteristic fusion |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10074041B2 (en) * | 2015-04-17 | 2018-09-11 | Nec Corporation | Fine-grained image classification by exploring bipartite-graph labels |
-
2016
- 2016-11-02 CN CN201610952920.2A patent/CN106503669B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105138973A (en) * | 2015-08-11 | 2015-12-09 | 北京天诚盛业科技有限公司 | Face authentication method and device |
| CN105404877A (en) * | 2015-12-08 | 2016-03-16 | 商汤集团有限公司 | Human face attribute prediction method and apparatus based on deep study and multi-task study |
| CN105912611A (en) * | 2016-04-05 | 2016-08-31 | 中国科学技术大学 | CNN based quick image search method |
| CN105912990A (en) * | 2016-04-05 | 2016-08-31 | 深圳先进技术研究院 | Face detection method and face detection device |
| CN105913025A (en) * | 2016-04-12 | 2016-08-31 | 湖北工业大学 | Deep learning face identification method based on multiple-characteristic fusion |
Non-Patent Citations (5)
| Title |
|---|
| Deep Learning Face Representation by Joint Identification-Verification;Yi Sun 等;《arXiv:1406.4773v1》;20140618;1-9 * |
| FaceNet: A Unified Embedding for Face Recognition and Clustering;Florian Schroff等;《2015 IEEE conference on computer vision and pattern recognition（CVPR）》;20150612;815-823 * |
| 基于GoogLeNet多阶段连带优化的图像描述;汤鹏杰 等;《井冈山大学学报(自然科学版)》;20160930;第37卷(第5期);47-57 * |
| 基于卷积神经网络的图像生成方式分类方法;李巧玲等;《网络与信息安全学报》;20160930;第2卷(第9期);第2.3节 * |
| 多任务学习及卷积神经网络在人脸识别中的应用;邵蔚元等;《计算机工程与应用》;20160701;第52卷(第13期);摘要，第2.2-2.3节，第3.1-3.5节，图2 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106503669A (en) | 2017-03-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106503669B (en) | Training and recognition method and system based on multitask deep learning network | |
| Liu et al. | Multi-modal clique-graph matching for view-based 3d model retrieval | |
| CN109800648B (en) | Face detection and recognition method and device based on face key point correction | |
| CN107977609B (en) | Finger vein identity authentication method based on CNN | |
| CN105243154B (en) | Remote sensing image retrieval method based on notable point feature and sparse own coding and system | |
| CN103605972A (en) | Non-restricted environment face verification method based on block depth neural network | |
| CN105138974B (en) | A kind of multi-modal Feature fusion of finger based on Gabor coding | |
| CN111339930A (en) | Face recognition method combining mask attribute loss function | |
| CN104932804A (en) | Intelligent virtual assembly action recognition method | |
| Huang et al. | Clothing landmark detection using deep networks with prior of key point associations | |
| CN111401303B (en) | Cross-visual angle gait recognition method with separated identity and visual angle characteristics | |
| TWI774271B (en) | Key point detection method, electronic device and computer readable storage medium | |
| CN110516533A (en) | A kind of pedestrian based on depth measure discrimination method again | |
| CN108960171B (en) | Method for converting gesture recognition into identity recognition based on feature transfer learning | |
| CN108875907A (en) | A kind of fingerprint identification method and device based on deep learning | |
| Ren et al. | A deep and structured metric learning method for robust person re-identification | |
| CN112766229A (en) | Human face point cloud image intelligent identification system and method based on attention mechanism | |
| CN111476222A (en) | Image processing method, image processing device, computer equipment and computer readable storage medium | |
| CN107122725A (en) | A kind of face identification method and its system based on joint sparse discriminant analysis | |
| CN110414431B (en) | Face recognition method and system based on elastic context relation loss function | |
| CN103714340A (en) | Self-adaptation feature extracting method based on image partitioning | |
| CN114821786A (en) | Gait recognition method based on human body contour and key point feature fusion | |
| Ibrahim et al. | A hyprid technique for human footprint recognition | |
| CN105550642A (en) | Gender identification method and system based on multi-scale linear difference characteristic low-rank expression | |
| Hou et al. | A face detection algorithm based on two information flow block and retinal receptive field block |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | ||
| CP03 | Change of name, title or address |
Address after: 401122 5 stories, Block 106, West Jinkai Avenue, Yubei District, Chongqing Patentee after: Chongqing Zhongke Yuncong Technology Co., Ltd. Address before: 401122, Chongqing North New District, Mount Huangshan Avenue, mercury science and technology building, B District, are six floor Patentee before: CHONGQING ZHONGKE YUNCONG TECHNOLOGY CO., LTD. |