CN110097035A - A kind of facial feature points detection method based on 3D human face rebuilding - Google Patents
A kind of facial feature points detection method based on 3D human face rebuilding Download PDFInfo
- Publication number
- CN110097035A CN110097035A CN201910410426.7A CN201910410426A CN110097035A CN 110097035 A CN110097035 A CN 110097035A CN 201910410426 A CN201910410426 A CN 201910410426A CN 110097035 A CN110097035 A CN 110097035A
- Authority
- CN
- China
- Prior art keywords
- human face
- characteristic point
- model
- face
- photo
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 33
- 230000001815 facial effect Effects 0.000 title claims abstract description 21
- 238000013527 convolutional neural network Methods 0.000 claims abstract description 13
- 238000012549 training Methods 0.000 claims description 12
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000003287 optical effect Effects 0.000 claims description 2
- 238000005457 optimization Methods 0.000 claims description 2
- 238000013519 translation Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 13
- 230000007547 defect Effects 0.000 abstract description 2
- 230000000694 effects Effects 0.000 abstract description 2
- 230000036544 posture Effects 0.000 description 11
- 238000012360 testing method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
Abstract
The invention belongs to technical field of face recognition, are related to a kind of facial feature points detection method based on 3D human face rebuilding.The present invention uses 3D face universal model, realizes the corresponding relationship between characteristic point by 3D human face rebuilding, realizes the detection of big posture human face characteristic point by cascade form using concatenated convolutional neural network.The result shows that the present invention can solve defect of the existing method on big posture facial feature points detection, and it can be realized better detection effect.
Description
Technical field
The invention belongs to technical field of face recognition, are related to a kind of facial feature points detection side based on 3D human face rebuilding
Method.
Background technique
Facial feature points detection, that is, facial modeling, face alignment, are on the basis of Face datection, on face
The characteristic point such as corners of the mouth, canthus, nose and facial contour etc. positioned, facial feature points detection is many human face analysis
Important component, as face character analysis, face verification, face replacement and recognition of face.Therefore the face of robustness
Feature point detection algorithm has great importance, and studies especially for big posture facial feature points detection with greater need for expansion.
In recent years, facial feature points detection task achieved major progress.Conventional method is mainly used based on recurrence
The iteration that method carries out characteristic point position updates, and random forest, SVM and Adaboost algorithm is such as used to detect as characteristic point
Device, but the character representation that these algorithms extract is limited, and there is no consider to the shape constraining between human face characteristic point.For
It solves the problems, such as this, is then suggested based on markov random file come the method for the spatial relationship between simulation of facial characteristic point
Come, this method simultaneously divides human face region, the progress characteristic point detection of the region that samples out to it, effective solution with
The related space constraint problem of characteristic point, but this method is only done well on small posture face sample.By means of convolution
Neural network, feature point extraction may be implemented completely automation and high efficiency, relatively before method performance on have very big mention
It rises.Meanwhile cascading shape regression model and achieving good achievement on characteristic point Detection task, this method uses recurrence mould
Type, study is from face appearance to the mapping function of face face shape model parameter, that is, characteristic point position, to establish from outer
See the corresponding relationship of shape.Therefore, the big appearance of more Shandong nation how is carried out using cascade Recurrent networks model and neural network
State facial feature points detection is current research direction.
Summary of the invention
For the above-mentioned prior art, there are problems or deficiency, in order to realize the big posture facial feature points detection of robust, this
Invention provides a kind of facial feature points detection method based on 3D human face rebuilding and concatenated convolutional neural network.
The present invention is realized by following steps:
Step 1, input photo are simultaneously pre-processed, so that the face in photo meets characteristic point detection demand.
The input photo that this process expands algorithm to sample has carried out pretreatment operation, mainly in detection input photo
Human face region cuts and zooms to fixed size.
Step 2,3D human face rebuilding.
It is detected using human face characteristic point detection algorithm, obtains 68 characteristic point (xk,yk), k=1,2 ..., 68 is right
3D characteristic point coordinate on the 3D faceform answered is (Uk,Vk,Wk), i.e. point in 3D dimension module.Where point in 3D model
Coordinate system is known as world coordinate system, by spin matrix R and translation vector t, 3D characteristic point can be transformed in camera coordinates system,
Obtain corresponding 3D coordinate points (Xk,Yk,Zk).Corresponding relationship are as follows:
For the point in camera coordinates system, 2D plane is projected into according to camera parameter, generates corresponding 2D coordinate points
Position (xk,yk):
Wherein fxAnd fyIt is the focal length in the direction x and y, cxAnd cyIt is the optical centre of image, s is scale factor.
Define the projection relation of 3D characteristic point and 2D characteristic point are as follows:
Wherein, P is camera matrix, and P=[[1,0,0], [0,1,0]] ignores the third dimension after projecting, and above formula is simplified
Are as follows:
The process is as shown in Figure 1.
Based on BFM three-dimensional face model, by (Uk,Vk,Wk) it is expressed as average shapeWith shape principal component Si, texture it is main at
Divide TiLinear combination, then definition projected by 3D coordinate points to the corresponding relationship of 2D point are as follows:
It establishes 3D and rebuilds optimization aim model are as follows:
Wherein γiIt is PCA coefficient, i.e. form factor, σiFor corresponding principal component deviation.By solving object module, obtain
So that the value on 68 projecting characteristic points to two-dimensional surface in 3D model reaches with the distance between original 2D human face characteristic point difference
The smallest parameter completes 3D human face rebuilding.
Step 3, the PNCC feature for solving input human face photo:
PCNN character representation inputs the corresponding 3D model of human face photo, and feature includes two steps: first to the mould after reconstruction
Type is normalized:
Wherein, SdFor the coordinate value of three dimensions at 3D model midpoint,For the average value of three dimensions, obtained after normalization
To NCC feature NCCd.It is stored using three of them dimension as rgb value with the format of image, then uses Z-Buffer to NCCdMake
It is rendered for the 3D face after the projection of color, obtains PNCC feature.
PNCC=Z-Buffer (Xprojection,NCCd)
Step 4, training concatenated convolutional neural network:
Construct the concatenated convolutional neural network that each network has identical network structure, network structure as shown in Fig. 2,
Using human face photo and PNCC feature as input, using the parameter of reconstruction model as output.Concatenated convolutional neural network is carried out
Training, the training data used are to expand to generate by 3D human face rebuilding, and training data distribution is as shown in Figure 3.Using training
Good concatenated convolutional neural network carries out facial feature points detection to input photo.
The invention has the advantages that being realized between characteristic point using 3D face universal model by 3D human face rebuilding
Corresponding relationship realizes the detection of big posture human face characteristic point using concatenated convolutional neural network by cascade form.The result shows that
The present invention can solve defect of the existing method on big posture facial feature points detection, and can be realized preferably detection effect
Fruit.
Detailed description of the invention
Fig. 1 is the corresponding relationship of 2D human face characteristic point and 3D feature；
The network structure that Fig. 2 concatenated convolutional neural network uses；
Fig. 3 training data and test data face sample gesture distribution schematic diagram；
Fig. 4 is the test result using trained model in the proper manners sheet comprising various postures.
Specific embodiment
Below shown in Fig. 4 for human face photo, the invention will be further described.
The present invention is while to guarantee the feature of small posture face sample to solve the problems, such as big posture facial feature points detection
Point detection accuracy.Training data of the invention comes from AFLW2000-3D data set, which is the base in AFLW data set
Expanded first by 3D human face rebuilding on plinth and generated, each sample provides 68 human face characteristic points, chooses at random after training
The photo in photo and 2K the AFLW-3D data set in 21K AFLW data sets is selected to be tested to detect the property of this method
Energy.
The system environments of experiment is Ubuntu16.04, and hardware system is Intel i7-6700HQ, processor GTX
1080Ti video card.Using TensorFlow frame training concatenated convolutional neural network, and network level-one level-one is trained, current network
Loss reaches retraining next stage network after certain value.Network parameter setting are as follows: learning rate initial value is 0.001, and momentum value is set
It is 0.9.50 epochs convergences of network training.The face sample comprising various postures is carried out using trained network special
Sign point detection, as a result as shown in Figure 4.Compared with existing facial feature points detection technology, the present invention is on big posture face sample
It can be realized the testing result of robustness, and can guarantee the accuracy of detection for small posture sample.
Claims (1)
1. a kind of facial feature points detection method based on 3D human face rebuilding, comprising the following specific steps
Step 1, input photo are simultaneously pre-processed, so that the face in photo meets characteristic point detection demand；
Step 2,3D human face rebuilding:
It is detected using human face characteristic point detection algorithm, obtains 68 characteristic point (xk,yk), k=1,2 ..., 68 is corresponding
3D characteristic point coordinate on 3D faceform is (Uk,Vk,Wk)；The coordinate system where point in 3D model is known as world coordinate system,
By spin matrix R and translation vector t, 3D characteristic point is transformed in camera coordinates system, obtains corresponding 3D coordinate points (Xk,Yk,
Zk), corresponding relationship are as follows:
For the point in camera coordinates system, 2D plane is projected into according to camera parameter, generates corresponding 2D coordinate points position
(xk,yk):
Wherein fxAnd fyIt is the focal length in the direction x and y, cxAnd cyIt is the optical centre of image, s is scale factor；
Define the projection relation of 3D characteristic point and 2D characteristic point are as follows:
Wherein, P is camera matrix, and P=[[1,0,0], [0,1,0]] ignores the third dimension after projecting, and above formula is simplified are as follows:
Based on BFM three-dimensional face model, by (Uk,Vk,Wk) it is expressed as average shapeWith shape principal component Si, texture principal component Ti
Linear combination, then definition projected by 3D coordinate points to the corresponding relationship of 2D point are as follows:
It establishes 3D and rebuilds optimization aim model are as follows:
Wherein γiIt is PCA coefficient, i.e. form factor, σiFor corresponding principal component deviation；By solve object module, obtain so that
The distance between value on 68 projecting characteristic points to two-dimensional surface and original 2D human face characteristic point in 3D model difference reaches minimum
Parameter, complete 3D human face rebuilding；
Step 3, the PNCC feature for solving input human face photo:
PCNN character representation inputs the corresponding 3D model of human face photo, and feature includes two steps: first to the model after reconstruction into
Row normalization:
Wherein, SdFor the coordinate value of three dimensions at 3D model midpoint,For the average value of three dimensions, obtained after normalization
NCC feature NCCd, stored using three of them dimension as rgb value with the format of image, then use Z-Buffer to NCCdAs
3D face after the projection of color is rendered, and PNCC feature is obtained.
PNCC=Z-Buffer (Xprojection,NCCd)
Step 4, training concatenated convolutional neural network:
The concatenated convolutional neural network that each network has identical network structure is constructed, is made with human face photo and PNCC feature
To input, using the parameter of reconstruction model as output；Concatenated convolutional neural network is trained, the training data used is logical
It crosses 3D human face rebuilding and expands generation；
Facial feature points detection is carried out to input photo using trained concatenated convolutional neural network.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910410426.7A CN110097035A (en) | 2019-05-15 | 2019-05-15 | A kind of facial feature points detection method based on 3D human face rebuilding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910410426.7A CN110097035A (en) | 2019-05-15 | 2019-05-15 | A kind of facial feature points detection method based on 3D human face rebuilding |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110097035A true CN110097035A (en) | 2019-08-06 |
Family
ID=67448461
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910410426.7A Pending CN110097035A (en) | 2019-05-15 | 2019-05-15 | A kind of facial feature points detection method based on 3D human face rebuilding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110097035A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348406A (en) * | 2019-07-15 | 2019-10-18 | 广州图普网络科技有限公司 | Parameter deducing method and device |
| CN110569768A (en) * | 2019-08-29 | 2019-12-13 | 四川大学 | construction method of face model, face recognition method, device and equipment |
| CN111401157A (en) * | 2020-03-02 | 2020-07-10 | 中国电子科技集团公司第五十二研究所 | Face recognition method and system based on three-dimensional features |
| CN112434795A (en) * | 2020-12-01 | 2021-03-02 | 中国科学院空天信息创新研究院 | Novel three-dimensional artificial neuron device and artificial neural network recognition system |
| US11170203B2 (en) | 2019-11-27 | 2021-11-09 | National Central University | Training data generation method for human facial recognition and data generation apparatus |
| TWI758662B (en) * | 2019-11-27 | 2022-03-21 | 國立中央大學 | Training data generation method for human facial recognition and data generation apparatus |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104899563A (en) * | 2015-05-29 | 2015-09-09 | 深圳大学 | Two-dimensional face key feature point positioning method and system |
| CN106203248A (en) * | 2014-09-05 | 2016-12-07 | 三星电子株式会社 | Method and apparatus for face recognition |
| CN108805056A (en) * | 2018-05-29 | 2018-11-13 | 电子科技大学 | A kind of monitoring camera-shooting face sample extending method based on 3D faceforms |
-
2019
- 2019-05-15 CN CN201910410426.7A patent/CN110097035A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106203248A (en) * | 2014-09-05 | 2016-12-07 | 三星电子株式会社 | Method and apparatus for face recognition |
| CN104899563A (en) * | 2015-05-29 | 2015-09-09 | 深圳大学 | Two-dimensional face key feature point positioning method and system |
| CN108805056A (en) * | 2018-05-29 | 2018-11-13 | 电子科技大学 | A kind of monitoring camera-shooting face sample extending method based on 3D faceforms |
Non-Patent Citations (2)
| Title |
|---|
| XIANGYU ZHU等: "Face Alignment Across Large Poses: A 3D Solution", 《2016IEEE》 * |
| 王亮: "基于3D人脸重建的大姿态人脸样本扩充与识别研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110348406A (en) * | 2019-07-15 | 2019-10-18 | 广州图普网络科技有限公司 | Parameter deducing method and device |
| CN110348406B (en) * | 2019-07-15 | 2021-11-02 | 广州图普网络科技有限公司 | Parameter estimation method and device |
| CN110569768A (en) * | 2019-08-29 | 2019-12-13 | 四川大学 | construction method of face model, face recognition method, device and equipment |
| CN110569768B (en) * | 2019-08-29 | 2022-09-02 | 四川大学 | Construction method of face model, face recognition method, device and equipment |
| US11170203B2 (en) | 2019-11-27 | 2021-11-09 | National Central University | Training data generation method for human facial recognition and data generation apparatus |
| TWI758662B (en) * | 2019-11-27 | 2022-03-21 | 國立中央大學 | Training data generation method for human facial recognition and data generation apparatus |
| CN111401157A (en) * | 2020-03-02 | 2020-07-10 | 中国电子科技集团公司第五十二研究所 | Face recognition method and system based on three-dimensional features |
| CN112434795A (en) * | 2020-12-01 | 2021-03-02 | 中国科学院空天信息创新研究院 | Novel three-dimensional artificial neuron device and artificial neural network recognition system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110097035A (en) | A kind of facial feature points detection method based on 3D human face rebuilding | |
| CN110458939B (en) | Indoor scene modeling method based on visual angle generation | |
| Sheikh et al. | Exploring the space of a human action | |
| Jiang et al. | Dual attention mobdensenet (damdnet) for robust 3d face alignment | |
| CN108171133B (en) | Dynamic gesture recognition method based on characteristic covariance matrix | |
| Varol et al. | Monocular 3D reconstruction of locally textured surfaces | |
| CN112950775A (en) | Three-dimensional face model reconstruction method and system based on self-supervision learning | |
| Lee et al. | Finding optimal views for 3D face shape modeling | |
| CN108196681A (en) | The real-time touch-control system of air-borne imagery is realized according to recognition of face and laser image | |
| Castelan et al. | A coupled statistical model for face shape recovery from brightness images | |
| Bichsel | Automatic interpolation and recognition of face images by morphing | |
| CN106127818A (en) | A kind of material appearance based on single image obtains system and method | |
| WO2018080533A1 (en) | Real-time generation of synthetic data from structured light sensors for 3d object pose estimation | |
| Kong et al. | Structure from category: A generic and prior-less approach | |
| Moreno-Noguer et al. | Stochastic exploration of ambiguities for nonrigid shape recovery | |
| Kao et al. | Towards 3d face reconstruction in perspective projection: Estimating 6dof face pose from monocular image | |
| Zheng et al. | Extracting non-negative basis images using pixel dispersion penalty | |
| CN104252715B (en) | Single line image-based three-dimensional reconstruction method | |
| Hill et al. | Aging the human face-a statistically rigorous approach | |
| CN109934926B (en) | Model data processing method, device, readable storage medium and equipment | |
| CN115830241A (en) | Realistic three-dimensional face texture reconstruction method based on neural network | |
| Taheri et al. | Joint albedo estimation and pose tracking from video | |
| Ma et al. | A lighting robust fitting approach of 3D morphable model for face reconstruction | |
| Meng et al. | Sample-aware data augmentor for scene text recognition | |
| Chandar et al. | 3D face model estimation based on similarity transform using differential evolution optimization |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190806 |