CN110705355A - Face pose estimation method based on key point constraint - Google Patents

Face pose estimation method based on key point constraint Download PDF

Info

Publication number
CN110705355A
CN110705355A CN201910817335.5A CN201910817335A CN110705355A CN 110705355 A CN110705355 A CN 110705355A CN 201910817335 A CN201910817335 A CN 201910817335A CN 110705355 A CN110705355 A CN 110705355A
Authority
CN
China
Prior art keywords
key point
neural network
layer
face
training
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910817335.5A
Other languages
Chinese (zh)
Inventor
王枫
胡庆浩
程健
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Artificial Intelligence Chip Innovation Institute Institute Of Automation Chinese Academy Of Sciences
Original Assignee
Nanjing Artificial Intelligence Chip Innovation Institute Institute Of Automation Chinese Academy Of Sciences
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Artificial Intelligence Chip Innovation Institute Institute Of Automation Chinese Academy Of Sciences filed Critical Nanjing Artificial Intelligence Chip Innovation Institute Institute Of Automation Chinese Academy Of Sciences
Priority to CN201910817335.5A priority Critical patent/CN110705355A/en
Publication of CN110705355A publication Critical patent/CN110705355A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161Detection; Localisation; Normalisation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06NCOMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00Computing arrangements based on biological models
    • G06N3/02Neural networks
    • G06N3/04Architecture, e.g. interconnection topology
    • G06N3/045Combinations of networks
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168Feature extraction; Face representation

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Human Computer Interaction (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Mathematical Physics (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Molecular Biology (AREA)
  • Evolutionary Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a face pose estimation method based on key point constraint, which comprises the steps of obtaining a portrait to be estimated, establishing a training set, marking key points through the training set, and marking 68 key points on faces in all the training set through a preset neural network model; and normalizing the key points to obtain corresponding key point coordinates, inputting the key point coordinates into a preset human face attitude angle detection neural network, and outputting a human face attitude angle value by the preset human face attitude angle detection neural network. The human face attitude angle detection method based on the neural network carries out key point positioning on a human face image to be detected through the preset key point neural network to obtain 68 key points, and detects the key points through the preset human face attitude angle detection neural network to obtain a human face attitude angle value.

Description

Face pose estimation method based on key point constraint
Technical Field
The invention relates to a human face posture estimation method, and belongs to the technical field of image information processing.
Background
The human face pose estimation plays an important role in non-rigid registration and three-dimensional reconstruction in the intelligent driving vision technology and realization of a new user interaction method. In the past, there are several methods for face modeling, and two main methods are a method based on key points and a parameterized characterization model. In recent years, a method for directly extracting two-dimensional face keys by using a deep learning tool has become a main method for analyzing face pose due to flexibility and robustness in occlusion and large face pose change. The traditional head pose estimation and calculation method is to obtain two-dimensional key points from a face and then use an average human head model autumn festival to solve the corresponding problem from two dimensions to three dimensions. But this method relies entirely on the accuracy of the keypoint detection and the construction of an average head model.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: the method is used for solving the problems that the traditional method for estimating the key points of the face detail posture is difficult to realize accurate detection on the key points of the face detail posture, and the detection effect on the face posture with large shielding and posture change is poor.
In order to achieve the purpose, the invention provides the following technical scheme:
the embodiment of the invention provides a face pose estimation method based on key point constraint, which comprises the following steps:
step 1, establishing a training set, wherein the training set comprises a plurality of face samples, and key points are marked on face images in the training set;
step 2, establishing a multitask convolution neural network model, and performing key point positioning on all face samples in a training set through a preset key point neural network model to obtain 68 key points;
and 3, carrying out normalization processing on the key points to obtain corresponding key point coordinates by the established training set, inputting the key point coordinates into the established multitask convolutional neural network model, and outputting a face posture angle value by the multitask convolutional neural network model.
And 4, training the convolutional neural network designed in the step 2 by using the training sample obtained in the step 1, and finishing the training when the error calculated by the training reaches an expected value, and obtaining the parameters of the convolutional neural network model.
And 5, testing the sample in the step 1 by using the model obtained by training in the step 3, and outputting a final face pose estimation result.
Further, the multitask convolutional neural network model established in the step 2 comprises an input layer, a first layer full-connection layer, a second layer full-connection layer and an output layer.
Further, the dimension of the input layer is 1 × 136, the dimension of the first fully-connected layer is 136 × 68, the dimension of the second fully-connected layer is 68 × 3, and the dimension of the output layer is 1 × 3.
Further, the estimation of the face pose specifically includes: and normalizing the key points to obtain corresponding key point coordinates, entering the key point coordinates through an input layer, sequentially processing the key point coordinates through a first full-connection layer and a second full-connection layer, and finally outputting a face pose angle value through an output layer.
Further, the face pose angle value includes a horizontal angle value, a tilt angle value, and a pitch angle value.
Further, the input layer can output the key points and the Euler angles of the human face postures at the same time.
Compared with the prior art, the invention adopting the technical scheme has the following technical effects:
the method is based on the face pose estimation of the key point constraint, introduces the key point information of the face as the supervision information, optimizes the results of three Euler angles obtained by direct regression, can not only simultaneously carry out key point prediction and face pose estimation, but also has more accurate effect compared with the prior face pose estimation method.
The foregoing description is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood and to implement them in accordance with the contents of the description, the following detailed description is given with reference to the preferred embodiments of the present invention and the accompanying drawings. The detailed description of the present invention is given in detail by the following examples and the accompanying drawings.
Drawings
FIG. 1 is a schematic flow chart of the estimation method of the present invention;
Detailed Description
The present invention is further described with reference to the accompanying drawings and the detailed description, and it should be noted that any combination of the following described embodiments or technical features can be made to form a new embodiment without conflict.
As shown in fig. 1, a face pose estimation method based on key point constraint of the present invention includes:
step 1, establishing a training set, wherein the training set comprises a plurality of human face samples to be detected, and key points are marked on human face images in the training set;
step 2, establishing a multitask convolutional neural network model, wherein the established multitask convolutional neural network model comprises an input layer, a first layer full-connection layer, a second layer full-connection layer and an output layer; carrying out key point positioning on the face samples in all training sets through a preset key point neural network model to obtain 68 key points; and calculating a rotation matrix according to the 68-point two-dimensional coordinates obtained by normalization, converting the rotation matrix into an Euler angle, calculating the loss by using the Euler angle and the Euler angle obtained by direct network regression and the label simultaneously, wherein the loss of the overall attitude estimation is the sum of two losses.
And 3, carrying out normalization processing on the key points to obtain corresponding key point coordinates by the established training set, inputting the key point coordinates into the established multitask convolutional neural network model, and outputting face attitude angle values by the multitask convolutional neural network model, wherein the face attitude angle values correspond to three Euler angles of pitch, yaw and roll.
The estimation of the face pose specifically comprises the following steps: and normalizing the key points to obtain corresponding key point coordinates, and inputting the key point coordinates into a preset human face posture detection neural network model. The key point coordinates enter through the input layer and are processed through the first full-connection layer and the second full-connection layer in sequence, and finally the face pose angle value is output through the output layer. The face posture angle value comprises a horizontal rotation angle value, a tilt angle value and a pitch angle value; the horizontal turning angle value is the left-right turning range value of the human face, the inclination angle value is the degree value of the inclination of the human face, and the pitching angle value is the upward or downward range value of the human face.
The input layer can simultaneously output key points and the Euler angles of the human face postures; the dimension of the input layer is 1 x 136, the dimension of the first fully-connected layer is 136 x 68, the dimension of the second fully-connected layer is 68 x 3, and the dimension of the output layer is 1 x 3.
And 4, training the convolutional neural network designed in the step 2 by using the training sample obtained in the step 1, and finishing the training when the error calculated by the training reaches an expected value, and obtaining the parameters of the convolutional neural network model.
And 5, testing the sample in the step 1 by using the model obtained by training in the step 3, and outputting a final face pose estimation result.
The foregoing is merely a preferred embodiment of the invention and is not intended to limit the invention in any manner; those skilled in the art can readily practice the invention as shown and described in the drawings and detailed description herein; however, those skilled in the art should appreciate that they can readily use the disclosed conception and specific embodiments as a basis for designing or modifying other structures for carrying out the same purposes of the present invention without departing from the scope of the invention as defined by the appended claims; meanwhile, any changes, modifications, and evolutions of the equivalent changes of the above embodiments according to the actual techniques of the present invention are still within the protection scope of the technical solution of the present invention.

Claims (5)

1. A face pose estimation method based on key point constraint is characterized by comprising the following steps:
step 1, acquiring an image to be detected; establishing a training set, wherein the training set comprises a plurality of face samples,
marking key points for the face images in the multiple training sets;
step 2, establishing a multitask convolution neural network model, and performing key point positioning on all face samples in a training set through a preset key point neural network model to obtain 68 key points;
step 3, the established training set is used for carrying out normalization processing on the key points to obtain corresponding key point coordinates, the key point coordinates are input into an established multitask convolution neural network model, and the multitask convolution neural network model outputs a face posture angle value;
step 4, training the convolutional neural network designed in the step 2 by using the training sample obtained in the step 1, finishing the training when the error calculated by the training reaches an expected value, and obtaining the parameters of the convolutional neural network model;
and 5, testing the sample in the step 1 by using the model obtained by training in the step 3, and outputting a final result of the face pose angle.
2. The face pose estimation method based on the key point constraint according to claim 1, characterized in that: the multitask convolution neural network model established in the step 2 comprises an input layer, a first layer full connection layer, a second layer full connection layer and an output layer.
3. The face pose estimation method based on the key point constraint according to claim 2, characterized in that: the method is characterized in that: the dimension of the input layer is 1 x 136, the dimension of the first fully-connected layer is 136 x 68, the dimension of the second fully-connected layer is 68 x 3, and the dimension of the output layer is 1 x 3.
4. The face pose estimation method based on the key point constraint according to claim 2, characterized in that: : the detection of the face pose angle specifically comprises the following steps: and normalizing the key points to obtain corresponding key point coordinates, entering the key point coordinates through an input layer, sequentially processing the key point coordinates through a first full-connection layer and a second full-connection layer, and finally outputting a face pose angle value through an output layer.
5. The face pose angle detection method based on neural network as claimed in claim 1, wherein: the face pose angle value comprises a horizontal angle value, a tilt angle value and a pitch angle value.
CN201910817335.5A 2019-08-30 2019-08-30 Face pose estimation method based on key point constraint Pending CN110705355A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910817335.5A CN110705355A (en) 2019-08-30 2019-08-30 Face pose estimation method based on key point constraint

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910817335.5A CN110705355A (en) 2019-08-30 2019-08-30 Face pose estimation method based on key point constraint

Publications (1)

Publication Number Publication Date
CN110705355A true CN110705355A (en) 2020-01-17

Family

ID=69194278

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910817335.5A Pending CN110705355A (en) 2019-08-30 2019-08-30 Face pose estimation method based on key point constraint

Country Status (1)

Country Link
CN (1) CN110705355A (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112699784A (en) * 2020-12-29 2021-04-23 深圳市普渡科技有限公司 Face orientation estimation method and device, electronic equipment and storage medium
WO2022089360A1 (en) * 2020-10-28 2022-05-05 广州虎牙科技有限公司 Face detection neural network and training method, face detection method, and storage medium
CN118038560A (en) * 2024-04-12 2024-05-14 魔视智能科技(武汉)有限公司 Method and device for predicting face pose of driver

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107122705A (en) * 2017-03-17 2017-09-01 中国科学院自动化研究所 Face critical point detection method based on three-dimensional face model
CN109359526A (en) * 2018-09-11 2019-02-19 深圳大学 A kind of face pose estimation, device and equipment
CN109359537A (en) * 2018-09-14 2019-02-19 杭州宇泛智能科技有限公司 Human face posture angle detecting method neural network based and system

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107122705A (en) * 2017-03-17 2017-09-01 中国科学院自动化研究所 Face critical point detection method based on three-dimensional face model
CN109359526A (en) * 2018-09-11 2019-02-19 深圳大学 A kind of face pose estimation, device and equipment
CN109359537A (en) * 2018-09-14 2019-02-19 杭州宇泛智能科技有限公司 Human face posture angle detecting method neural network based and system

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2022089360A1 (en) * 2020-10-28 2022-05-05 广州虎牙科技有限公司 Face detection neural network and training method, face detection method, and storage medium
CN112699784A (en) * 2020-12-29 2021-04-23 深圳市普渡科技有限公司 Face orientation estimation method and device, electronic equipment and storage medium
WO2022143264A1 (en) * 2020-12-29 2022-07-07 深圳市普渡科技有限公司 Face orientation estimation method and apparatus, electronic device, and storage medium
CN118038560A (en) * 2024-04-12 2024-05-14 魔视智能科技(武汉)有限公司 Method and device for predicting face pose of driver

Similar Documents

Publication Publication Date Title
CN108764048B (en) Face key point detection method and device
CN109325398B (en) Human face attribute analysis method based on transfer learning
CN106599830B (en) Face key point positioning method and device
CN110246181B (en) Anchor point-based attitude estimation model training method, attitude estimation method and system
CN111160269A (en) Face key point detection method and device
CN105740780B (en) Method and device for detecting living human face
WO2017049677A1 (en) Facial key point marking method
CN110705355A (en) Face pose estimation method based on key point constraint
CN106355147A (en) Acquiring method and detecting method of live face head pose detection regression apparatus
CN103324938A (en) Method for training attitude classifier and object classifier and method and device for detecting objects
CN106599810B (en) A kind of head pose estimation method encoded certainly based on stack
CN113361542B (en) Local feature extraction method based on deep learning
CN112613097A (en) BIM rapid modeling method based on computer vision
WO2023151237A1 (en) Face pose estimation method and apparatus, electronic device, and storage medium
WO2022142214A1 (en) Vehicle pose determination method and apparatus, vehicle control method and apparatus, vehicle, and storage medium
CN112818969A (en) Knowledge distillation-based face pose estimation method and system
CN111259735B (en) Single-person attitude estimation method based on multi-stage prediction feature enhanced convolutional neural network
CN110135277B (en) Human behavior recognition method based on convolutional neural network
CN113011401B (en) Face image posture estimation and correction method, system, medium and electronic equipment
CN111028319A (en) Three-dimensional non-photorealistic expression generation method based on facial motion unit
CN110827304A (en) Traditional Chinese medicine tongue image positioning method and system based on deep convolutional network and level set method
CN116249607A (en) Method and device for robotically gripping three-dimensional objects
CN111199207A (en) Two-dimensional multi-human body posture estimation method based on depth residual error neural network
Yin et al. Estimation of the fundamental matrix from uncalibrated stereo hand images for 3D hand gesture recognition
CN110288026A (en) A kind of image partition method and device practised based on metric relation graphics

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
CB02 Change of applicant information

Address after: 211000 floor 3, building 3, Qilin artificial intelligence Industrial Park, 266 Chuangyan Road, Nanjing, Jiangsu

Applicant after: Zhongke Nanjing artificial intelligence Innovation Research Institute

Address before: 211000 3rd floor, building 3, 266 Chuangyan Road, Jiangning District, Nanjing City, Jiangsu Province

Applicant before: NANJING ARTIFICIAL INTELLIGENCE CHIP INNOVATION INSTITUTE, INSTITUTE OF AUTOMATION, CHINESE ACADEMY OF SCIENCES

CB02 Change of applicant information
RJ01 Rejection of invention patent application after publication

Application publication date: 20200117

RJ01 Rejection of invention patent application after publication