WO2020199906A1 - 人脸关键点检测方法、装置、设备及存储介质 - Google Patents

人脸关键点检测方法、装置、设备及存储介质 Download PDF

Info

Publication number
WO2020199906A1
WO2020199906A1 PCT/CN2020/079493 CN2020079493W WO2020199906A1 WO 2020199906 A1 WO2020199906 A1 WO 2020199906A1 CN 2020079493 W CN2020079493 W CN 2020079493W WO 2020199906 A1 WO2020199906 A1 WO 2020199906A1
Authority
WO
WIPO (PCT)
Prior art keywords
face
image
correction
original
posture
Prior art date
Application number
PCT/CN2020/079493
Other languages
English (en)
French (fr)
Inventor
项伟
黄秋实
Original Assignee
广州市百果园信息技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 广州市百果园信息技术有限公司 filed Critical 广州市百果园信息技术有限公司
Publication of WO2020199906A1 publication Critical patent/WO2020199906A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/60—Rotation of a whole image or part thereof
    • G06T3/608—Skewing or deskewing, e.g. by two-pass or three-pass rotation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/161—Detection; Localisation; Normalisation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation

Definitions

  • the embodiments of the present application relate to the field of image recognition technology, such as methods, devices, devices, and storage media for detecting key points of a human face.
  • face key point detection In the field of computer vision, face key point detection has always received extensive attention from academia and industry. Its main function is to accurately locate key points on the face (such as eyes, nose, mouth corners, facial contour points, etc.). To prepare for subsequent face image processing (such as face alignment, face recognition), at present, face key point detection has played a more important role in application scenarios such as biological information verification, surveillance security, and live video.
  • neural network models are often used to realize face key point detection.
  • larger-scale neural network models are often used.
  • the usual method is to add more large-angle and large-posture face training samples to train the current neural network model, but in actual operation
  • Such large-angle and large-posture face training samples are not easy to obtain, and under the premise of limiting the size of the network, the increase in sample complexity will often lead to a decrease in the accuracy of the neural network model in detecting normal face images.
  • the detection time of image frames will be increased, and the real-time requirements of detection cannot be guaranteed.
  • the embodiments of the present application provide a method, device, device, and storage medium for detecting key points of a face to optimize the method for detecting key points of a face in related technologies, ensuring real-time performance and improving the detection accuracy of key points of a face.
  • an embodiment of the present application provides a method for detecting key points of a face, including:
  • an embodiment of the present application provides a face key point detection device, including:
  • An information acquisition module configured to acquire the original face image of the current frame to be tested, and to acquire the posture correction information of the face in the original face image of the current frame to be tested;
  • An image correction module configured to correct the face posture in the original image of the face according to the posture correction information to obtain a face correction image
  • the key point determination module is set to adopt a key point detection network model to perform face key point detection on the face correction image to obtain correction key points;
  • the key point correction module is configured to perform reverse posture correction on the corrected key point according to the posture correction information to obtain the target face key point of the original image of the face.
  • an embodiment of the present application provides a computer device, including:
  • At least one processor At least one processor
  • the storage device is set to store at least one program
  • the at least one program is executed by the at least one processor, so that the at least one processor implements the face key point detection method provided in the embodiment of the first aspect of the present application.
  • an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the face key point detection method provided in the embodiment of the first aspect of the present application is implemented.
  • FIG. 1a shows a schematic flowchart of a method for detecting key points of a face according to an embodiment of the present application
  • Figure 1b shows an example diagram of an original image of a human face in an embodiment of the present application
  • Fig. 1c shows an example of a face correction image formed after correcting the original face image in an embodiment of the present application
  • Figure 1d shows a schematic diagram of the implementation of correcting key points in an embodiment of the present application
  • Fig. 1e shows an example diagram of correction key points detected from a face correction image in an embodiment of the present application
  • FIG. 1f shows an example diagram of the key points of the target face formed after the key points are reversely adjusted and corrected in an embodiment of the present application
  • FIG. 2a shows a schematic flowchart of another method for detecting key points of a face according to an embodiment of the present application
  • Figure 2b shows the effect of setting special effect stickers based on the key points of the target face detected in Figure 2a;
  • Figure 2c shows a block diagram of the realization of face key point detection in an embodiment of the present application
  • FIG. 3 shows a schematic structural block diagram of a face key point detection apparatus provided by an embodiment of the present application
  • Fig. 4 shows a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present application.
  • Figure 1a shows a schematic flow chart of a face key point detection method provided by an embodiment of the present application. This method is suitable for the case of face key point detection on a face image. The method can be detected by face key points
  • the device executes, where the device can be implemented by software and/or hardware, and is generally integrated on a computer device, for example, can be integrated as a plug-in in application software with human-computer interaction.
  • the execution subject of the face key point detection method provided in this embodiment may be a computer device serving as a request terminal in data information interaction.
  • the computer device may include: mobile terminals, tablet computers, desktop computers, and the like.
  • this embodiment can perform face key point detection on video frames containing person images.
  • the method for detecting key points of a face includes steps S101 to S104.
  • step S101 the original face image of the current frame to be tested is obtained, and the posture correction information of the face in the original face image of the current frame to be tested is obtained.
  • the current frame to be tested may refer to an image frame that is currently to be subjected to face key point detection.
  • the current frame to be tested may be from A video frame obtained from a short video in the playing state, where the short video may be a video formed by pre-capture recording; the current frame to be measured may also be a video frame obtained from a live video captured in real time.
  • this embodiment is mainly used to realize the detection of key points of a human face, it can be preset that there should be a human face image in the acquired current frame to be measured.
  • the face image contained in the current frame to be tested may be recorded as an original face image, and the original face image may be regarded as a detection object for face key point detection.
  • the face in the original face image of the current frame to be tested, the face may be in a non-positive special posture, such as tilted head, yaw, and pitch, etc., in order to accurately and efficiently realize the key point detection of the special posture face
  • the posture correction information can be understood as correction information required to correct the posture of the face in the original image of the face.
  • the posture correction information may be the pixel coordinate offset information for posture correction, the posture information of the face in the three-dimensional space, or the face in the historical frame where the key point detection has been performed. Posture information.
  • the face pose information in the three-dimensional space of the face in the original face image of the current frame to be measured may be determined by analysis, and the obtained pose information of the face in the current frame to be measured is used as the pose correction information; It is also possible to associate the face posture information in the previous frame as the posture correction information by determining the face posture information of the face in the three-dimensional space in the original face image of the previous frame; it can also be determined through analysis to determine the current pose.
  • the pixel coordinate offset information of each pixel of the original image of the face in the frame to be measured is used as the posture correction information.
  • the present embodiment determines the face pose information of the face in the three-dimensional space as the pose correction information.
  • the face posture information can be regarded as angle information presented in terms of pitch, yaw, and roll when the face is relative to the spatial coordinate system to which the face belongs in the world space coordinate system. Therefore, when the present embodiment uses face posture information as posture correction information, the posture correction information may include: face posture angle and corresponding angle value; said face posture angle includes: pitch angle, yaw angle, and Roll corner.
  • step S102 the face pose in the original image of the face is corrected according to the pose correction information to obtain a face correction image.
  • correcting the posture of the face can be regarded as correcting and adjusting the displayed position of the face in the current frame to be measured.
  • corresponding correction operations can be performed based on the specific values contained in the posture correction information.
  • this step can obtain the coordinate offset value corresponding to the pixel in the pixel coordinate offset information, and then directly compare the original image of the face in the current frame to be measured.
  • the coordinates of each pixel point are offset adjustment of the coordinate offset value, so that the image formed after the pixel point coordinate adjustment is recorded as a face correction image.
  • this step can obtain the face posture angle and its corresponding angle value in the face posture information, and then obtain the standard angle information of the face posture angle in the standard orientation, and then Based on the angle value of the face attitude angle and the standard angle information, the rotation angle when the original face image is corrected and adjusted to the standard orientation can be obtained, and finally the pixels in the original face image can be rotated by the rotation angle to obtain Correct the image for the rotated face.
  • FIG. 1b shows an example of an original image of a face in an embodiment of the present application
  • FIG. 1c shows an example of a corrected face image formed after the original image of a face is corrected in an embodiment of the present application Figure
  • Figure 1b it can be seen that the face pose of the original face image 100 presented in the figure is tilted to the right from the screen angle of view, as shown in Figure 1c, it can be seen that the original image of the face in Figure 1b 100
  • a face correction image 101 presented in a forward pose is formed.
  • step S103 a key point detection network model is used to perform face key point detection on the face correction image to obtain correction key points.
  • the key point detection network model may be a pre-trained deep convolutional neural network model with a moderate scale, and the correction key point may be understood as the face key point detected from the face correction image .
  • this step can use the face correction image as input data, input the key point detection network model, and then obtain the correction key points output by the key point detection network model.
  • Fig. 1d shows a schematic diagram of the implementation of determining the key points of correction in an embodiment of the present application.
  • this step may first perform preprocessing on the face correction image 10 through the description in step 11 to obtain Set the length and width of the input image, and then input the deep convolutional neural network model 14 formed after training, and finally obtain the output correction key points in the form of coordinates.
  • step S104 inverse posture correction is performed on the correction key points according to the posture correction information to obtain the target face key points of the original image of the face.
  • the reverse pose correction may be equivalent to reversely restoring the correction key points detected from the face correction image to the original face image.
  • the face correction image is obtained through the correction of the posture correction information.
  • the correction key points are reversed based on the posture correction information to obtain the corresponding correction key points in the original face image.
  • the key points of the target face may be equivalent to reversely restoring the correction key points detected from the face correction image to the original face image.
  • this step can also obtain the coordinate offset value corresponding to the pixel point in the pixel coordinate offset information, and then directly perform the coordinate on the obtained correction detection point.
  • the reverse offset of the offset value, the coordinate value obtained after the reverse offset can be used to represent the key points of the target face in the original face image.
  • this step can obtain the rotation angle determined when the above S102 is implemented, and then perform the reverse rotation of the rotation angle based on the set reference axis for the correction detection point, and the same .
  • the corresponding coordinate value after reverse rotation can also be used to identify the key points of the target face in the original face image.
  • FIG. 1e shows an example diagram of the correction key points detected from the face correction image in an embodiment of the present application
  • FIG. 1f shows the reverse adjustment and correction key points in an embodiment of the present application.
  • the face correction image contains the detected key points of correction (not all the key points of correction are shown in the figure), and the detected key points of correction clearly mark the person in the forward pose Face contour;
  • the original face image contains the key points of the target face formed after the correction key points are reversely adjusted (the key points of the target face are not shown in the figure), confirm The key points of the target face clearly mark the contour of the face with the head tilted to the right.
  • the face key point detection method provided by the embodiments of the present application can perform posture correction on the original face image through the acquired posture correction information before performing key point detection, so that the existing key point detection network model is used to correct the face after correction. Correct the image for key point detection, so that according to the obtained detection result, the key points of the original image can be obtained in reverse.
  • this embodiment adds the technical realization of face pose correction, so as to ensure that special face images (such as large angles, large At the same time, the detection accuracy of the key point detection is avoided while the detection time of the key point detection is increased, and the detection effect of both real-time and accuracy is achieved.
  • Figure 2a shows a schematic flowchart of another face key point detection method provided by an embodiment of the present application.
  • This embodiment is refined on the basis of the foregoing embodiment.
  • the current pending The posture correction information of the face in the original face image of the measured frame is refined as follows: the face posture information of the original face image in the previous frame of the current frame to be measured is obtained as the person in the current frame to be measured Correction information about the posture of the face in the original face image.
  • this embodiment will correct the face posture in the original face image according to the posture correction information to obtain a face correction image, which may include: taking the original face image and the posture correction information as Input the data and input the image alignment model to output the corrected face correction image.
  • this embodiment will perform reverse posture correction on the correction key points according to the posture correction information to obtain the target face key points of the original face image, which may include: a rotation determined according to the posture correction information Angle, reverse rotation of the correction key point to obtain the target face key point of the original image of the face.
  • the method for detecting key points of a face includes steps S201 to S205.
  • step S201 the original face image of the current frame to be measured is obtained.
  • the current frame to be measured in this step can be obtained from a short video captured in advance, or from a live video captured in real time, and the obtained frame to be tested contains a face image.
  • step S202 the face posture information of the original face image in the previous frame of the current frame to be tested is acquired as the posture correction information of the face in the original face image of the current frame to be tested.
  • this step exemplarily changes the current frame to be tested
  • the face posture information of the original face image in the previous frame is used as the posture correction information, where the face posture information can be understood as the posture presentation information of the face in the three-dimensional space in the original face image, and the face posture
  • the information may be the face pose angles of the face relative to the world coordinate system (right-hand coordinate system) in a three-dimensional space, and the face pose angles may include: pitch angle, yaw angle, and roll angle.
  • the face pose information of the original image of the face in the previous frame can be determined by using the original face image of the frame as the input of the pose network model.
  • the corresponding output of the pose network model can be obtained.
  • the face pose angle of the original face image in the previous frame and its corresponding angle value are used as pose correction information; in addition, the face pose information of the original face image in the previous frame can also be derived from this
  • the key points of the face detected in the original face image of the frame are determined as the input of the pose detection network model.
  • the face pose angle corresponding to the original image of the face in the previous frame can also be obtained in this step.
  • the angle value is used as posture correction information.
  • the face in the original face image of the current frame to be tested is the first appearance (that is, the face does not exist in the original face image of the previous frame), and the preset standard correction information can be directly obtained As the posture correction information of the face in the original image of the face.
  • this embodiment sets the standard correction information as the face pose angle presented in the three-dimensional space coordinates relative to the world coordinate system when the face is presented in the standard orientation. For example, the pose angle of each face at this time can be set The angle value is 0.
  • the following operations from S203 to S205 can also be performed.
  • the obtained face correction image is still the original face image, and based on the face correction
  • the correction key points determined by the image can be directly regarded as the target face key points of the original image of the face.
  • step S203 the original face image and the posture correction information are used as input data, and the image alignment model is input to output a corrected face correction image.
  • the image alignment model can be understood as a model that performs face pose correction on an image containing a face based on related pose correction information. After correcting the face pose based on the image alignment model, a corrected face image relative to the original image of the face can be obtained.
  • the process of realizing the face pose correction can be described as: firstly, analyze and determine the rotation angle required for face pose correction from the input pose correction information, and then determine the face pose correction from the input original face image.
  • the correction area finally, the above-mentioned rotation angle rotation is performed on the pixels in the area to be corrected in the original image of the face, so as to form an outputable face correction image after the rotation.
  • the face pose correction based on the image alignment model in this step is equivalent to a rotation of the image based on the rotation angle.
  • the time for forming the face correction image in this step is spent on the processing time of the entire face key point detection It is almost negligible.
  • step S204 a key point detection network model is used to perform face key point detection on the face correction image to obtain correction key points.
  • the key point detection network model is a pre-trained 3-channel convolutional neural network model.
  • the face correction image can be preprocessed into a 3*70*70 face image, and then the face image is used as a convolution With the input data of the neural network model, the coordinate values of 106 correction key points are finally obtained.
  • the scale of the key point detection network model can be set to any size. This embodiment takes into account the requirements and limitations of computing resources and processing speed on the computer equipment as the execution subject, and it needs to ensure the normal posture of the key points of the face The scale is minimized under the premise of detection accuracy.
  • step S205 the correction key point is reversely rotated according to the rotation angle determined by the posture correction information to obtain the target face key point of the original image of the face.
  • the aforementioned S203 uses the original face image and posture correction information as the input data of the image alignment model. While the image alignment model outputs the face correction image, it can also output the rotation angle required to form the face correction image. The rotation angle may be determined based on posture correction information.
  • the determining operation may be to first obtain the face pose angle in the pose correction information, and the preset standard orientation that the corrected image should have, and then determine the face from the three-dimensional angle by the face pose angle value.
  • the corresponding rotation angle when projected to a two-dimensional plane in space and presented in a standard orientation.
  • the rotation angle output by the image correction model can be obtained, and then the correction key point determined in S204 is reversely rotated by the rotation angle.
  • the coordinate value determined after the reverse rotation is equivalent to the target face key in the original face image The coordinate value of the point.
  • the key points of the target face may be used to set visual special effects on the face in the original face image of the current frame to be measured.
  • the visual special effect setting may include: setting a special effect sticker on a human face, setting a face-changing special effect, and the like.
  • special effect stickers on the face such as setting special effect stickers under the eyes, or setting a long beard under the mouth, etc.
  • face changing such as cutting out the eyes, mouth, and nose from the face Wait and replace it to the face selected by the user, or replace it to the application default replacement face.
  • Figure 2b shows the effect of setting special effect stickers based on the key points of the target face detected in Figure 2a.
  • the position of the face in the original image of the face can be accurately located.
  • it can be determined The positions of the two eyes are shown. Therefore, when the user pre-selects to set the crying visual effect, the relevant special effect function can present the crying special effect in the form of crying special effect stickers under the located eyes.
  • the method for detecting key points of a face refines the process of obtaining the posture correction information of the face in the original image of the face, and at the same time, refines the method of obtaining the face correction image.
  • the key points of the target face are obtained.
  • the technical solution of this embodiment mainly obtains the corresponding face posture information from the original face image of the previous frame as the posture correction information corresponding to the current frame to be measured. This method takes into account the face in the previous frame and the current frame.
  • the relevance of the image ensures the accuracy of the posture correction information, thereby avoiding the cumbersome operations of obtaining special face image samples and sample training in related technologies.
  • this embodiment considers obtaining face correction images through the image alignment model.
  • the detection model in the related technology is used to obtain the key points for correction and reverse rotation to obtain the key points of the target face, which reduces the overall time consumption of the key point detection of the face image, thereby achieving the detection effect of both real-time and accuracy.
  • acquiring the face pose information of the original face image in the previous frame of the current frame to be tested may further include: acquiring the face key detected from the original face image in the previous frame Point; according to the key points of the face, determine the face pose information of the original image of the face in the previous frame.
  • the face key points of the original face image in the previous frame can also be detected.
  • the face key corresponding to the previous frame can be directly obtained.
  • the face pose information of the original image of the face in the previous frame can be obtained through the face orientation network model.
  • the determining the face pose information of the original image of the face in the previous frame according to the face key points includes: normalizing the face key points to obtain the face The normalized coordinates corresponding to the key points; using the normalized coordinates as input data, input the face to the network model, and obtain the face pose of the original image of the face in the previous frame from the output of the face to the network model information.
  • the purpose of normalizing the key points of the face is to improve the accuracy of determining the face posture information.
  • the face orientation network model may be a fully connected network composed of fully connected and activation layers, and may include three fully connected layers and an activation layer. Based on the face orientation network model, assuming the number of key points of the face It is 106 coordinate values. After inputting the 106 coordinate values into the face orientation network model, the angle values of three face attitude angles can be output. The three face attitude angles include yaw angle, pitch angle, and roll angle. This embodiment Finally, the above three face pose angles and their corresponding angle values are used as the face pose correction information in the original face image of the current frame to be measured.
  • the process of normalizing the key points of the human face and obtaining the normalized coordinates corresponding to the key points of the human face may include: determining from the original face image of the previous frame that the human face is included Circumscribed rectangle; determine the upper-left vertex of the circumscribed rectangle as the origin of the set coordinate system; scale the length and width of the circumscribed rectangle to a length of 1 under the set coordinate system to obtain the key to the face after scaling The normalized coordinates of the point.
  • the input data is only a certain amount of values, and the input scale is larger than the input scale of the input data required by the above key point detection network model It is much smaller. Therefore, the time consumed in determining the posture correction information in this part is also negligible in the processing time of the entire face key point detection.
  • the face key point detection of a face in an image frame in this embodiment is equivalent to the realization based on the face orientation network model, the image alignment model, and the key point detection network model in turn, because the face faces the network model
  • the processing time of the image alignment model is almost negligible, and the network scale of the key point detection network model is minimized.
  • this embodiment can effectively ensure the face key point detection of this embodiment Real-time.
  • the input image alignment model to output the corrected face correction image can also be refined into: correcting according to the posture Information, determining the rotation angle of the face in the original face image to the standard orientation; rotating the face in the original face image to the standard orientation through the rotation angle to form a corrected face image after the orientation is corrected.
  • the standard orientation can be set as: the angle value of the corresponding roll angle of the human face presented in the two-dimensional plane image is 0 when restored to the three-dimensional space.
  • the purpose of executing the above steps in this embodiment is to determine a rotation angle based on the posture correction information, and based on the rotation angle, rotate the pixel coordinates corresponding to the face in the original image to form a face correction image, which can ensure the restoration of the face in the face correction image After reaching the three-dimensional space, the angle value of the corresponding roll angle is 0.
  • the rotation angle is a parameter value of a two-dimensional plane.
  • a two-dimensional parameter value based on the three-dimensional posture correction information
  • the acquisition of the rotation angle is related to the projection rotation matrix corresponding to the conversion.
  • this embodiment can firstly based on the face attitude angle conversion in the posture correction information The rotation sequence in two dimensions is then determined, and a rotation angle calculation formula corresponding to the rotation sequence is determined. Finally, a rotation angle can be obtained according to the angle value of the face posture angle in the posture correction information and the rotation angle calculation formula.
  • the rotation angle of the face to the standard orientation in the original image of the face is determined according to the posture correction information.
  • the realization process may include: obtaining the posture angle of the face in the preset posture correction information Determine the rotation angle calculation formula corresponding to the rotation sequence from the preset rotation formula association table; substitute the angle value of the face attitude angle into the rotation angle calculation formula to obtain the face image The rotation angle of the middle face to the standard orientation.
  • the calculation formula for the rotation angle of the rotation angle is mainly derived from the projection conversion matrix required when the face posture angle in the three-dimensional posture correction information is converted to two-dimensional, and the determination of the projection conversion matrix is related to the face posture.
  • the order of rotation of the angles is related. For example, suppose that the face attitude angle is expressed as the pitch angle rotating around the y'axis of the face's own space coordinate system, and the yaw angle rotating around the face's own space coordinate system z'axis.
  • the corresponding projection rotation matrix can be expressed as:
  • the rotation angle calculation formula can be expressed as:
  • x arctan((cos(r)sin(p)sin(t) ⁇ sin(r)cos(t))/cos(r)cos(p)), where x is the rotation angle.
  • this embodiment can pre-set the corresponding rotation angle calculation formula when the face pose angle is converted in different rotation order, and then select the corresponding rotation angle calculation formula as needed, and finally set the angle value of the face pose angle Substituting the rotation angle calculation formula, the current rotation angle can be obtained.
  • the face in the original face image is rotated to a standard orientation through the rotation angle to obtain a corrected face image after orientation correction.
  • the implementation process may include: identifying the face in the original face image Determine the rectangular area containing the human face; use the vertical axis of the coordinate system where the original image of the human face is located as the reference axis, and rotate each pixel in the rectangular area relative to the reference axis. Angle; Obtain a rectangular area with the standard orientation to form a corrected face image after orientation correction.
  • the angle rotation of the pixel coordinates of the area where the face is located in the original image of the face can be considered.
  • This embodiment can recognize that the original image of the face contains the person based on the face blur recognition method in the related technology.
  • each pixel in the rectangular area is rotated by the rotation angle relative to the reference axis, and then a face correction image is formed based on the coordinates of the rotated pixel.
  • the head of the face in the corrected face correction image is presented in the state of the vertical axis of the coordinate system.
  • Fig. 2c shows a block diagram of the realization of face key point detection in an embodiment of the present application.
  • the face picture 22 of the second frame can be considered as the current frame to be tested
  • the face picture 21 of the first frame is the previous frame of the current frame to be tested
  • the face picture 21 of the first frame is shown in Figure 2c.
  • the face key points are known information.
  • the face key points of the face picture 21 in the first frame are used as input data, which is input to the face facing network model 23, and the face facing network model 23 outputs the first frame face picture 21
  • the angle values of the pitch, yaw, and roll of the three face angles of the middle face; then, the angle values of the pitch, yaw, and roll of the three face angles can be used as the posture correction information of the second face picture 22 and used as input
  • the data is input to the image alignment model 24, and the image alignment model 24 outputs the face correction picture 25 and the rotation angle x corresponding to the face picture 22 of the second frame; after that, the face correction picture 25 can be input to the key point detection network as input data Model 26, key point detection network model 26, output 106 correction key points of the face correction picture 25; finally, the 106 correction key points can be reversely rotated by the determined rotation angle x, and the 106th correction point can be obtained after reverse rotation.
  • the target face key points of the face picture 22 of the second frame can be reused for the key point detection of the next frame (the face picture of the third frame).
  • FIG. 3 shows a schematic block diagram of the structure of a face key point detection device provided by an embodiment of the present application.
  • the device is suitable for the case of face key point detection on a face image.
  • the device can be software and/or
  • the hardware is implemented and is generally integrated on a computer device.
  • the device includes: an information acquisition module 31, an image correction module 32, a key point determination module 33, and a key point correction module 34.
  • the information acquisition module 31 is configured to acquire the original face image of the current frame to be tested, and acquire the posture correction information of the face in the original face image of the current frame to be tested.
  • the image correction module 32 is configured to correct the face posture in the original face image according to the posture correction information to obtain a face correction image.
  • the key point determination module 33 is configured to use a key point detection network model to perform face key point detection on the face correction image to obtain correction key points.
  • the key point correction module 34 is configured to perform reverse posture correction on the corrected key point according to the posture correction information to obtain the target face key point of the original image of the face.
  • the device for checking key points of a face provided by an embodiment of the present application can perform posture correction on the original face image through the acquired posture correction information before performing key point detection, thereby adopting the existing key point detection network model to correct the post-correction
  • the face correction image performs key point detection, so that the face key points of the original image can be obtained inversely according to the obtained detection results.
  • this embodiment adds the technical realization of face pose correction, so that it can ensure that special face images (such as large angles, large At the same time, the detection accuracy of the key point detection is avoided while the detection time of the key point detection is increased, and the detection effect of both real-time and accuracy is achieved.
  • the information acquisition module 31 includes:
  • the image acquisition unit is set to acquire the original face image of the current frame to be tested
  • the correction information acquiring unit is configured to acquire the face posture information of the original face image in the previous frame of the current frame to be tested as the posture correction information of the face in the original face image of the current frame to be tested.
  • the correction information acquiring unit includes:
  • the historical information acquiring subunit is configured to acquire the key points of the face detected from the original face image in the previous frame
  • the posture information determining subunit is set to determine the face posture information of the original image of the face in the previous frame according to the key points of the face.
  • the posture information determination subunit may be configured to: normalize the key points of the face, obtain the normalized coordinates corresponding to the key points of the face, and use the normalized coordinates as Input data, input the face to the network model, and obtain the face pose information of the original image of the face in the previous frame from the output of the face to the network model.
  • the posture correction information includes: a face posture angle and a corresponding angle value; the face posture angle includes: a pitch angle, a yaw angle, and a roll angle.
  • the image correction module 32 includes:
  • the face correction unit is configured to use the original face image and the posture correction information as input data, and input the image alignment model to output a corrected face correction image
  • the face correction unit includes:
  • An angle determination subunit configured to determine the rotation angle of the face to the standard orientation in the original image of the face according to the posture correction information
  • the image rotation subunit is configured to rotate the human face in the original image of the human face to a standard orientation through the rotation angle to form a corrected image of the human face after the orientation is corrected.
  • the angle determination subunit may be configured to: obtain the rotation sequence of the face pose angles in the preset posture correction information; determine the rotation sequence corresponding to the rotation sequence from the preset rotation formula association table Rotation angle calculation formula; substitute the angle value of the face posture angle into the rotation angle calculation formula to obtain the rotation angle of the face to the standard orientation in the face image.
  • the image rotation subunit may be configured to: recognize a human face in the original image of the human face, determine a rectangular region containing the human face, and use the vertical axis of the coordinate system where the original image of the human face is located. As a reference axis, each pixel in the rectangular area is rotated by the rotation angle relative to the reference axis; a rectangular area with the standard orientation is obtained to form a corrected face image after orientation correction.
  • the key point correction module 34 may be set as:
  • the correction key point is reversely rotated to obtain the target face key point of the original image of the face.
  • the current frame to be measured is obtained from a short video captured in advance, or from a live video captured in real time;
  • the detected key points of the target face are used to set the visual effects of the face in the original image of the corresponding face.
  • Fig. 4 shows a schematic diagram of the hardware structure of a computer device provided by an embodiment of the present application.
  • the computer device includes a processor and a storage device. At least one instruction is stored in the storage device, and the instruction is executed by the processor, so that the computer device executes the face key point detection method as described in the foregoing method embodiment.
  • the computer equipment may include: a processor 40, a storage device 41, a display screen 42, an input device 43, an output device 44, and a communication device 45.
  • the number of processors 40 in the computer device may be at least one, and one processor 40 is taken as an example in FIG. 4.
  • the number of storage devices 41 in the computer equipment may be at least one, and one storage device 41 is taken as an example in FIG. 4.
  • the processor 40, the storage device 41, the display screen 42, the input device 43, the output device 44, and the communication device 45 of the computer equipment may be connected by a bus or other methods. In FIG. 4, a bus connection is taken as an example.
  • the storage device 41 can be configured to store software programs, computer-executable programs, and modules, such as the program instructions/modules corresponding to the embodiments of the present application (for example, the face key point detection provided in the above embodiments)
  • the storage device 41 may mainly include a storage program area and a storage data area.
  • the storage program area may store an operating device and an application program required for at least one function; the storage data area may store data created according to the use of computer equipment.
  • the storage device 41 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices.
  • the storage device 41 may include a memory remotely provided with respect to the processor 40, and these remote memories may be connected to a computer device through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the display screen 42 is set to display data according to instructions of the processor 40, and is also set to receive touch operations on the display screen 42 and send corresponding signals to the processor 40 or other devices.
  • the display screen 42 when the display screen 42 is an infrared screen, it further includes an infrared touch frame, which is arranged around the display screen 42, and it can also be set to receive infrared signals and transfer the infrared signals Send to the processor 40 or other computer equipment.
  • the communication device 45 is configured to establish a communication connection with other computer equipment, and it may be at least one of a wired communication device and a wireless communication device.
  • the input device 43 can be set to receive input digital or character information, and to generate key signal input related to the user settings and function control of the computer equipment. It can also be a camera set to obtain images and a pickup computer to obtain audio in video data. equipment.
  • the output device 44 may include video computer equipment such as a display screen and audio computer equipment such as a speaker. It should be noted that the composition of the input device 43 and the output device 44 can be set according to actual conditions.
  • the processor 40 executes various functional applications and data processing of the computer equipment by running the software programs, instructions, and modules stored in the storage device 41, that is, realizes the aforementioned face key point detection method.
  • the processor 40 executes at least one program stored in the storage device 41, the following operations are implemented: obtain the original face image of the current frame to be measured, and obtain the face in the original face image of the current frame to be measured Correct the posture of the face in the original image of the face according to the posture correction information to obtain a face correction image; use the key point detection network model to perform face keying on the face correction image Point detection to obtain correction key points; performing reverse posture correction on the correction key points according to the posture correction information to obtain the target face key points of the original image of the face.
  • the embodiments of the present application also provide a computer-readable storage medium.
  • the computer device can execute the method for detecting key points of a human face as described in the foregoing embodiment.
  • the face key point detection method described in the foregoing embodiment includes: obtaining an original face image of a current frame to be tested, and obtaining posture correction information of the face in the original face image of the current frame to be tested; Correct the face pose in the original image of the face according to the posture correction information to obtain a face correction image; adopt a key point detection network model to perform face key point detection on the face correction image to obtain correction Key points; performing reverse posture correction on the corrected key points according to the posture correction information to obtain the target face key points of the original image of the face.
  • the computer software product can be stored in a computer-readable storage medium, such as a floppy disk, Read-Only Memory (ROM), Random Access Memory (RAM), Flash memory (FLASH), hard disk or optical disk, etc., including several instructions to make a computer device (can be a robot, personal A computer, a server, or a network device, etc.) execute the method for detecting key points of a face described in any embodiment of the present application.
  • a computer device can be a robot, personal A computer, a server, or a network device, etc.
  • each part of this application can be implemented by hardware, software, firmware, or a combination thereof.
  • multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device.
  • a logic gate circuit configured to implement logic functions for data signals Discrete logic circuits, ASICs with suitable combinational logic gate circuits, Programmable Gate Array (PGA), Field Programmable Gate Array (FPGA), etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • General Health & Medical Sciences (AREA)
  • Human Computer Interaction (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

本申请实施例公开了人脸关键点检测方法、装置、设备及存储介质。该方法包括：获取当前待测帧的人脸原始图像，并获取当前待测帧的人脸原始图像中人脸的姿态纠正信息；根据姿态纠正信息对人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；采用关键点检测网络模型，对人脸纠正图像进行人脸关键点检测，获得纠正关键点；根据姿态纠正信息对纠正关键点进行逆向姿态纠正，以获得人脸原始图像的目标人脸关键点。

Description

人脸关键点检测方法、装置、设备及存储介质
本申请要求在2019年3月29日提交中国专利局、申请号为201910252374.5的中国专利申请的优先权，该申请的全部内容通过引用结合在本申请中。
技术领域
本申请实施例涉及图像识别技术领域，例如人脸关键点检测方法、装置、设备及存储介质。
背景技术
在计算机视觉领域，人脸关键点检测一直受到学术界和工业界的广泛关注，其主要功能是准确地定位出人脸上的关键点(比如眼睛、鼻子、嘴角、脸部轮廓点等)，为后续的人脸图像处理(如人脸对齐、人脸识别)做准备，目前，人脸关键点检测已在生物信息验证、监控安防、视频直播等应用场景中均起到较重要的作用。
相关技术中多采用神经网络模型来实现人脸关键点检测，为保证检测结果的精度常采用较大规模的神经网络模型，但因关键点检测执行终端的计算资源为达到实时处理的效果，通常又会对神经网络模型的大小进行限制，由此便导致了执行终端对包含大角度、大姿态(45度仰头，45度低头，歪头至90度等情况)人脸的图像进行人脸关键点检测的效果不佳。
为提高包含大角度、大姿态人脸的关键点检测效果，通常采用的方法是对加入更多大角度、大姿态的人脸训练样本对当前采用的神经网络模型进行训练，但在实际操作中，这种大角度大姿态的人脸训练样本并不容易获取，且在限制网络规模的前提下，样本复杂性的增加往往会导致神经网络模型在对正常人脸图像检测时的精度下降，此外，在为保证检测精度增大网络规模的情况下，又会增加图像帧的检测时间，无法保证检测的实时性要求。
发明内容
本申请实施例提供了人脸关键点检测方法、装置、设备及存储介质，以优化相关技术中人脸关键点的检测方法，保证实时性的同时提高人脸关键点的检测精度。
第一方面，本申请实施例提供了一种人脸关键点检测方法，包括：
获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；
根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；
采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点；
根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
第二方面，本申请实施例提供一种人脸关键点检测装置，包括：
信息获取模块，设置为获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；
图像纠正模块，设置为根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；
关键点确定模块，设置为采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点；
关键点纠正模块，设置为根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
第三方面，本申请实施例提供了一种计算机设备，包括：
至少一个处理器；
存储装置，设置为存储至少一个程序；
所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现本申请第一方面实施例提供的人脸关键点检测方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现本申请第一方面实施例提供的人脸关键点检测方法。
附图说明
图1a给出了本申请一实施例提供的一种人脸关键点检测方法的流程示意图；
图1b给出了本申请一实施例中人脸原始图像的示例图；
图1c给出了本申请一实施例中纠正人脸原始图像后所形成人脸纠正图像的 示例图；
图1d给出了本申请一实施例中纠正关键点的实现示意图；
图1e给出了本申请一实施例中从人脸纠正图像中检测到的纠正关键点的示例图；
图1f给出了本申请一实施例中逆向调整纠正关键点后所形成目标人脸关键点的示例图；
图2a给出了本申请一实施例提供的另一种人脸关键点检测方法的流程示意图；
图2b给出了基于图2a所检测目标人脸关键点进行特效贴纸设置的效果展示图；
图2c给出了本申请一实施例中人脸关键点检测的实现框图；
图3给出了本申请一实施例提供的一种人脸关键点检测装置的结构方框示意图；
图4给出了本申请一实施例提供的一种计算机设备的硬件结构示意图。
具体实施方式
图1a给出了本申请一实施例提供的一种人脸关键点检测方法的流程示意图，该方法适用于对人脸图像进行人脸关键点检测的情况，该方法可以由人脸关键点检测装置执行，其中，该装置可由软件和/或硬件实现，并一般作为集成在计算机设备上，例如可作为插件集成在具备人机交互的应用软件中。
需要说明的是，本实施例所提供人脸关键点检测方法的执行主体可以是数据信息交互中作为请求终端的计算机设备，所述计算机设备可以包括：移动终端、平板电脑以及台式电脑等。此外，本实施例可以对包含人物图像的视频帧进行人脸关键点检测。
如图1a所示，本申请实施例提供的一种人脸关键点检测方法，包括步骤S101至步骤S104。
在步骤S101中，获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
在本实施例中，所述当前待测帧可指当前待进行人脸关键点检测的图像帧，示例性地，根据本实施例所提供方法的适用场景，所述当前待测帧可以为从处 于播放状态短视频中获取的视频帧，其中所述短视频可以是预先捕获录制形成的视频；所述当前待测帧也可为从实时捕获的直播视频中获取的视频帧。
需要说明的是，因本实施例主要用于实现人脸关键点的检测，因此可预先设定所获取的当前待测帧中应当存在人脸图像。本实施例可以将当前待测帧中包含的人脸图像记为人脸原始图像，所述人脸原始图像可看做进行人脸关键点检测的检测对象。
在本实施例中，当前待测帧的人脸原始图像中人脸可能处于非正向的特殊姿态情况，如歪头、偏转以及俯仰等，为准确高效实现对特殊姿态人脸的关键点检测，本实施例考虑在进行关键点检测前先对人脸的姿态进行纠正。所述姿态纠正信息可理解为对人脸原始图像中人脸的姿态进行纠正所需的纠正信息。所述姿态纠正信息可以是姿态纠正的像素坐标偏移信息，还可以是人脸在三维空间中具备的姿态信息，也可以是已进行关键点检测的历史帧中的人脸在三维空间中具备的姿态信息。
本实施例中，可以通过分析确定当前待测帧的人脸原始图像中人脸在三维空间中的人脸姿态信息，将获得的当前待测帧中人脸的姿态信息作为该姿态纠正信息；还可以通过确定前一帧的人脸原始图像中人脸在三维空间中的人脸姿态信息，来将该前一帧中人脸姿态信息关联作为该姿态纠正信息；也可以通过分析确定构成当前待测帧中人脸原始图像的每个像素点的像素坐标偏移信息，将像素坐标偏移信息作为该姿态纠正信息。
可以知道的是，本实施例将人脸在三维空间中的人脸姿态信息确定为姿态纠正信息。示例性的，所述人脸姿态信息可看作人脸在世界空间坐标系下相对人脸所属空间坐标系时，从俯仰、偏航以及滚转等方面所呈现出的角度信息。由此，本实施例将人脸姿态信息作为姿态纠正信息时，所述姿态纠正信息可包括：人脸姿态角及相应的角度值；所述人脸姿态角包括：俯仰角、偏航角以及滚转角。
在步骤S102中，根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像。
在本实施例中，对人脸姿态进行纠正可看作对人脸在当前待测帧中所显示位置的矫正调整。示例性的，在获取姿态纠正信息后，就可基于姿态纠正信息 包含的具体值来进行相应的纠正操作。
示例性地，在姿态纠正信息为像素坐标偏移信息的情况下，本步骤可以获得像素坐标偏移信息中像素点对应的坐标偏移值，然后直接对当前待测帧的人脸原始图像中每个像素点的坐标进行所述坐标偏移值的偏移调整，从而将像素点坐标调整后形成的图像记为人脸纠正图像。
示例性地，在姿态纠正信息为人脸姿态信息的情况下，本步骤可以获得人脸姿态信息中人脸姿态角及其相应角度值，然后获取标准朝向时人脸姿态角的标准角度信息，之后基于人脸姿态角的角度值及标准角度信息，可以获得人脸原始图像纠正调整为标准朝向时的旋转角度，最终可将人脸原始图像中的像素点进行所述旋转角度的旋转，从而获得旋转后的人脸纠正图像。
在本实施例中，图1b给出了本申请一实施例中人脸原始图像的示例图；图1c给出了本申请一实施例中纠正人脸原始图像后所形成人脸纠正图像的示例图；如图1b所示，可以看出图中所呈现人脸原始图像100的人脸姿态以屏幕视角为向右歪头，如图1c所示，可以看出对图1b中人脸原始图像100基于本步骤进行人脸姿态纠正后形成了以正向姿态呈现的人脸纠正图像101。
在步骤S103中，采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点。
在本实施例中，所述关键点检测网络模型可为一个预先训练的规模适中的深度卷积神经网络模型，所述纠正关键点可理解为从人脸纠正图像中检测到的人脸关键点。其中，本步骤可以将人脸纠正图像作为输入数据，输入该关键点检测网络模型，然后获得关键点检测网络模型输出的纠正关键点。
示例性地，图1d给出了本申请一实施例中确定纠正关键点的实现示意图，如图1d所示，本步骤可以先经过步骤11中的描述对人脸纠正图像10进行预处理，获得设定长宽的待输入图像，然后输入训练后形成的深度卷积神经网络模型14，最终获得输出的以坐标形式表示的纠正关键点。
在步骤S104中，根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
在本实施例中，所述逆向姿态纠正可相当于将从人脸纠正图像中检测到的纠正关键点逆向还原到人脸原始图像中。根据上述S102的表述，可知通过姿态 纠正信息纠正获得了人脸纠正图像，本步骤则再次根据姿态纠正信息对纠正关键点进行逆向姿态纠正，以此获得到纠正关键点在人脸原始图像中对应的目标人脸关键点。
示例性地，在姿态纠正信息为像素坐标偏移信息的情况下，本步骤同样可以获得像素坐标偏移信息中像素点对应的坐标偏移值，之后直接对获得的纠正检测点进行所述坐标偏移值的逆向偏移，逆向偏移后获得的坐标值就可用来表示人脸原始图像中的目标人脸关键点。
示例性地，在姿态纠正信息为人脸姿态信息的情况下，本步骤可以获得实现上述S102时确定出的旋转角度，然后对纠正检测点基于设定参照轴进行所述旋转角度的逆向旋转，同样，逆向旋转后的对应的坐标值也可用来标识人脸原始图像中的目标人脸关键点。
接上述S102的示例图，图1e给出了本申请一实施例中从人脸纠正图像中检测到的纠正关键点的示例图，图1f给出了本申请一实施例中逆向调整纠正关键点后所形成目标人脸关键点的示例图。如图1e所示，可以看出，人脸纠正图像中包含了检测到的纠正关键点(图中未呈现出全部纠正关键点)，检测出的纠正关键点清晰标示了正向姿态下的人脸轮廓；又如图1f所示，可以看出，人脸原始图像中包含了对纠正关键点逆向调整后形成的目标人脸关键点(图中未呈现出全部目标人脸关键点)，确定的目标人脸关键点清晰标示了向右歪头状态下的人脸轮廓。
本申请实施例提供的人脸关键点检测方法，能够在进行关键点检测前先通过获取的姿态纠正信息对人脸原始图像进行姿态矫正，从而采用已有关键点检测网络模型对矫正后人脸纠正图像进行关键点检测，由此根据获得的检测结果就能够逆向获得原始图像的人脸关键点。与相关技术中的检测方法相比，本实施例增加了人脸姿态矫正的技术实现，从而能够在不增加关键点检测网络模型规模的情况下，保证对特殊人脸图像(如大角度、大姿态)进行关键点检测时的检测精度，同时避免了关键点检测的检测时间的增大，进而达到了兼具实时性和准确性的检测效果。
图2a给出了本申请一实施例提供的另一种人脸关键点检测方法的流程示意图，本实施例在上述实施例基础上进行细化，在本实施例中，将获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息，细化为：获取所述当前待测帧 的前一帧中人脸原始图像的人脸姿态信息，作为所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
同时，本实施例将根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像，可包括：将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像。
此外，本实施例将根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点，可包括：根据所述姿态纠正信息确定的旋转角度，对所述纠正关键点进行逆向旋转，以获得所述人脸原始图像的目标人脸关键点。
如图2a所示，本实施例提供的人脸关键点检测方法，包括步骤S201至步骤S205。
在步骤S201中，获取当前待测帧的人脸原始图像。
示例性地，本步骤中的当前待测帧可以从预先捕获的短视频中获取，也可以从实时捕获的直播视频中获取，且所获取的当前待测帧中包含了人脸图像。
在步骤S202中，获取所述当前待测帧的前一帧中人脸原始图像的人脸姿态信息，作为所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
在本实施例中，考虑到所获取当前待测帧的人脸原始图像中人脸与前一帧中的同一张人脸其展示姿态存在关联性，本步骤示例性的将当前待测帧的前一帧中人脸原始图像的人脸姿态信息作为所述姿态纠正信息，其中，所述人脸姿态信息可理解为人脸原始图像中人脸在三维空间下姿态呈现信息，所述人脸姿态信息可以是人脸在三维空间下相对世界坐标系(右手坐标系)呈现的人脸姿态角，所述人脸姿态角可以包括：俯仰角、偏航角以及滚转角。
示例性的，所述前一帧中人脸原始图像的人脸姿态信息，可以将该帧的人脸原始图像作为姿态网络模型的输入来确定，本步骤由此可获取姿态网络模型输出的对应前一帧中人脸原始图像的人脸姿态角及其相应角度值，并将其作为姿态纠正信息；此外，所述前一帧中人脸原始图像的人脸姿态信息，也可将从该帧人脸原始图像中检测出的人脸关键点作为姿态检测网络模型的输入来确定，本步骤同样可获取姿态网络模型输出的对应前一帧中人脸原始图像的人脸姿态角及其相应角度值作为姿态纠正信息。
需要说明的是，在当前待测帧的人脸原始图像中的人脸为首次出现的情况(即该人脸不存在与前一帧人脸原始图像)，可直接获取预设的标准纠正信息作为人脸原始图像中人脸的姿态纠正信息。示例性地，本实施例设定所述标准纠正信息为人脸以标准朝向呈现时在三维空间坐标下相对世界坐标系呈现的人脸姿态角，如，可设定此时每个人脸姿态角的角度值为0。
此外，可以理解的是，上述标准纠正信息作为姿态纠正信息时，同样可以执行下述S203至S205的操作，该种情况下，获得的人脸纠正图像仍为人脸原始图像，而基于人脸纠正图像确定出的纠正关键点实则可直接看作人脸原始图像的目标人脸关键点。
在步骤S203中，将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像。
在本实施例中，所述图像对齐模型可理解为一个对包含人脸的图像基于相关的姿态纠正信息进行人脸姿态纠正的模型。基于所述图像对齐模型进行人脸姿态纠正后，可以获得相对人脸原始图像的人脸纠正图像。
对于图像对齐模型，其实现人脸姿态纠正的过程可描述为：首先从输入的姿态纠正信息中分析确定出人脸姿态纠正时所需的旋转角度，然后从输入的人脸原始图像中确定待纠正区域，最终，对人脸原始图像中待纠正区域的像素点进行上述旋转角度的旋转，从而在旋转后形成可输出的人脸纠正图像。
需要说明的是，本步骤基于图像对齐模型实现的人脸姿态纠正，相当于基于旋转角度对图像进行了一个旋转，该步骤形成人脸纠正图像的时间消耗在整个人脸关键点检测的处理时间中几乎可以忽略不计。
在步骤S204中，采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点。
示例性地，关键点检测网络模型为预先训练的3通道卷积神经网络模型，可以先将人脸纠正图像预处理成3*70*70的人脸图像，然后将该人脸图像作为卷积神经网络模型的输入数据，最终获得输出的106个纠正关键点的坐标值。
一般地，所述关键点检测网络模型的规模可设置任意大小，本实施例考虑到作为执行主体的计算机设备上计算资源以及处理速度的要求和限制，其需要能保证正常姿态的人脸关键点检测精度的前提下使得规模最小化。
在步骤S205中，根据所述姿态纠正信息确定的旋转角度，对所述纠正关键点进行逆向旋转，以获得所述人脸原始图像的目标人脸关键点。
在本实施例中，上述S203将人脸原始图像和姿态纠正信息作为图像对齐模型的输入数据，图像对齐模型输出人脸纠正图像的同时，还可以输出形成人脸纠正图像所需的旋转角度，所述旋转角度可基于姿态纠正信息确定。
示例性地，该确定操作可以是先获得姿态纠正信息中的人脸姿态角，以及预先设置的纠正后图像应当具有的标准朝向，然后通过人脸姿态角角度值就可确定出人脸从三维空间下投影到二维平面且以标准朝向呈现时所对应的旋转角度。本步骤可以获取图像纠正模型输出的旋转角度，然后对上述S204确定的纠正关键点进行所述旋转角度的逆向旋转，最终，逆向旋转后确定的坐标值相当于人脸原始图像中目标人脸关键点的坐标值。
需要说明的是，所述目标人脸关键点可用于对当前待测帧的人脸原始图像中人脸进行视觉特效设置。示例性的，所述视觉特效设置可以包括：在人脸上设置特效贴纸以及换脸特效设置等。对于在人脸上设置特效贴纸，如在眼睛下面设置大哭的特效贴纸，又如在嘴巴下面设置长长的胡须等；对于换脸特效设置，如从人脸中抠出眼睛、嘴巴、鼻子等并将其置换到用户选定人脸上，或置换到应用默认置换人脸上。
图2b给出了基于图2a所检测目标人脸关键点进行特效贴纸设置的效果展示图。如图2b所示，基于本实施例检测出的目标人脸关键点，能够准确定位到人脸在人脸原始图像中的位置，示例性的，基于目标人脸关键点的坐标值，可以确定出两只眼睛的所在位置，由此，在用户预先选取设置哭的视觉特效的情况下，相关特效功能就可在定位出的眼睛下面以大哭特效贴纸的形式呈现出哭的特效。
本申请实施例提供的一种人脸关键点检测方法，细化了人脸原始图像中人脸的姿态纠正信息的获取过程，同时，细化了人脸纠正图像的获得方式，此外，还细化了目标人脸关键点的获得方式。本实施例的技术方案，主要通过从前一帧的人脸原始图像中获取相应的人脸姿态信息来作为当前待测帧对应的姿态纠正信息，该方式考虑了前一帧与当前帧中人脸图像的关联性，保证了姿态纠正信息的准确度，从而避免了相关技术中对特殊人脸图像样本获取及样本训练的繁琐操作，同时，本实施例考虑通过图像对齐模型获得人脸纠正图像，之后在 采用相关技术中的检测模型获得纠正关键点并逆向旋转获得目标人脸关键点，降低了人脸图像关键点检测的整体消耗时间，进而达到了兼具实时性和准确性的检测效果。
在一实施例中，获取所述当前待测帧的前一帧中人脸原始图像的人脸姿态信息，还可包括：获取从所述前一帧中人脸原始图像检测到的人脸关键点；根据所述人脸关键点，确定所述前一帧中人脸原始图像的人脸姿态信息。
可以理解的是，基于本实施例提供的上述人脸关键点检测方法，同样可以检测出前一帧中人脸原始图像的人脸关键点，本步骤可直接获取到前一帧对应的人脸关键点，并基于人脸关键点，通过人脸朝向网络模型，即可获得前一帧中人脸原始图像的人脸姿态信息。
在一实施例中，所述根据所述人脸关键点，确定所述前一帧中人脸原始图像的人脸姿态信息，包括：归一化所述人脸关键点，获得所述人脸关键点对应的归一化坐标；将所述归一化坐标作为输入数据，输入人脸朝向网络模型，从人脸朝向网络模型的输出获得所述前一帧中人脸原始图像的人脸姿态信息。
在本实施例中，对人脸关键点归一化的目的在于提高人脸姿态信息确定的准确度。同时，所述人脸朝向网络模型可以是一个由全连接与激活层组成的全连接网络，可以包含三层全连接和一个激活层，基于该人脸朝向网络模型，假设人脸关键点的数量为106个坐标值，将106个坐标值输入该人脸朝向网络模型后，可以输出三个人脸姿态角的角度值，三个人脸姿态角包括偏航角、俯仰角以及滚转角，本实施例最终将上述三个人脸姿态角及其相应角度值作为当前待测帧的人脸原始图像中人脸的姿态纠正信息。
示例性地，上述归一化所述人脸关键点，获得所述人脸关键点对应的归一化坐标的过程可以包括：从所述前一帧的人脸原始图像中确定包含人脸的外接矩形；将所述外接矩形的左上角顶点确定为设定坐标系的原点；将所述外接矩形的长宽在所述设定坐标系下放缩至长度为1，获得放缩后人脸关键点的归一化坐标。
可以知道的是，本实施例中采用人脸朝向网络模型进行人脸姿态信息确定时，其输入数据仅为一定量的数值，其输入规模比上述关键点检测网络模型所需输入数据的输入规模要小很多，因此，本部分确定姿态纠正信息的时间消耗 在整个人脸关键点检测的处理时间中同样可以忽略不计。
基于上述表述，可知本实施例的对于一个图像帧中人脸的人脸关键点检测相当于依次基于人脸朝向网络模型、图像对齐模型以及关键点检测网络模型来实现，因人脸朝向网络模型和图像对齐模型处理时间几乎可以忽略不计，而关键点检测网络模型的网络规模进行了最小化限制，与相关技术中的解决方法相比，本实施例可以有效保证本实施例人脸关键点检测的实时性。
在一实施例中，对于将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像，也可细化为：根据所述姿态纠正信息，确定所述人脸原始图像中人脸到标准朝向的旋转角度；将所述人脸原始图像中的人脸通过所述旋转角度旋转至标准朝向，形成朝向纠正后的人脸纠正图像。
在本实施例中，所述标准朝向可设定为：呈现在二维平面图像中的人脸，其还原到三维空间下所对应滚转角的角度值为0。本实施例上述步骤的执行目的在于通过姿态纠正信息确定一个旋转角度，基于该旋转角度对原始图像中人脸对应的像素坐标进行旋转形成人脸纠正图像，能够保证人脸纠正图像中的人脸还原到三维空间后，其所对应滚转角的角度值为0。
通过上述描述，可知旋转角度为一个二维平面的参数值，基于三维形式的姿态纠正信息确定一个适用二维的参数值时，需要考虑将三维姿态纠正信息中的人脸姿态角转换到二维下，且可认为旋转角度的获取与转换时对应的投影旋转矩阵有关，考虑人脸姿态角转换先后顺序对投影旋转矩阵的影响，因此本实施例首先可以基于姿态纠正信息中人脸姿态角转换到二维时的旋转先后顺序，然后确定一个对应该旋转先后顺序的旋转角度计算公式，最终可根据姿态纠正信息中人脸姿态角的角度值及旋转角度计算公式获得一个旋转角度。
在一实施例中，根据所述姿态纠正信息，确定所述人脸原始图像中人脸到标准朝向的旋转角度，其实现过程可以包括：获取预先设置的所述姿态纠正信息中人脸姿态角的旋转先后顺序；从预设旋转公式关联表中确定所述旋转先后顺序对应的旋转角度计算公式；将所述人脸姿态角的角度值代入所述旋转角度计算公式，获得所述人脸图像中人脸到标准朝向的旋转角度。
在本实施例中，旋转角度的旋转角度计算公式主要基于三维姿态纠正信息 中的人脸姿态角转换到二维下时所需的投影转换矩阵推导获得，而投影转换矩阵的确定与人脸姿态角的旋转先后顺序有关，示例性地，假设人脸姿态角分别表示为绕人脸自身空间坐标系y’轴旋转的俯仰角pitch、绕人脸自身空间坐标系z’轴旋转的偏航角yaw以及绕人脸自身空间坐标系x’轴旋转的滚转角roll，且pitch、yaw和roll的角度值分别为p,t和r，以人脸姿态角的旋转先后顺序为yaw→pitch→roll为例，其对应的投影旋转矩阵可表示为：
Figure PCTCN2020079493-appb-000001
此时，可基于上述旋转矩阵以及标准朝向时设定的滚转角为0，就可推导出相应的旋转角度计算公式，该旋转角度计算公式可表示为：
x＝arctan((cos(r)sin(p)sin(t)–sin(r)cos(t))/cos(r)cos(p))，其中x为旋转角度。
可以知道的是，本实施例可预先设置人脸姿态角以不同旋转先后顺序转换时对应的旋转角度计算公式，然后根据需要选定相应的旋转角度计算公式，最终将人脸姿态角的角度值代入旋转角度计算公式，就可获得当前的旋转角度。
在一实施例中，将所述人脸原始图像中人脸通过所述旋转角度旋转至标准朝向，获得朝向纠正后的人脸纠正图像，其实现过程可以包括：识别所述人脸原始图像中的人脸，确定包含所述人脸的矩形区域；以所述人脸原始图像所在坐标系的纵轴为参照轴，将所述矩形区域中每个像素点相对所述参照轴旋转所述旋转角度；获得具备所述标准朝向的矩形区域，形成朝向纠正后的人脸纠正图像。
在本实施例中，可以考虑对人脸原始图像中人脸所在区域进行像素点坐标的角度旋转，本实施例可基于相关技术中的人脸模糊识别方法，识别出人脸原始图像中包含人脸的矩形区域，然后对矩形区域中每个像素点相对参照轴进行旋转角度的旋转，然后基于旋转后的像素点坐标形成人脸纠正图像。
基于上述描述，可知假设人脸原始图像中人脸存在歪头这样的特殊姿态，则朝向纠正后的人脸纠正图像中人脸的头部则以所在坐标系的纵轴的状态呈现。
示例性地，图2c给出了本申请一实施例中人脸关键点检测的实现框图。如 图2c所示，可认为第2帧人脸图片22为当前待测帧，第1帧人脸图片21为当前待测帧的前一帧，图2c中将第1帧人脸图片21的人脸关键点作为已知信息，首先第1帧人脸图片21的人脸关键点作为输入数据，输入至人脸朝向网络模型23，人脸朝向网络模型23则输出第1帧人脸图片21中人脸的三个人脸姿态角pitch、yaw和roll的角度值；然后，三个人脸姿态角pitch、yaw和roll的角度值可作为第2帧人脸图片22的姿态纠正信息，并作为输入数据输入至图像对齐模型24，图像对齐模型24则输出第2帧人脸图片22对应的人脸纠正图片25和旋转角度x；之后，人脸纠正图片25可作为输入数据输入至关键点检测网络模型26，关键点检测网络模型26，则输出人脸纠正图片25的106个纠正关键点；最终，106个纠正关键点可以通过已确定的旋转角度x进行逆向旋转，逆向旋转后获得106个第2帧人脸图片22的目标人脸关键点。
上述描述可依次循环，如，第2帧人脸图片22的目标人脸关键点又可重新用于下一帧(第3帧人脸图片)的关键点检测。
图3给出了本申请一实施例提供的一种人脸关键点检测装置的结构方框示意图，该装置适用于对人脸图像进行人脸关键点检测的情况，该装置可由软件和/或硬件实现，并一般集成在计算机设备上，如图3所示，该装置包括：信息获取模块31、图像纠正模块32、关键点确定模块33以及关键点纠正模块34。
其中，信息获取模块31，设置为获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
图像纠正模块32，设置为根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像。
关键点确定模块33，设置为采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点。
关键点纠正模块34，设置为根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
本申请实施例提供的一种人脸关键点检查装置，能够在进行关键点检测前先通过获取的姿态纠正信息对人脸原始图像进行姿态矫正，从而采用已有关键点检测网络模型对矫正后人脸纠正图像进行关键点检测，由此根据获得的检测结果就能够逆向获得原始图像的人脸关键点。与相关技术中的检测方法相比， 本实施例增加了人脸姿态矫正的技术实现，从而能够在不增加关键点检测网络模型规模的情况下，保证对特殊人脸图像(如大角度、大姿态)进行关键点检测时的检测精度，同时避免了关键点检测的检测时间的增大，进而达到了兼具实时性和准确性的检测效果。
在一实施例中，信息获取模块31，包括：
图像获取单元，设置为获取当前待测帧的人脸原始图像；
纠正信息获取单元，设置为获取所述当前待测帧的前一帧中人脸原始图像的人脸姿态信息，作为所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
在一实施例中，纠正信息获取单元包括：
历史信息获取子单元，设置为获取从所述前一帧中人脸原始图像检测到的人脸关键点；
姿态信息确定子单元，设置为根据所述人脸关键点，确定所述前一帧中人脸原始图像的人脸姿态信息。
在一实施例中，所述姿态信息确定子单元，可以设置为：归一化所述人脸关键点，获得所述人脸关键点对应的归一化坐标；将所述归一化坐标作为输入数据，输入人脸朝向网络模型，从人脸朝向网络模型的输出获得所述前一帧中人脸原始图像的人脸姿态信息。
在一实施例中，所述姿态纠正信息包括：人脸姿态角及相应的角度值；所述人脸姿态角包括：俯仰角、偏航角以及滚转角。
在一实施例中，图像纠正模块32，包括：
人脸纠正单元，设置为将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像
在一实施例中，人脸纠正单元，包括：
角度确定子单元，设置为根据所述姿态纠正信息，确定所述人脸原始图像中人脸到标准朝向的旋转角度；
图像旋转子单元，设置为将所述人脸原始图像中的人脸通过所述旋转角度旋转至标准朝向，形成朝向纠正后的人脸纠正图像。
在一实施例中，角度确定子单元，可以设置为：获取预先设置的所述姿态纠正信息中人脸姿态角的旋转先后顺序；从预设旋转公式关联表中确定所述旋转先后顺序对应的旋转角度计算公式；将所述人脸姿态角的角度值代入所述旋转角度计算公式，获得所述人脸图像中人脸到标准朝向的旋转角度。
在一实施例中，图像旋转子单元，可以设置为：识别所述人脸原始图像中的人脸，确定包含所述人脸的矩形区域；以所述人脸原始图像所在坐标系的纵轴为参照轴，将所述矩形区域中每个像素点相对所述参照轴旋转所述旋转角度；获得具备所述标准朝向的矩形区域，形成朝向纠正后的人脸纠正图像。
在一实施例中，关键点纠正模块34，可以设置为：
根据所述姿态纠正信息确定的旋转角度，对所述纠正关键点进行逆向旋转，以获得所述人脸原始图像的目标人脸关键点。
在一实施例中，所述当前待测帧从预先捕获的短视频中获取，或者从实时捕获的直播视频中获取；
检测到的目标人脸关键点用于所对应人脸原始图像中人脸的视觉特效设置。
图4给出了本申请一实施例提供的一种计算机设备的硬件结构示意图，该计算机设备包括：处理器和存储装置。存储装置中存储有至少一条指令，且指令由所述处理器执行，使得所述计算机设备执行如上述方法实施例所述的人脸关键点检测方法。
参照图4，该计算机设备可以包括：处理器40、存储装置41、显示屏42、输入装置43、输出装置44以及通信装置45。该计算机设备中处理器40的数量可以是至少一个，图4中以一个处理器40为例。该计算机设备中存储装置41的数量可以是至少一个，图4中以一个存储装置41为例。该计算机设备的处理器40、存储装置41、显示屏42、输入装置43、输出装置44以及通信装置45可以通过总线或者其他方式连接，图4中以通过总线连接为例。
存储装置41作为一种计算机可读存储介质，可设置为存储软件程序、计算机可执行程序以及模块，如本申请实施例对应的程序指令/模块(例如，上述实施例所提供人脸关键点检测装置中的信息获取模块31、图像纠正模块32、关键点确定模块33以及关键点纠正模块34等)。存储装置41可主要包括存储程序 区和存储数据区，其中，存储程序区可存储操作装置、至少一个功能所需的应用程序；存储数据区可存储根据计算机设备的使用所创建的数据等。此外，存储装置41可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实例中，存储装置41可包括相对于处理器40远程设置的存储器，这些远程存储器可以通过网络连接至计算机设备。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
一般而言，显示屏42设置为根据处理器40的指示显示数据，还设置为接收作用于显示屏42的触摸操作，并将相应的信号发送至处理器40或其他装置。在一实施例中，在显示屏42为红外屏的情况下，其还包括红外触摸框，该红外触摸框设置在显示屏42的四周，其还可以设置为接收红外信号，并将该红外信号发送至处理器40或者其他计算机设备。
通信装置45，设置为与其他计算机设备建立通信连接，其可以是有线通信装置和无线通信装置中至少之一。
输入装置43可设置为接收输入的数字或者字符信息，以及产生与计算机设备的用户设置以及功能控制有关的键信号输入，还可以是设置为获取图像的摄像头以及获取视频数据中音频的拾音计算机设备。输出装置44可以包括显示屏等视频计算机设备以及扬声器等音频计算机设备。需要说明的是，输入装置43和输出装置44的组成可以根据实际情况设定。
处理器40通过运行存储在存储装置41中的软件程序、指令以及模块，从而执行计算机设备的各种功能应用以及数据处理，即实现上述的人脸关键点检测方法。
示例性的，处理器40执行存储装置41中存储的至少一个程序时，实现如下操作：获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点；根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
本申请实施例还提供一种计算机可读存储介质，所述存储介质中的程序由计算机设备的处理器执行时，使得计算机设备能够执行如上述实施例所述的人脸关键点检测方法。示例性的，上述实施例所述的人脸关键点检测方法包括：获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点；根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
需要说明的是，对于装置、计算机设备、存储介质实施例而言，由于其与方法实施例基本相似，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
通过以上关于实施方式的描述，所属领域的技术人员可以清楚地了解到，本申请可借助软件及必需的通用硬件来实现，当然也可以通过硬件实现，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对相关技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品可以存储在计算机可读存储介质中，如计算机的软盘、只读存储器(Read-Only Memory,ROM)、随机存取存储器(Random Access Memory,RAM)、闪存(FLASH)、硬盘或光盘等，包括若干指令用以使得一台计算机设备(可以是机器人，个人计算机，服务器，或者网络设备等)执行本申请任意实施例所述的人脸关键点检测方法。
值得注意的是，上述人脸关键点检测装置中，所包括的各个单元和模块只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。
应当理解，本申请的各部分可以用硬件、软件、固件或它们的组合来实现。在上述实施方式中，多个步骤或方法可以用存储在存储器中且由合适的指令执行装置执行的软件或固件来实现。例如，如果用硬件来实现，和在另一实施方式中一样，可用本领域公知的下列技术中的任一项或他们的组合来实现：具有设置为对数据信号实现逻辑功能的逻辑门电路的离散逻辑电路，具有合适的组合逻辑门电路的专用集成电路，可编程门阵列(Programmable Gate Array，PGA)， 现场可编程门阵列(Field Programmable Gate Array，FPGA)等。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本申请的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不一定指的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在至少一个实施例或示例中以合适的方式结合。

Claims (14)

  1. 一种人脸关键点检测方法，包括：
    获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；
    根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；
    采用关键点检测网络模型，对所述人脸纠正图像进行人脸关键点检测，获得纠正关键点；
    根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
  2. 根据权利要求1所述的方法，其中，所述获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息，包括：
    获取所述当前待测帧的前一帧中人脸原始图像的人脸姿态信息，作为所述当前待测帧的人脸原始图像中人脸的姿态纠正信息。
  3. 根据权利要求2所述的方法，其中，获取所述当前待测帧的前一帧中人脸原始图像的人脸姿态信息，包括：
    获取从所述前一帧中人脸原始图像检测到的人脸关键点；
    根据所述人脸关键点，确定所述前一帧中人脸原始图像的人脸姿态信息。
  4. 根据权利要求3所述的方法，其中，所述根据所述人脸关键点，确定所述前一帧中人脸原始图像帧的人脸姿态信息，包括：
    归一化所述人脸关键点，获得所述人脸关键点对应的归一化坐标；
    将所述归一化坐标作为输入数据，输入人脸朝向网络模型，从所述人脸朝向网络模型的输出获得所述前一帧中人脸原始图像的人脸姿态信息。
  5. 根据权利要求1所述的方法，其中，所述姿态纠正信息包括：人脸姿态角及相应的角度值；所述人脸姿态角包括：俯仰角、偏航角以及滚转角。
  6. 根据权利要求1-5任一所述的方法，其中，所述根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像，包括：
    将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像。
  7. 根据权利要求6所述的方法，其中，将所述人脸原始图像及所述姿态纠正信息作为输入数据，输入图像对齐模型，以输出纠正后的人脸纠正图像，包括：
    根据所述姿态纠正信息，确定所述人脸原始图像中人脸到标准朝向的旋转角度；
    将所述人脸原始图像中的人脸通过所述旋转角度旋转至标准朝向，形成朝向纠正后的人脸纠正图像。
  8. 根据权利要求7所述的方法，其中，所述根据所述姿态纠正信息，确定所述人脸原始图像中人脸到标准朝向的旋转角度，包括：
    获取预先设置的所述姿态纠正信息中人脸姿态角的旋转先后顺序；
    从预设旋转公式关联表中确定所述旋转先后顺序对应的旋转角度计算公式；
    将所述人脸姿态角的角度值代入所述旋转角度计算公式，获得所述人脸图像中人脸到标准朝向的旋转角度。
  9. 根据权利要求7所述的方法，其中，所述将所述人脸原始图像中人脸通过所述旋转角度旋转至标准朝向，获得朝向纠正后的人脸纠正图像，包括：
    识别所述人脸原始图像中的人脸，确定包含所述人脸的矩形区域；
    以所述人脸原始图像所在坐标系的纵轴为参照轴，将所述矩形区域中每个像素点相对所述参照轴旋转所述旋转角度；
    获得具备所述标准朝向的矩形区域，形成朝向纠正后的人脸纠正图像。
  10. 根据权利要求1所述的方法，其中，所述根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点包括：
    根据所述姿态纠正信息确定的旋转角度，对所述纠正关键点进行逆向旋转，以获得所述人脸原始图像的目标人脸关键点。
  11. 根据权利要求1所述的方法，其中，
    所述当前待测帧从预先捕获的短视频中获取，或者从实时捕获的直播视频中获取；
    检测到的目标人脸关键点用于所对应人脸原始图像中人脸的视觉特效设置。
  12. 一种人脸关键点检测装置，包括：
    信息获取模块，设置为获取当前待测帧的人脸原始图像，并获取所述当前待测帧的人脸原始图像中人脸的姿态纠正信息；
    图像纠正模块，设置为根据所述姿态纠正信息对所述人脸原始图像中的人脸姿态进行纠正，获得人脸纠正图像；
    关键点确定模块，设置为采用关键点检测网络模型，对所述人脸纠正图像 进行人脸关键点检测，获得纠正关键点；
    关键点纠正模块，设置为根据所述姿态纠正信息对所述纠正关键点进行逆向姿态纠正，以获得所述人脸原始图像的目标人脸关键点。
  13. 一种计算机设备，包括：
    至少一个处理器；
    存储装置，设置为存储至少一个程序；
    所述至少一个程序被所述至少一个处理器执行，使得所述至少一个处理器实现如权利要求1-11任一项所述的人脸关键点检测方法。
  14. 一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机程序，所述计算机程序被处理器执行时，实现如权利要求1-11任一项所述的人脸关键点检测方法。
PCT/CN2020/079493 2019-03-29 2020-03-16 人脸关键点检测方法、装置、设备及存储介质 WO2020199906A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910252374.5 2019-03-29
CN201910252374.5A CN109961055A (zh) 2019-03-29 2019-03-29 人脸关键点检测方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2020199906A1 true WO2020199906A1 (zh) 2020-10-08

Family

ID=67025377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079493 WO2020199906A1 (zh) 2019-03-29 2020-03-16 人脸关键点检测方法、装置、设备及存储介质

Country Status (2)

Country Link
CN (1) CN109961055A (zh)
WO (1) WO2020199906A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112241709A (zh) * 2020-10-21 2021-01-19 北京字跳网络技术有限公司 图像处理方法、胡子变换网络的训练方法、装置
CN112543343A (zh) * 2020-11-27 2021-03-23 广州华多网络科技有限公司 基于连麦直播的直播画面处理方法、装置及电子设备
CN113449657A (zh) * 2021-07-05 2021-09-28 中山大学 一种基于人脸关键点的深度伪造人脸视频检测方法、系统及介质
CN113627394A (zh) * 2021-09-17 2021-11-09 平安银行股份有限公司 人脸提取方法、装置、电子设备及可读存储介质

Families Citing this family (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109961055A (zh) * 2019-03-29 2019-07-02 广州市百果园信息技术有限公司 人脸关键点检测方法、装置、设备及存储介质
CN112330526B (zh) * 2019-08-05 2024-02-09 深圳Tcl新技术有限公司 一种人脸转换模型的训练方法、存储介质及终端设备
CN110853725A (zh) * 2019-09-30 2020-02-28 许之敏 基于视觉反馈的面部功能训练方法、设备及存储介质
CN110728621B (zh) * 2019-10-17 2023-08-25 北京达佳互联信息技术有限公司 一种面部图像的换脸方法、装置、电子设备及存储介质
CN112700396A (zh) * 2019-10-17 2021-04-23 中国移动通信集团浙江有限公司 一种人脸图片光照评价方法、装置、计算设备和存储介质
CN110956106B (zh) * 2019-11-20 2023-10-10 广州方硅信息技术有限公司 一种直播开播的处理方法、装置、存储介质及设备
CN110933452B (zh) * 2019-12-02 2021-12-03 广州酷狗计算机科技有限公司 萌脸礼物显示方法、装置及存储介质
CN111476096A (zh) * 2020-03-06 2020-07-31 深圳壹账通智能科技有限公司 一种人脸图像关键点方法、装置、计算机设备及存储介质
CN111402122A (zh) * 2020-03-20 2020-07-10 北京字节跳动网络技术有限公司 图像的贴图处理方法、装置、可读介质和电子设备
CN111612712B (zh) * 2020-05-19 2023-04-25 济南博观智能科技有限公司 一种人脸端正度的确定方法、装置、设备及介质
CN113056907A (zh) * 2020-07-28 2021-06-29 深圳市大疆创新科技有限公司 拍摄方法、拍摄装置及存储介质
CN111898552B (zh) * 2020-07-31 2022-12-27 成都新潮传媒集团有限公司 一种判别人员关注目标物的方法、装置及计算机设备
CN111815519B (zh) * 2020-09-01 2021-04-30 南京甄视智能科技有限公司 旋转目标检测方法、装置、系统与可读介质
CN112613409A (zh) * 2020-12-24 2021-04-06 咪咕动漫有限公司 手部关键点检测方法、装置、网络设备及存储介质
CN112990032B (zh) * 2021-03-23 2022-08-16 中国人民解放军海军航空大学航空作战勤务学院 人脸图像处理方法及装置
CN113160036B (zh) * 2021-04-19 2022-09-20 金科智融科技(珠海)有限公司 一种保持脸型不变的图像换脸方法

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090060290A1 (en) * 2007-08-27 2009-03-05 Sony Corporation Face image processing apparatus, face image processing method, and computer program
CN103605965A (zh) * 2013-11-25 2014-02-26 苏州大学 一种多姿态人脸识别方法和装置
KR20150127381A (ko) * 2014-05-07 2015-11-17 포항공과대학교 산학협력단 얼굴 특징점 추출 방법 및 이를 수행하는 장치
CN105550637A (zh) * 2015-12-04 2016-05-04 小米科技有限责任公司 轮廓点定位方法及装置
CN105913416A (zh) * 2016-04-06 2016-08-31 中南大学 一种自动分割三维人脸模型区域的方法
CN108470328A (zh) * 2018-03-28 2018-08-31 百度在线网络技术（北京）有限公司 用于处理图像的方法和装置
CN108875492A (zh) * 2017-10-11 2018-11-23 北京旷视科技有限公司 人脸检测及关键点定位方法、装置、系统和存储介质
CN109961055A (zh) * 2019-03-29 2019-07-02 广州市百果园信息技术有限公司 人脸关键点检测方法、装置、设备及存储介质

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108875526B (zh) * 2018-01-05 2020-12-25 北京旷视科技有限公司 视线检测的方法、装置、系统及计算机存储介质
CN109359526B (zh) * 2018-09-11 2022-09-27 深圳大学 一种人脸姿态估计方法、装置和设备

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090060290A1 (en) * 2007-08-27 2009-03-05 Sony Corporation Face image processing apparatus, face image processing method, and computer program
CN103605965A (zh) * 2013-11-25 2014-02-26 苏州大学 一种多姿态人脸识别方法和装置
KR20150127381A (ko) * 2014-05-07 2015-11-17 포항공과대학교 산학협력단 얼굴 특징점 추출 방법 및 이를 수행하는 장치
CN105550637A (zh) * 2015-12-04 2016-05-04 小米科技有限责任公司 轮廓点定位方法及装置
CN105913416A (zh) * 2016-04-06 2016-08-31 中南大学 一种自动分割三维人脸模型区域的方法
CN108875492A (zh) * 2017-10-11 2018-11-23 北京旷视科技有限公司 人脸检测及关键点定位方法、装置、系统和存储介质
CN108470328A (zh) * 2018-03-28 2018-08-31 百度在线网络技术（北京）有限公司 用于处理图像的方法和装置
CN109961055A (zh) * 2019-03-29 2019-07-02 广州市百果园信息技术有限公司 人脸关键点检测方法、装置、设备及存储介质

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112241709A (zh) * 2020-10-21 2021-01-19 北京字跳网络技术有限公司 图像处理方法、胡子变换网络的训练方法、装置
CN112543343A (zh) * 2020-11-27 2021-03-23 广州华多网络科技有限公司 基于连麦直播的直播画面处理方法、装置及电子设备
CN112543343B (zh) * 2020-11-27 2024-02-23 广州华多网络科技有限公司 基于连麦直播的直播画面处理方法、装置及电子设备
CN113449657A (zh) * 2021-07-05 2021-09-28 中山大学 一种基于人脸关键点的深度伪造人脸视频检测方法、系统及介质
CN113449657B (zh) * 2021-07-05 2022-08-30 中山大学 一种基于人脸关键点的深度伪造人脸视频检测方法、系统及介质
CN113627394A (zh) * 2021-09-17 2021-11-09 平安银行股份有限公司 人脸提取方法、装置、电子设备及可读存储介质
CN113627394B (zh) * 2021-09-17 2023-11-17 平安银行股份有限公司 人脸提取方法、装置、电子设备及可读存储介质

Also Published As

Publication number Publication date
CN109961055A (zh) 2019-07-02

Similar Documents

Publication Publication Date Title
WO2020199906A1 (zh) 人脸关键点检测方法、装置、设备及存储介质
US10212337B2 (en) Camera augmented reality based activity history tracking
CN108537721B (zh) 全景图像的处理方法、装置及电子设备
WO2018214365A1 (zh) 图像校正方法、装置、设备、系统及摄像设备和显示设备
CN105981368B (zh) 在成像装置中的照片构图和位置引导
WO2015074476A1 (zh) 一种图像处理方法、装置和存储介质
US20230132407A1 (en) Method and device of video virtual background image processing and computer apparatus
US20200394392A1 (en) Method and apparatus for detecting face image
CN110163087B (zh) 一种人脸姿态识别方法及系统
US11120535B2 (en) Image processing method, apparatus, terminal, and storage medium
EP3992919B1 (en) Three-dimensional facial model generation method and apparatus, device, and medium
WO2022012085A1 (zh) 人脸图像处理方法、装置、存储介质及电子设备
WO2021175180A1 (zh) 视线确定方法、装置、电子设备和计算机可读存储介质
CN109670444B (zh) 姿态检测模型的生成、姿态检测方法、装置、设备及介质
BR112021007087A2 (pt) método e aparelho de processamento de imagem, e dispositivo
WO2020019504A1 (zh) 一种机器人屏幕解锁方法、装置、智能设备及存储介质
US20220215507A1 (en) Image stitching
CN110427849B (zh) 人脸姿态确定方法、装置、存储介质和电子设备
JP2023517121A (ja) 画像処理及び画像合成方法、装置及びコンピュータプログラム
WO2022105569A1 (zh) 页面方向识别方法、装置、设备及计算机可读存储介质
WO2022002262A1 (zh) 基于计算机视觉的字符序列识别方法、装置、设备和介质
WO2021197466A1 (zh) 眼球检测方法、装置、设备及存储介质
WO2020223940A1 (zh) 姿势预测方法、计算机设备和存储介质
CN109120854A (zh) 图像处理方法、装置、电子设备及存储介质
CN112017212B (zh) 人脸关键点跟踪模型的训练、跟踪方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20781765

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20781765

Country of ref document: EP

Kind code of ref document: A1