CN115797991A - Method for generating recognizable face image according to face side image - Google Patents
Method for generating recognizable face image according to face side image Download PDFInfo
- Publication number
- CN115797991A CN115797991A CN202210816400.4A CN202210816400A CN115797991A CN 115797991 A CN115797991 A CN 115797991A CN 202210816400 A CN202210816400 A CN 202210816400A CN 115797991 A CN115797991 A CN 115797991A
- Authority
- CN
- China
- Prior art keywords
- face
- point
- key points
- coordinates
- rotation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention provides a method for generating recognizable face images according to face side images. The method comprises the following steps: extracting face key points from the left and right side images of the face, and establishing a corresponding three-dimensional rotation model according to the change process of a face rotation path and a camera imaging rule; establishing a corresponding relation between the actually extracted left and right face key points and the face key points in the imaging plane deduced by the three-dimensional rotation model, converting the corresponding relation into an optimization problem, and solving the optimization problem to obtain unknown parameters in the three-dimensional rotation model; based on the three-dimensional rotation model and the solved parameters, the positions of the key points of the human face at any angle in the rotation path can be obtained, and an identifiable human face image can be generated according to the coordinates of the key points of the human face in the intermediate state. The invention directly expresses the rotation path of the key points of the human face in the two-dimensional image through a mathematical model and an optimization problem, and more quickly calculates the positions of the key points of the human face in the image shot under any visual angle.
Description
Technical Field
The invention relates to the technical field of face recognition, in particular to a method for generating recognizable face key points according to face side images.
Background
With the wide application of face recognition technology and the popularization of monitoring cameras, more and more scenes can recognize and judge faces through monitoring pictures. The face captured in the actual monitoring video is affected by many other factors, which hinders the accuracy of face recognition. For example, in many factories or rooms with large areas, the arrangement of the monitoring cameras is generally fixed on the walls around the rooms, and in addition, due to the characteristic that the human face can flexibly and freely rotate, the human face shot by video monitoring is generally the left and right side faces, but not the front face of the human face. The non-frontal face with large-angle deflection in the image cannot be effectively recognized by adopting the existing various face detection and recognition algorithms. Therefore, the face images at the angles convenient to identify need to be restored through the face side images at the different viewing angles, and the restored face images are input into a face comparison system, so that a clearer and more accurate identification result can be provided.
At present, no method for efficiently generating an identifiable face front image from a large-angle face side image exists in the prior art.
Disclosure of Invention
The embodiment of the invention provides a method for generating an identifiable face image according to a face side image, so as to effectively generate the identifiable face image from the large-angle face side image.
In order to achieve the purpose, the invention adopts the following technical scheme.
A method of generating identifiable frontal face key points from face side images, comprising:
extracting key points of a left face and a right face from the shot left side image and right side image, abstracting the shooting process of the left side face and the right side face into a circular-arc-shaped face rotation path, and establishing a corresponding three-dimensional rotation model according to the change process of the face rotation path and the camera imaging rule;
establishing a corresponding relation between the actually extracted left and right face key points and the face key points in the imaging plane deduced by the three-dimensional rotating model, converting the corresponding relation into an optimization problem, solving the optimization problem by a quasi-Newton method, and solving unknown parameters in the three-dimensional rotating model;
based on the established three-dimensional rotation model and the obtained unknown parameters, the positions of the key points of the human face at any angle on the rotation path are calculated by setting different rotation angles, and an identifiable human face image is generated according to the intermediate state of the key points of the human face on the rotation path.
Preferably, the method for extracting the key points of the left and right faces of the left and right side images obtained by shooting by using the ERT algorithm abstracts the shooting process of the left and right side faces into an arc-shaped face rotation path, and establishes a corresponding three-dimensional rotation model according to the change process of the face rotation path and the camera imaging rule, and includes the following steps:
abstracting a shooting process into a circular arc-shaped rotation path, regarding images of left and right sides of a face as images obtained by shooting the face by rotating around an optical axis of a camera, establishing a three-dimensional rotation model between key points of the left and right sides of the face according to the rotation process of the shooting model, assuming that a Z axis is a straight line where the optical axis of the camera is located, after the camera shoots one image, rotating the face by a certain angle around the fixed Z axis, shooting a second image again, and setting the coordinates of the key points of the face before rotation as P (X) (namely, setting the coordinates of the key points of the face before rotation as P (X) i ,Y i ,Z i ) Rotating the point P clockwise by theta degrees around a Z axis to obtain a point P ', wherein the physical meanings of the point P and the point P' respectively correspond to the coordinates of the key point of the left face and the key point of the right face in a three-dimensional space, the point P rotates around the Z axis, the point P 'is equal to the Z coordinate of the point P, the point P and the point P' are projected to an XOY plane, the coordinates on the X axis and the Y axis are solved, and the rotation process of the shooting model is expressed by a matrix as follows:
wherein θ represents a rotation angle, (X) i ,Y i ) And (X) i ',Y i ') respectively represent the projection of the point P and the point P ' on the XOY plane, the coordinates of the point P ' being:
(X i ·cosθ-Y i ·sinθ,Y i ·cosθ+X i ·sinθ,Z i ) (2)
the three-dimensional coordinates are projected into a two-dimensional plane by an imaging principle of a simulation camera, a projection plane is arranged in a three-dimensional rotation model, the projection plane is an imaging plane, a three-dimensional space point P and a point P' are projected onto the imaging plane, the three-dimensional coordinates of a human face key point in the three-dimensional space in the projection plane are calculated, the three-dimensional coordinates of the human face key point in the projection plane are converted into a coordinate system, and the three-dimensional coordinates are converted into a pixel coordinate system from the three-dimensional coordinate system.
Preferably, the projecting the three-dimensional space point P and the point P' onto the imaging plane to calculate the three-dimensional coordinates of the key points of the human face in the three-dimensional space in the projection plane includes:
setting projection points of a point P and a point P ' on an imaging plane as P _ T and P _ T ', wherein projection lines in parallel projection are parallel lines, Z coordinates of the projection points P _ T and P _ T ' are equal to those of the point P and the point P ', projecting the P _ T and the P _ T ' to an XOY plane, and calculating coordinates on x and y axes, wherein a straight line T is x = -y, P is T (X i ,Y i ) Is the projection point of point P on XOY plane, P I Is the projection point of P _ t on the XOY plane, the projection mode is parallel orthographic projection, P T P I Perpendicular to the TOZ plane;
P T P I t and P I
Straight line P T P I The expression of (a) is:
x＝y+X i -Y i (3)
so straight line P T P I Point of intersection P with straight line T I The coordinates of (a) are:
from the coordinates of the point P, the coordinates of the projected point P _ t are calculated as:
from the coordinates of point P ', the coordinates of point P _ t' are calculated as:
preferably, the converting the three-dimensional coordinates of the key points of the human face in the projection plane into the coordinate system of the pixel from the three-dimensional coordinate system comprises:
establishing a plane rectangular coordinate system TOZ in a projection plane T-Z, wherein the origin of the coordinate system is (0, 0), calculating the coordinates of a projection point P _ T and a point P _ T' in the coordinate system, and respectively marking the coordinates as P A And P B . As can be seen from FIG. 3, P A And P B Is equal to the Z coordinates of points P _ t and P _ t', P I Is the projection point of P _ t on XOY plane, then P A The abscissa of (a) is OP I According to the coordinate of the projection point P _ t in the three-dimensional coordinate system O-XYZ, the coordinate P of the projection point in the TOZ coordinate system is calculated A ：
According to the coordinate of the point P _ t', the coordinate P of the projection point in the TOZ coordinate system is calculated B ：
According to the three-dimensional rotation model, the shooting process of the camera is simulated, and theoretically the coordinate change of the human face key point from the left face to the right face is deduced.
Preferably, the establishing a corresponding relationship between the actually extracted left and right face key points and the face key points in the imaging plane derived from the three-dimensional rotation model, converting the corresponding relationship into an optimization problem, and solving the optimization problem by a quasi-newton method to obtain unknown parameters in the three-dimensional rotation model includes:
corresponding the actually extracted key points of the left and right faces to the coordinates of the key points of the faces in the images calculated by the formulas 7 and 8 one by one to establish a corresponding relationship, and corresponding the key points P of the left face A Will beIt is expressed in matrix form as follows:
wherein
68 key points which represent the actually extracted key points of the left face, i =1,2 B The coordinates of (c) are as follows:
wherein
68 key points representing the actual extraction of the right face, i =1, 2., 68;
introducing a scaling factor S, a rotation matrix E and a translation matrix b, adjusting the actually extracted key points to correspond to the theoretical key point positions calculated by the model, wherein the corresponding relation is as follows:
wherein, the scaling factor S is a constant greater than 0, and the rotation matrix E and the translation matrix b are as follows:
where α represents the rotation angle between the actually extracted key point and the theoretical key point in the model, b 1 And b 2 Respectively representing the deviation of the actually extracted key points and theoretical key points in the model on the horizontal and vertical axes, and obtaining a set of equations according to the corresponding relation of the formulas (9) and (11), as follows:
wherein the content of the first and second substances,
and
respectively representing the coordinates of key points actually extracted by the left face and the right face, wherein the coordinates are known variables; (X) i ,Y i ,Z i ) I =1, 2., 68 is the three-dimensional coordinates of the face key points of the left image in the model, as unknown parameters; theta represents the rotation angle of the face from the position in the left image to the position in the right image, and is an unknown parameter; the scaling factor S, the rotation matrix E and the translation matrix b are also unknown variables, and equation (13) is reduced as follows
Wherein, S, alpha, b 1 、b 2 θ and (X) i ) i＝1,2,3,...,68 All variables to be solved are 73 in total.
Find a set of parameters (S, α, b) 1 ,b 2 ,θ,X i ) i＝1,2,3,...,68 Minimizing the sum of the squared residuals on both sides of the equation in equation (13), solving the following optimization problem:
the least squares optimization problem is solved using a quasi-newton method, and the solution of the set of optimization problems is used to approximate the solution of the unknowns in the substitute equation.
Preferably, the step of calculating the positions of the face key points at any angle on the rotation path by setting different rotation angles based on the established three-dimensional rotation model and the obtained unknown parameters, and generating an identifiable face image according to the intermediate state of the face key points on the rotation path includes:
on a rotation path in the three-dimensional rotation model, a rotation angle of any angle is given to calculate the coordinates of key points of a human face in an image under the rotation angle, and the specific calculation formula is as follows:
wherein the content of the first and second substances,
the coordinates of key points of the face extracted from the left image are represented by beta, which represents any rotation angle on the path and has the value range of [0, theta ]]Substituting the parameters obtained by the formulas (15) and (16) into the formula (17), calculating the coordinates of the key points of the face in the rotation path, drawing the coordinates of the key points of the face in a two-dimensional scatter diagram, dividing the calculated theta into 10 equal parts, drawing the scatter diagram of the key points of the face in sequence when the change of the beta angle represents the change of the rotation angle, and calculating the coordinates of the key points of the face according to the formula
And generating a face image convenient to identify according to the front face key point at the midpoint of the whole path.
According to the technical scheme provided by the embodiment of the invention, the corresponding relation of the key points of the human face in the two-dimensional image is directly calculated through the solving of the mathematical model and the optimization problem, and the positions of the key points of the human face in the image shot under any visual angle are more quickly calculated. Under a larger shooting view angle, the method can still reconstruct the rotation path of the key points of the human face.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained according to these drawings without creative efforts.
FIG. 1 is a flowchart illustrating a process of generating recognizable key points of a human face from a side image of the human face according to an embodiment of the present invention;
fig. 2 is a simulation diagram of a left-right side face rotation shooting process provided in an embodiment of the present invention;
FIG. 3 is a three-dimensional rotation model diagram according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a planar XOY according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a parallel projection according to an embodiment of the present invention;
FIG. 6 is a diagram illustrating a process of calculating coordinates of proxels according to an embodiment of the present invention;
fig. 7 is a 3DMM face key point scatter diagram provided in an embodiment of the present invention;
fig. 8 is a schematic diagram of a 3DMM face key point projection scattergram according to an embodiment of the present invention;
fig. 9 is a schematic diagram of key point positions of a human face when shooting a left image and a right image according to an embodiment of the present invention;
fig. 10 is a schematic representation diagram of projection points of left and right human faces calculated according to a three-dimensional rotation model in a pixel coordinate system according to an embodiment of the present invention;
fig. 11 is a schematic diagram of key points of a face extracted from left and right images according to an embodiment of the present invention;
fig. 12 is a schematic diagram of a rotation path of a key point of a human face according to an embodiment of the present invention;
fig. 13 is a schematic diagram of a rotation path of a face key point with a rotation angle of 1.76 according to an embodiment of the present invention;
fig. 14 is a schematic diagram of a rotation path of a key point of a human face with a rotation angle of 1.8 according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or coupled. As used herein, the term "and/or" includes any and all combinations of one or more of the associated listed items.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
For the convenience of understanding the embodiments of the present invention, the following description will be further explained by taking several specific embodiments as examples in conjunction with the drawings, and the embodiments are not to be construed as limiting the embodiments of the present invention.
The invention recovers the key points of the human face at any angle in the rotation path through the key points of the left and right side faces of the human face. Firstly, key points of left and right faces are extracted from the captured left and right side images by using an ERT (Ensemble of Regression Trees) algorithm. Secondly, the shooting process of the left and right sides of the face is abstracted into a circular arc-shaped rotation path. And establishing a corresponding three-dimensional rotation model according to the change process of the face rotation path and the camera imaging rule. And then, corresponding the actually extracted left and right face key points to the face key points in the imaging plane deduced by the three-dimensional rotation model. Then, the corresponding relation is converted into an optimization problem, and unknown parameters in the model are solved through a quasi-Newton method. And finally, calculating the coordinates of the key points of the human face of any rotating path on the rotating path by setting different rotating angles based on the established three-dimensional rotating model and the obtained unknown parameters.
The processing flow chart of the method for generating the recognizable face image according to the face side image is shown in figure 1, and mainly comprises two parts of three-dimensional rotation modeling and optimization problem solving.
Three-dimensional rotation modeling: first, the shooting process is abstracted into a circular arc-shaped path, and as shown in fig. 2, the images of the left and right side faces can be regarded as images obtained by the human face rotating around the optical axis of the camera. Next, a three-dimensional rotation model between the key points of the left and right sides of the face is established according to the rotation process of the shooting model, as shown in fig. 3. Assuming that the Z axis is a straight line where the optical axis of the camera is located, after the camera shoots one image, the face of the person rotates for a certain angle around the fixed Z axis, and a second image is shot again. The coordinates of key points of the human face before rotation are set as P (X) i ,Y i ,Z i ) Rotating the point P clockwise by theta degrees around the Z axis to obtain a point P ', wherein the physical meanings of the point P and the point P' respectively correspond to the key point of the left face and the key point of the right faceCoordinates of the point in three-dimensional space. Since the point P rotates around the Z axis, the point P 'is equal to the Z coordinate of the point P, and the point P' are projected onto the XOY plane, and the coordinates on the X and Y axes are solved, as shown in fig. 4. The above rotation process can be represented by a matrix as:
wherein θ represents a rotation angle, (X) i ,Y i ) And (X) i ',Y i ') represent the projection of point P and point P' on the XOY plane, respectively. Thus, the coordinates of point P' are:
(X i ·cosθ-Y i ·sinθ,Y i ·cosθ+X i ·sinθ,Z i ) (2)
then, the three-dimensional coordinates are projected into a two-dimensional plane by simulating the imaging principle of a camera. As can be seen from fig. 3, the three-dimensional rotation model only takes into account the rotation angle of the human face. Therefore, in order to simplify the camera imaging model, the human face in the image is directly regarded as the parallel projection of the human face in the three-dimensional space, so that the error generated by the transformation of the coordinate system is reduced, and the subsequent calculation is convenient. Parallel projection refers to an image formed by projecting a cluster of parallel lines from a certain direction on a certain plane through points in a three-dimensional space, and is generally used to represent a corresponding relationship between the points and the plane, as shown in fig. 5. Accordingly, a projection plane, which is an imaging plane, is set in the three-dimensional rotational model, and as shown in fig. 3, it is a plane formed by a straight line T: x + y =0 and a Z axis. Therefore, the coordinates of the key points of the human face in the three-dimensional space in the projection plane can be calculated by projecting the three-dimensional space points P and P' onto the imaging plane. The coordinates of the projected points P _ t and P _ t 'of the point P and the point P' on the imaging plane are calculated as follows.
Specifically, since the projection lines in the parallel projection are parallel lines, the Z coordinates of the projected points P _ t and P _ t 'are equal to the Z coordinates of the points P and P'. Then, P _ t and P _ t' are projected onto the XOY plane, and coordinates on the x and y axes are calculated. As shown in fig. 6, wherein the straight line T: x = -y, P T (X i ,Y i ) Is the projection point of point P on XOY plane, P I The projection point of P _ t on XOY plane is P because the projection mode is parallel positive projection T P I Perpendicular to the TOZ plane, and because the straight line T is in the TOZ plane, therefore:
P T P I t and P I
Thus, a straight line P can be obtained T P I The expression of (a) is:
x＝y+X i -Y i (3)
straight line P T P I Point of intersection P with straight line T I The coordinates of (a) are:
then, from the coordinates of the point P, the coordinates of the projection point P _ t are calculated as:
similarly, the coordinate of the point P _ t 'is calculated as follows according to the coordinate of the point P':
through the three-dimensional rotation model, three-dimensional coordinates of key points of the left face and the right face of the human face in a projection plane are calculated. However, the key points of the human face extracted from the image are two-dimensional coordinates in a pixel coordinate system, and the two coordinates cannot be directly matched with each other, so that the three-dimensional coordinates of the projection points need to be converted into the pixel coordinate system from the three-dimensional coordinate system. Specifically, a rectangular plane coordinate system TOZ is first established in the projection plane T-Z, with the origin of the coordinate system being (0, 0). Next, the coordinates of the projected point P _ t and the point P _ t' in the coordinate system are calculated and are respectively denoted as P A And P B . As can be seen from FIG. 3, P A And P B Is equal to the Z-coordinate of the point P _ t and the point P _ t'. Shown in FIG. 6, P I Is the projection point of P _ t on XOY plane, then P A The abscissa of (a) is OP I Length of (d). Therefore, from the coordinates of the projection point P _ t in the three-dimensional coordinate system O-XYZ, the coordinates P of the projection point in the TOZ coordinate system can be calculated A ：
Similarly, according to the coordinates of the point P _ t', the coordinate P of the projection point in the TOZ coordinate system can be calculated B ：
And finally deducing the coordinate change of the human face key point from the left face to the right face in theory according to the shooting process of the three-dimensional rotating model simulation camera. To verify the correctness of the model, we will forward derive it through a three-dimensional face model. Here, a common 3d mm three-dimensional face shape appearance model is selected, which contains 68 three-dimensional points corresponding to 68 key points of the face. Since it is only necessary to verify whether the projection coordinates of the 68 key points of the three-dimensional face on the imaging plane conform to the extracted 68 key points of the face, we do not need to fit the face, and only need to know the 3D coordinate positions corresponding to the 68 key points of the face in the general model, for example:
nose tip (1.23, 36.94, 8.41)
Underrun (0.10, 0.88, -77.06)
Left eye angle (45.17, -4.89, 38.08)
Right canthus: (-46.04, -7.03,37.47)
Left mouth angle: (28.79,3.21, -28.07)
Right mouth angle: (-28.92,2.24, -28.61)
Firstly, 68 key points of the 3D face are drawn in a three-dimensional space, and the positions of the key points are adjusted to be in accordance with the face shooting position of the model shown in figure 3, namely the face front faces towards the Z axis. As shown in fig. 7, the straight line is the Z-axis, and the asterisk is the origin of coordinates of the three-dimensional coordinate system. Then, the projection coordinates of the three-dimensional face on the imaging plane are calculated according to the formula 7 and the formula 8, and a two-dimensional scatter diagram is drawn, as shown in fig. 8, it can be seen that the projection result conforms to the key point position on the front face of the face. Then, according to the formula 2, the three-dimensional face model is rotated left and right around the fixed shaft, the rotation angle is assumed to be ± 15 °, and the shooting positions of the left side and the right side of the face are simulated respectively, as shown in fig. 9. And finally, respectively calculating and drawing projection points of the left and right groups of face key points by using a formula 7 and a formula 8, as shown in fig. 10. Comparing the key points of the face extracted actually, as can be seen from fig. 11, the trend of the facial contour of the two is the same, thus proving the correctness of the model.
Solving an optimization problem:
and finally, deducing a rotation transformation relation from the left face to the right face in the image according to the constructed three-dimensional rotation model of the key points of the face. Then, the actually extracted left and right face key points are in one-to-one correspondence with the face key point coordinates in the image calculated by the formulas 7 and 8. Left side face keypoint corresponding point P A Is expressed in the form of a matrix, as follows:
wherein
Representing the 68 key points actually extracted by the left face, i =1, 2. In the same way, the corresponding point P of the right side face key point B The coordinates of (c) are as follows:
wherein
Represent the actual 68 key points extracted for the right face, i =1,2.
Although the position of the camera is always fixed in the shooting process, the face is rotated and changed in position during shooting, and some deviation is difficult to avoid, so that in an actual situation, the key points of the face cannot be guaranteed to be completely arranged according to theoretical key point positions, therefore, a scaling factor S, a rotation matrix E and a translation matrix b are introduced to adjust the actually extracted key points to correspond to the theoretical key point positions calculated by the model, and the corresponding relationship is as follows:
wherein the scaling factor S is a constant greater than 0, and the rotation matrix E and the translation matrix b are as follows
Where α represents the rotation angle between the actually extracted key point and the theoretical key point in the model, b 1 And b 2 And respectively representing the deviation of the actually extracted key points and the theoretical key points in the model on the horizontal and vertical axes.
In summary, a set of equations can be obtained from the correspondence of equations 9 and 11 as follows
Wherein, the first and the second end of the pipe are connected with each other,
and
respectively representing the coordinates of key points actually extracted by the left face and the right face, wherein the coordinates are known variables; (X) i ,Y i ,Z i ) I =1, 2., 68 is the three-dimensional coordinates of the face key points of the left image in the model, as unknown parameters; theta represents the rotation angle of the face from the position in the left image to the position in the right image, and is an unknown parameter; in addition, the scaling factor S, the rotation matrix E and the translation matrix b are also unknown variables. Equation 13 is simplified as follows
Wherein, S, alpha, b 1 、b 2 θ and (X) i ) i＝1,2,3,...,68 All variables to be solved are 73 in total. This equation will be solved below.
Since the system of equations shown in equation 14 contains 73 unknown parameters, which are much larger than 68 known data samples, these parameters cannot be calculated from the equation solution. Therefore, the method is converted into two groups of optimization problems, and the solution thought of the least square method is adopted to calculate the solution of the parameters. In particular, a set of parameters (S, α, b) is found 1 ,b 2 ,θ,X i ) i＝1,2,3,...,68 The sum of the squared residuals of the two sides of the equation in equation 13 is minimized, i.e. the following optimization problem is calculated:
the solution of the set of optimization problems is then used to approximate the solution of the unknowns in the replacement equation. Here, a quasi-newton method is selected to solve the least squares optimization problem. Considering that the unknown quantity contained in equation 16 is small, the optimization problem is solved first. For the example shown in FIG. 11, the calculation results are as follows:
α＝0.006,b 2 ＝0,S＝1.006,min＝51867
the variables alpha, b 2 Substituting S into the formula 15, and calculating by using a quasi-Newton method again to obtain:
θ＝1.636,b 1 ＝-899.937,min＝0.006，(X i ) i＝1,2,3,...,68 comprises the following steps:
as can be seen from the three-dimensional rotation model shown in fig. 3, on this rotation path, given a rotation angle of any angle, the key point coordinates of the face in the image at the rotation angle can be calculated, and the specific calculation formula is as follows
Wherein, the first and the second end of the pipe are connected with each other,
the coordinates of key points of the face extracted from the left image are represented by beta, which represents any rotation angle on the path and has the value range of [0, theta ]]In the meantime. The parameters obtained in the above example are substituted into formula (17), the coordinates of the key points of the face in the rotation path are calculated, and the coordinates are drawn in a two-dimensional scatter diagram, and the change of the angle beta represents the change of the rotation angle. Dividing the calculated theta into 10 equal parts, and drawing a scatter diagram of key points of the face in turn, as shown in FIG. 12, wherein when the calculated theta is equal to the key points of the face
Then it is exactly the face keypoint at the midpoint of the entire path, i.e. the face keypoint at the front, as shown in fig. 12-e. And generating a face image convenient to identify according to the face key points on the front face.
In order to effectively illustrate the performance of the present invention, the experimental results are shown and analyzed by using several sets of data, thereby proving that the present invention has excellent performance.
Fig. 13 shows the rotation path of the key points of the face when the rotation angle is 1.7, and fig. 14 shows the rotation path of the key points of the face when the rotation angle is 1.8.
In summary, the front face key point reconstruction method based on three-dimensional rotation modeling provided by the embodiment of the invention can realize reconstruction of the whole rotation path of the face key point under the condition that the face image has large parallax, and has great advantages in speed.
The invention directly calculates the rotation path of the key points of the human face in the two-dimensional image by solving a mathematical model and an optimization problem, and more efficiently calculates the positions of the key points of the human face in the image shot under any visual angle. Under a larger shooting visual angle, the invention can still reconstruct the rotation path of the key point of the human face.
Those of ordinary skill in the art will understand that: the figures are schematic representations of one embodiment, and the blocks or processes shown in the figures are not necessarily required to practice the present invention.
From the above description of the embodiments, it is clear to those skilled in the art that the present invention can be implemented by software plus necessary general hardware platform. Based on such understanding, the technical solutions of the present invention may be embodied in the form of software products, which can be stored in a storage medium, such as ROM/RAM, magnetic disk, optical disk, etc., and include instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute the method according to the embodiments or some parts of the embodiments.
The embodiments in the present specification are described in a progressive manner, and the same and similar parts among the embodiments are referred to each other, and each embodiment focuses on different points from other embodiments. In particular, for apparatus or system embodiments, since they are substantially similar to method embodiments, they are described in relative terms, as long as they are described in partial descriptions of method embodiments. The above-described embodiments of the apparatus and system are merely schematic, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only for the preferred embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (6)
1. A method for generating an identifiable face image from a face side image, comprising:
extracting key points of a left face and a right face from the shot left side and right side images, abstracting the shooting process of the left side and the right side faces into a circular-arc-shaped face rotation path, and establishing a corresponding three-dimensional rotation model according to the change process of the face rotation path and the camera imaging rule;
establishing a corresponding relation between the actually extracted left and right face key points and the face key points in the imaging plane deduced by the three-dimensional rotation model, converting the corresponding relation into an optimization problem, solving the optimization problem by a quasi-Newton method, and solving unknown parameters in the three-dimensional rotation model;
based on the established three-dimensional rotation model and the obtained unknown parameters, the positions of the key points of the human face at any angle on the rotation path are calculated by setting different rotation angles, and an identifiable human face image is generated according to the intermediate state of the key points of the human face on the rotation path.
2. The method according to claim 1, wherein the method for extracting key points of left and right faces from the left and right side images obtained by shooting by using the ERT algorithm, abstracting the shooting process of the left and right side faces into a circular-arc-shaped face rotation path, and establishing a corresponding three-dimensional rotation model according to the change process of the face rotation path and the camera imaging rule comprises:
abstracting a shooting process into an arc-shaped rotation path, taking images of left and right side faces as images obtained by shooting the face by rotating around an optical axis of a camera, establishing a three-dimensional rotation model between key points of the face on the left and right sides according to the rotation process of the shooting model, assuming that a Z axis is a straight line where the optical axis of the camera is located, after the camera shoots one image, rotating the face by a certain angle around the fixed Z axis, shooting a second image again, and setting the coordinates of the key points of the face before rotation as P (X) (namely X is the coordinate of the key points of the face before rotation) i ,Y i ,Z i ) Rotating the point P clockwise by theta degrees around the Z axis to obtain a point P ', wherein the physical meanings of the point P and the point P' respectively correspond to the coordinates of the key point of the left face and the key point of the right face in a three-dimensional space, the point P rotates around the Z axis, the point P 'is equal to the Z coordinate of the point P, the point P and the point P' are projected to an XOY plane, the coordinates on the X axis and the Y axis are solved, and the rotation process of the shooting model is expressed by a matrix as follows:
wherein θ represents a rotation angle, (X) i ,Y i ) And (X) i ',Y i ') respectively represent the projection of the point P and the point P ' on the XOY plane, the coordinates of the point P ' being:
(X i ·cosθ-Y i ·sinθ,Y i ·cosθ+X i ·sinθ,Z i ) (2)
the three-dimensional coordinates are projected into a two-dimensional plane through the imaging principle of a simulation camera, a projection plane is arranged in a three-dimensional rotation model, the projection plane is an imaging plane, a three-dimensional space point P and a point P' are projected onto the imaging plane, the three-dimensional coordinates of a human face key point in the three-dimensional space in the projection plane are calculated, the three-dimensional coordinates of the human face key point in the projection plane are converted into a coordinate system, and the three-dimensional coordinates are converted into a pixel coordinate system from the three-dimensional coordinate system.
3. The method according to claim 2, wherein said projecting the three-dimensional space point P and the point P' onto the imaging plane, calculating three-dimensional coordinates of the face key points in the three-dimensional space in the projection plane, comprises:
setting projection points of a point P and a point P ' on an imaging plane as P _ T and P _ T ', wherein projection lines in parallel projection are parallel lines, Z coordinates of the projection points P _ T and P _ T ' are equal to those of the point P and the point P ', projecting the P _ T and the P _ T ' to an XOY plane, and calculating coordinates on x and y axes, wherein a straight line T is x = -y, P is T (X i ,Y i ) Is the projection point of point P on XOY plane, P I Is the projection point of P _ t on XOY plane in parallel orthographic projection mode, P T P I Perpendicular to the TOZ plane;
P T P I t and P I
Straight line P T P I The expression of (c) is:
x＝y+X i -Y i (3)
so straight line P T P I Point of intersection P with straight line T I The coordinates of (a) are:
from the coordinates of the point P, the coordinates of the projected point P _ t are calculated as:
from the coordinates of point P ', the coordinates of point P _ t' are calculated as:
4. the method of claim 3, wherein the converting the three-dimensional coordinates of the face key points in the projection plane into the coordinate system from the three-dimensional coordinate system to the pixel coordinate system comprises:
establishing a plane rectangular coordinate system TOZ in a projection plane T-Z, wherein the origin of the coordinate system is (0, 0), calculating the coordinates of a projection point P _ T and a point P _ T' in the coordinate system, and respectively marking the coordinates as P A And P B . As can be seen from FIG. 3, P A And P B Is equal to the Z coordinates of points P _ t and P _ t', P I Is the projection point of P _ t on XOY plane, then P A The abscissa of (a) is OP I According to the coordinates of the projection point P _ t in the three-dimensional coordinate system O-XYZ, the coordinate P of the projection point in the TOZ coordinate system is calculated A ：
According to the coordinates of the point P _ t', the coordinates P of the projection point in the TOZ coordinate system are calculated B ：
According to the three-dimensional rotation model, the shooting process of the camera is simulated, and theoretically the coordinate change of the human face key point from the left face to the right face is deduced.
5. The method according to claim 2, 3 or 4, wherein the establishing a corresponding relationship between the key points of the left and right faces extracted actually and the key points of the faces in the imaging plane derived from the three-dimensional rotation model, converting the corresponding relationship into an optimization problem, and solving the optimization problem by a quasi-newton method to obtain unknown parameters in the three-dimensional rotation model comprises:
the actually extracted key points of the left and right faces and the key points calculated by the formulas 7 and 8The coordinates of key points of the face in the image are in one-to-one correspondence, a corresponding relation is established, and the corresponding point P of the key point of the face on the left side is corresponding to the point P A Is expressed in the form of a matrix, as follows:
wherein (X) i A ,Y i A ) 68 key points representing the actually extracted left face, i =1,2 B The coordinates of (c) are as follows:
wherein (X) i B ,Y i B ) 68 key points representing the actual extraction of the right face, i =1, 2., 68;
introducing a scaling factor S, rotating a matrix E and translating a matrix b, adjusting the actually extracted key points to correspond to the theoretical key point positions calculated by the model, wherein the corresponding relation is as follows:
wherein, the scaling factor S is a constant greater than 0, and the rotation matrix E and the translation matrix b are as follows:
where α represents the rotation angle between the actually extracted key point and the theoretical key point in the model, b 1 And b 2 Respectively representing the deviation of the actually extracted key points and the theoretical key points in the model on the horizontal and vertical axes, and obtaining a set of equations according to the corresponding relation of the formulas (9) and (11), as follows:
wherein (X) i A ,Y i A ) And (X) i B ,Y i B ) I =1, 2., 68 respectively represent the coordinates of key points actually extracted from the left face and the right face, and are known variables; (X) i ,Y i ,Z i ) I =1, 2., 68 is the three-dimensional coordinates of the key points of the face of the left image in the model, which are unknown parameters; theta represents a rotation angle from the position in the left image to the position in the right image of the face, and is an unknown parameter; the scaling factor S, the rotation matrix E and the translation matrix b are also unknown variables, and equation (13) is reduced as follows
Wherein, S, alpha, b 1 、b 2 θ and (X) i ) i＝1,2,3,...,68 All are variables to be solved, and the total number is 73.
Find a set of parameters (S, α, b) 1 ,b 2 ,θ,X i ) i＝1,2,3,...,68 Minimizing the sum of the squared residuals on both sides of the equation in equation (13), solving the following optimization problem:
the least squares optimization problem is solved using a quasi-newton method, and the solution of the set of optimization problems is used to approximate the solution of the unknowns in the substitute equation.
6. The method according to claim 5, wherein the step of calculating the positions of the key points of the face at any angle on the rotation path by setting different rotation angles based on the established three-dimensional rotation model and the obtained unknown parameters, and generating the recognizable face image according to the intermediate state of the key points of the face on the rotation path comprises:
on a rotation path in the three-dimensional rotation model, a rotation angle of any angle is given to calculate the coordinates of key points of the face in the image under the rotation angle, and a specific calculation formula is as follows:
wherein (X) i A ,Y i A ) The coordinates of key points of the face extracted from the left image are represented by beta, which represents any rotation angle on the path and has the value range of [0, theta ]]Substituting the parameters obtained by the formulas (15) and (16) into a formula (17), calculating the coordinates of the key points of the face in the rotation path, drawing the coordinates of the key points of the face in a two-dimensional scatter diagram, dividing the calculated theta into 10 equal parts, drawing the scatter diagram of the key points of the face in sequence when the change of the beta angle represents the change of the rotation angle, and if the calculated theta is equal to the change of the key points of the face in sequence, drawing the scatter diagram of the key points of the face in sequence when the parameters obtained by the formulas (15) and (16) are substituted into the formula (17)
And then, generating a face image convenient to identify for the front face key point at the middle point of the whole path according to the front face key point.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210816400.4A CN115797991A (en) | 2022-07-12 | 2022-07-12 | Method for generating recognizable face image according to face side image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210816400.4A CN115797991A (en) | 2022-07-12 | 2022-07-12 | Method for generating recognizable face image according to face side image |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115797991A true CN115797991A (en) | 2023-03-14 |
Family
ID=85431340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210816400.4A Pending CN115797991A (en) | 2022-07-12 | 2022-07-12 | Method for generating recognizable face image according to face side image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115797991A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116978102A (en) * | 2023-08-04 | 2023-10-31 | 深圳市英锐存储科技有限公司 | Face feature modeling and recognition method, chip and terminal |
-
2022
- 2022-07-12 CN CN202210816400.4A patent/CN115797991A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116978102A (en) * | 2023-08-04 | 2023-10-31 | 深圳市英锐存储科技有限公司 | Face feature modeling and recognition method, chip and terminal |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Panteleris et al. | Back to rgb: 3d tracking of hands and hand-object interactions based on short-baseline stereo | |
| EP2597597B1 (en) | Apparatus and method for calculating three dimensional (3D) positions of feature points | |
| US8933928B2 (en) | Multiview face content creation | |
| US20120162384A1 (en) | Three-Dimensional Collaboration | |
| CN108960045A (en) | Eyeball tracking method, electronic device and non-transient computer-readable recording medium | |
| CN108475327A (en) | three-dimensional acquisition and rendering | |
| CN107358633A (en) | Join scaling method inside and outside a kind of polyphaser based on 3 points of demarcation things | |
| JP2023514289A (en) | 3D face model construction method, 3D face model construction device, computer equipment, and computer program | |
| KR100560464B1 (en) | Multi-view display system with viewpoint adaptation | |
| Ye et al. | Free-viewpoint video of human actors using multiple handheld kinects | |
| US20240046557A1 (en) | Method, device, and non-transitory computer-readable storage medium for reconstructing a three-dimensional model | |
| WO2019062056A1 (en) | Smart projection method and system, and smart terminal | |
| CN111127524A (en) | Method, system and device for tracking trajectory and reconstructing three-dimensional image | |
| CN114119739A (en) | Binocular vision-based hand key point space coordinate acquisition method | |
| Elhayek et al. | Outdoor human motion capture by simultaneous optimization of pose and camera parameters | |
| CN114152217A (en) | Binocular phase expansion method based on supervised learning | |
| CN115797991A (en) | Method for generating recognizable face image according to face side image | |
| Roberts et al. | Estimating the gaze of a virtuality human | |
| Mulligan et al. | Stereo-based environment scanning for immersive telepresence | |
| Medien | Implementation of a low cost marker based infrared optical tracking system | |
| Shabanov et al. | Self-supervised depth denoising using lower-and higher-quality RGB-d sensors | |
| US20230334810A1 (en) | Methods, storage media, and systems for generating a three-dimensional coordinate system | |
| Su | Real-Time Capture and Rendering of Physical Scene with an Efficiently Calibrated RGB-D Camera Network | |
| Vupparaboina et al. | Smart camera networks: An analytical framework for auto calibration without ambiguity | |
| Naweed et al. | Enhancement of 3D capture of room-sized dynamic scenes with Pan-Tilt-Zoom cameras |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |