FR3018937A1 - METHOD FOR IMPROVED FACE MODELING FROM AN IMAGE - Google Patents

Abstract

The invention relates to a method for three-dimensional modeling of a face represented on a two-dimensional image, comprising: an acquisition (1000) of coordinates of at least one point of the surface of the face on the image; a first estimate (2200) of the shape of the face, the shape comprising a vertex array each corresponding to a point on the surface of a face, and the estimation of the pose of the face on the image, the method being characterized in that it further comprises a step of deforming (3200) the shape of the face to obtain a refined estimate of the shape of the face on the image, the deformation being continuous with respect to the first estimate of shape of the face and the coordinates of the projection on the image of at least one vertex being constrained to correspond to the position of a corresponding point of the face on the image. The invention is particularly applicable to the authentication or identification of individuals.

Description

FIELD OF THE INVENTION The invention relates to the field of three-dimensional modeling of faces of individuals represented on images. The invention applies in particular to the facial recognition of individuals, by generating a front view of an individual from a face model established from a non-frontal view. STATE OF THE ART The identification of individuals by face recognition is implemented by comparing two images of faces, and deducing from this comparison a score evaluating the resemblance between the faces on the images. When the compared faces do not appear on the image with the same pose, the resemblance score can be strongly degraded, even if the faces on the images come from the same person. This results in a significant loss of efficiency of identification processes implemented, since the pose of the face on the images is not the same. The optimal recognition efficiency is therefore achieved not only when two faces have the same pose on the compared images, but also when the faces are seen from the front, because this view provides the most information on the shape of the face. However, it is impossible to systematically obtain a face image of a face for identification. Indeed, in most situations, a previously recorded front image of an individual, such as for example in an identity document, is compared to an image of the individual acquired "on the fly" by a computer system. acquisition such as a surveillance camera. The image thus acquired is hardly ever a frontal image because the individual does not look through the acquisition system. In this case, image processing methods have been developed for generating, from an image of a face, an image of the same face, seen from the front.

To do this, the acquired image is processed to determine the three-dimensional shape of the face of the individual on the image, its pose, that is to say its position relative to a front view, as well as a representation of the texture of the face, that is, the physical appearance of the surface of the face superimposed on the three-dimensional structure of the shape of the face.

To determine the three-dimensional shape of the individual's face, it has been proposed to develop a deformable three-dimensional human face model, and then deform this model to minimize the gap between facial features on the face. image and corresponding points of the model.

In particular, it has been proposed in application FR 1261025 a three-dimensional model of human face, the model being a linear combination of a human face template (for example an average face shape made from several faces), and deformations between this template and a plurality of example faces.

The deformation of the model is also constrained to keep the shape of the model that of a human face, to avoid any risk of aberration even when the image from which the face model is generated is incomplete or contains noise. This model has already made it possible to improve the precision of the frontalized views of individuals and has improved the results of comparisons between the views of individuals obtained through models and the authentic views of individuals. However, the fact that the shape of a face can be obtained only by the linear combination of facial examples limits the accuracy of this model because the characteristic points of the face shape thus obtained can only approach those of the face authentic. In addition, it is necessary to acquire more examples of human faces (from which the linear combination is made) to complete the model and make it more precise. But this type of acquisition is long and expensive to achieve.

PRESENTATION OF THE INVENTION The purpose of the invention is to propose a method of three-dimensional modeling of a face that does not have the aforementioned drawbacks and that makes it possible to obtain a more precise shape. In particular, an object of the invention is to propose a method of modeling a face that makes it possible to obtain a face shape whose characteristic points correspond exactly to those of the face represented on the image from which the image is made. modelization.

In this regard, the invention relates to a method of three-dimensional modeling of a face represented on a two-dimensional image, comprising: a acquisition of coordinates of at least one point of the surface of the face on the image, a first estimate of the shape of the face, the shape comprising a vertex array each corresponding to a point on the surface of a face, and the estimation of the pose of the face on the image, the method being characterized in that it further comprises a step of deforming the shape of the face to obtain a refined estimate of the shape of the face on the image, the deformation being continuous with respect to the first estimate of shape of the face and the coordinates of the projection on the image of at least one vertex being constrained to correspond to the position of a corresponding point of the face on the image.

Advantageously, but optionally, the modeling method according to the invention further comprises at least one of the following features: the method comprises a second step of acquiring a set of points of the face, the vertex coordinates corresponding to said points of the face being constrained during the deformation step to correspond to the coordinates of said points detected during said step. the second acquisition step comprises displaying a two-dimensional projection of the face shape, superimposed on the face on the image, and locating the coordinates of one or more points of the face on the image by an operator by aligning one or more vertexes with corresponding points. during the deformation of the face shape, the coordinates of a vertex corresponding to a point of the face are constrained to belong to a straight line passing through the corresponding characteristic point of the face on the image and orthogonal to the plane of the face; picture. the continuous deformation of the shape of the face comprises the minimization of a derivative of the difference between the face shape obtained at the end of the first estimate and the shape obtained at the end of the deformation. the constraint of continuity of the deformation of the shape of the face comprises the minimization of the norm of the second derivative of the transformation of each triangle whose vertices are formed by three vertexes, said norm being of the type Es = ciikillvec (iliik) - 2 where N is the set of adjacent triangles of the face shape, i, j, k, and I the vertex indices, Aiik the deformation matrix of a triangle whose vertices are the vertices i, j, and k , with the concatenation operator of a matrix in a column vector, and found a constant weighting the contribution of the triangles ijk and ijl, depending solely on the shape of the face before deformation. The method further comprises approximating normals to triangles whose vertices are formed by three vertexes by additional variables, and deforming the shape of the face further comprises minimizing the difference between the additional variables and the normal ones. to the triangles of the shape of the face obtained after deformation. The method further comprises the repetition of the steps of acquisition of a set of points of the face, and of the constrained deformation of the shape of the face, said deformation step being implemented by aligning new vertices corresponding to points of the face; face to the corresponding points of the face on the image. The method further comprises the steps of: 0 from the estimation of the pose and the shape of the face of the individual on the image, generating a representation of the texture of the face of the individual, and 0 generate a front view of the individual's face.

The invention further relates to a processing unit, comprising processing means configured for implementing the modeling method according to the foregoing description.

The subject of the invention is also a computer program product, comprising code instructions for implementing the modeling method according to the preceding description when the latter is implemented by means of processing a unit. treatment. The invention also relates to a method for identifying or authenticating an individual, characterized in that it comprises: - from a face image of the individual, the generation of a view of face of the face of the individual by the implementation of the modeling method described above, and - the implementation of a face recognition treatment by comparing the front view of the face of the individual with a or several images of reference faces. Finally, the subject of the invention is an authentication or identification system for an individual, comprising a control server of an individual to be identified, and at least one management server of a reference image database. individuals listed, the control server comprising acquisition means adapted to proceed to the acquisition of an image of the face of the individual, the identification system of individuals being characterized in that one of the server control and the management server includes processing means adapted to implement the modeling method according to the foregoing description, and, from a front view of the face of a person obtained, implement a treatment face recognition by comparison with reference images of the base, in order to identify the individual.

The proposed method makes it possible to exactly match the positions of the characteristic points of the face image to those of the modeled face shape. The subsequent deformation of the face shape maintains the position of these characteristic points while adapting the general shape of the face so that it corresponds as closely as possible to the initial estimate, which was established to correspond to a human face. A more accurate modeling of the face represented on the image is thus obtained and the results of the comparisons made from images generated by this modeling are better.

DESCRIPTION OF THE FIGURES Other characteristics, objects and advantages of the present invention will appear on reading the detailed description which follows, with reference to the appended figures, given by way of non-limiting examples and in which: FIG. schematically the main steps of the modeling method according to one embodiment of the invention. - Figure 2 schematically shows the main steps of a method of generating a face model that can be implemented in the modeling process. - Figure 3 represents the characteristic points of a human face. FIG. 4 represents notations used for calculating a differentiation matrix. FIG. 5 represents the main steps of an embodiment of the estimation of the pose and the first estimate of the shape of the face in the modeling process. - Figures 6a and 6b respectively show the shape of the face and its superposition to the image after the first estimate. - Figures 7a and 7b show the shape of the face and its superposition to the image after the refined estimate. FIG. 7c represents the registration of other vertexes corresponding to characteristic points for the reiteration of the refined estimation step of the shape of the face. - Figures 8a and 8b show the shape of the face and its superposition to the image after the reiteration of the refined estimate. - Figure 8c shows a frontal image made from the modeling of the face. FIG. 9 represents an exemplary embodiment of a system for identifying or authenticating an individual from the modeling of the face.

DETAILED DESCRIPTION OF AT LEAST ONE EMBODIMENT OF THE INVENTION Method for Modeling a Face With reference to FIG. 1, a method of three-dimensional modeling of an object, preferably a face of an object, will be described. individual, represented on a two-dimensional image. This method may include an acquisition step (not shown in the figure) of the image on the basis of which is generated the three-dimensional model of the object on the image. This method is implemented by a processing unit 10, provided with processing means such as a processor, and code instructions whose execution by the processor allows the implementation of the method.

In computer science, every three-dimensional object such as a human face can be described using the following three elements: - The shape of the object, which is composed of a set of 3D vertexes, each vertex being a point of the object defined by coordinates along three orthogonal directions, and corresponding to a point on the surface of the object.

We denote by V the number of vertices v, of an object, each object being described by a matrix of dimension 3x V S = (vi, ..., vN) in which the vertices are arranged in columns. - The surface of the object: it is materialized by connecting vertex between them to form triangles. A list of triangles is thus defined for each object, each triangle being indicated by the three indexes of the corresponding columns of the matrix S. - A representation of the texture of the object: it is an image used to color the image. object in three dimensions obtained from its shape and its surface. The surface of the object defined by the triangle list is used to match the vertices of the object to a particular texture. First estimation of the shape and estimation of the pose The method comprises a first detection step 1000, on the face image from which the three-dimensional modeling is estimated coordinates of a set of points of the face. These points are advantageously characteristic points of the face shown in Figure 3, typically 22 in number, and which are the corners of the eyes, the ends of the mouth, nostrils, the tip of the nose, ears, etc.

This detection can be done automatically by an image processing software executed by the processing unit 10, or manually by an operator who points via a human-machine interface the corresponding points on the image of the face.

In this regard, reference may be made to the following publications: Yow et al. Feature-based human face detection. Image and Vision Computing, 15 (9): 713-735, 1997. Nikolaidis and Pitas, Facial feature extraction and determination of pose, Proc. Of the NOBLESSE VVorkshop on Nonlinear Model Based Image Analysis, page 257-262, 1998. - Lee et al. The method then comprises a step 2000 of estimation of the shape and the pose of the face on the image by generating a 3D vertex network whose positions match the shape of the face on the image. This step is implemented by the processing unit 10 by deforming a first 3D vertex array, which advantageously, but not exclusively, initially corresponds to a human face shape, so as to approach the shape of the face in the image . The first 3D vertex network can be a face shape of a particular individual, or be a middle shape on a sample of individuals. The initial vertex network can be loaded from a database or obtained by computer modeling or, if it is made from a particular individual, by acquiring the shape of the face of this individual by a scanner 3D. According to a preferred embodiment, the first 3D vertex array is a three-dimensional model of a face obtained according to the method described in application FR 1261025 and recalled hereinafter with reference to FIG.

This model is formulated mathematically as a linear combination of examples of individuals' faces, noted S = S ° + alS1 where S ° is a human face shape template, constituting the basis of the model, and S ° + SJ represents the facial shape of a particular example of a real individual. Therefore, SJ represents the gap between one of the face examples and the template.

The coefficients al are in turn determined later to distort the model S to make it correspond to the face of an individual that we want to identify. We will now describe the steps to obtain this model. During a step 2110, the processing unit 10 generates the template S ° of human face: it may be a face shape of a particular individual, or an average of facial shapes. 'a plurality of individuals. In all cases, the face shape or shapes are defined by a series of vertex corresponding to points of the face. These points comprise, among others, the characteristic points of a face described above with reference to FIG. 3, the number of N. As previously, these characteristic points can be manually marked by an operator from a frontal image. face, or they can be automatically detected by software executed by a processor such as the processing unit 10.

The human face template also includes the order of a few thousand other vertex acquired by a 3D scanner. During a step 2120, we proceed to the acquisition of examples of faces of real individuals. This acquisition is implemented in the same way as above, by identifying the characteristic points of the faces of the individuals to generate a list of vertex or by acquiring the totality of the shape of the face by 3D scanner. The shapes of faces thus acquired each correspond to a S ° + SJ. To construct the three-dimensional model, the SJ deviation between the face and the template is determined from the vertex lists of each face shape.

However, all the forms generated by the three-dimensional model S must be possible face shapes, and not mathematical aberrations.

To ensure this result, the processing unit puts all the examples of face shapes in correspondence, that is to say by associating each vertex of a face with a defined number. For example, a given number is attributed to the tip of the nose and another number to the left commissure of the lips. These numbers correspond to the vertex indices. The peculiarity of the template is that it is a form of face for which vertex indexing is already done. Therefore, vertex indexing of each example of a face shape is performed by mapping in step 2130 the vertices of each shape example with the vertices of the template. To do this, the processing unit deforms the template in a step 2131 iteratively (that is to say, changes the vertex coordinates of the 3D shape) to minimize the gap between the shape of the template and that of the face example, the deformed template always having to be a human face shape. The mathematical function to be minimized includes three terms. The first term serves to minimize the distance between the characteristic points of an example of a face and the corresponding points of the template. It is written: Ns Ilski - vkill2 i = 1 Where i is the index of a characteristic point, ski is a vertex of a point of an individual example face corresponding to the characteristic point i, vki is a vertex of a point of the template after deformation corresponding to the same characteristic point i, and Ns is the number of characteristic points in a face, for example 22.

It is therefore sought to modify the positions of the characteristic points of the template iteratively to correspond to the positions of the same characteristic points on the face example. The second term is used to match the surface of the face shape of the template with the surface of the shape of the face example. The function to be minimized represents the distance between the points of the template and the surface of the face example. It is noted: Where pvki is a point on the surface of the face example, ie a point corresponding to the projection, orthogonal to the surface of the template, of the vertex vki on the surface of the face of example. It is possible that the surface of the face example is incomplete, if for example it is obtained from a non-frontal image, and that points of the template do not correspond to any point of the face example. In this case, these points of the template are not taken into account. The third term constrains the deformed template to remain a real human face, even if the face example used for deformation of the template is incomplete or contains noise. This term makes the deformed template as "smooth" as possible, that is to say as continuous as possible, by minimizing the standard of the second derivative of the transformation of the template, at each iteration. This norm is expressed in the following way: IIA (v - vec (S °)) 112 Where v is the concatenation of the 3D vertices of the deformed template, and vec (S °) the same term for the template before transformation. v and vec (S °) are vectors of size 3V x 1. The derivation of a function being a linear operation, its computation can be realized by the multiplication of the function by a matrix. In this case, A is a vector differentiation matrix v-vec (S °), of dimension 3T x 3V, where T is the number of triangles of the surface of the template.

The derivative is calculated for each triangle t of the surface of the template, the derivative of the deformation of a triangle t being calculated with respect to triangles q neighbors of the triangle t, by approximation of the finite difference of the triangle t with the neighboring triangles q as follows: dg - dt cieNt wq't Where Nt is the set of triangles q neighbors to the triangle t, wo is a weighting factor that depends on the surfaces of triangles t and q, dt is the deformation of triangle t at the level of its barycenter, and bt is the position of the centroid of the triangle t. The distance between the centers of gravity and the weighting factor are calculated on the undeformed template S °.

With reference to FIG. 4, the weighting factor wo is the sum of the surfaces of two triangles whose base is the edge connecting the triangles t and q, and the opposite vertex to this base is respectively the barycentres bt of the triangle t and that bq of the triangle q.

To obtain the deformation dt of the triangle t at its center of gravity (that is to say the displacement of the center of gravity between the undeformed template and the deformed template), the deformation of the form is multiplied (v-vec (S °). )) by a matrix Bt of dimension 3 x 3V which is null everywhere except to the elements associated with the vertex of the triangle t. These elements are then equal 1/3.

The barycenter bt of the triangle t being the average of its three vertexes, the multiplication of this matrix B to the deformation (v-vec (S °)) makes it possible to obtain the displacement of the barycentre of the triangle. The matrix A, of dimension 3T x 3N, is obtained by vertically concatenating the set of matrices Bt associated with each triangle t, whose coefficients corresponding to the vertex of a triangle t are multiplied by the weighting factors wo and divided by the Distances between the centers of gravity Ilbq - It is noted that the differentiation matrix A depends solely on the surface of the undistorted template (list of triangles of S °), and not on the shape of the deformed template v. It is therefore constant.

The three terms described in detail above are minimized simultaneously, so we determine: Ns min lisi - v112 + yllA (v - vec (S °)) 112 i = ti = t Where K, and y are weighting coefficients of each term. This minimization can be solved linearly by decomposition into singular values. Since this minimization is iterative, at the beginning of the matching step, the template may be arbitrarily removed from the individual's face example, and thus the pv points, of the facial surface closest to the points. v, of the template are not well defined. We then set a low value for K compared to the other weights. In addition, a significant value is set for y to ensure that the transformation is almost rigid, that is to say that the shape of the face of the template is the least distorted possible. With each iteration of the minimization, one increases the value of K. At each iteration, the points pv, are searched on the surface of the example of face of individual, as being the closest to the points v, of the template deformed with this iteration. As the minimization is iterated, these points pv, are more and more reliable and the value of the coefficient y is decreased to make the comparison more flexible. This iterative mapping step is performed for each individual face example. It results in a deformed template, which corresponds to an example of a face, from which we can deduce the value of SJ, the deviation between the template and the example of the face. Thus, at the end of this step, we obtain a deformable three-dimensional face model, including the S ° template and the SJ deviations, which can be made linear combinations to obtain any individual face. Once this initial model obtained, the processing unit 10 uses it to obtain an estimate of the pose and the shape of the face of the individual on the image. The shape of the face on the image is estimated by iteratively deforming 2200 the model obtained at the end of step 2100 so that it corresponds to that of the face on the image, as indicated above. To do this, it can implement for example the method described in the application FR 1261025 and recalled below, with reference to Figure 5. Alternatively, one can implement the method described in the following publications: - Koterba and al, "Multi-View AAM Fitting and Camera Calibration", ICCV 2005, - Brand, ME, "Morphable 3D Models from Video," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), December 2001. - Ira Kemelmacher-Shlizerman Steven M. Seitz. "Face Reconstruction in the VVild." The processing unit implements a step 2210 of so-called "rigid" estimation of the pose, or position, of the face on the image. The estimate is called rigid because it does not include deformation of the face.

The pose is defined relative to a reference by using six parameters: three rotation angles, two translation parameters and a scale factor and is defined as follows: p = s. R. v + t Where p is a two-dimensional vector, comprising the X and Y coordinates of the projection of each vertex v in three dimensions, s is the scale parameter, R is a 2 x 3 matrix of which the two lines are the first two lines of a rotation matrix, and t is a translation vector in X and Y. The rotation matrix is written according to the angles of Euler ax, ay, and a, as follows: I 1 0 0 \ / cos (a) 0 - sin (ay) cos (%) -sin (a) 0 \\ V \ 1 0 0) (o 0 cos (a) sin (a) 0 1 0 cos (a) sin (a) 0) i 1 0 -sin (a) cos (ax) sin (ay) 0 cos (aY 0 0 1 To estimate the pose, we exploit the positions of the points identified during the step 1000 Optionally, step 1000 can then be implemented only at this time, then the positions of these points are compared with the vertex projections of the corresponding points of a typical example of a face, which may be in the form the SO template used to generate the three-dimensional model. performed by iteratively modifying the pose of the face template, by varying the parameters mentioned above, to minimize the difference between the projections of the vertex of the face of the individual and the template as follows: 22 minII - (sRvi + t) 112 S, ax, ay, az, ti = 1 Where p, is the position of a characteristic point i on the image and v, is a vertex of the corresponding point i of the template. Depending on the image of the individual available, which is non-frontal, certain characteristic points may be invisible on the image or their position may be uncertain. Each characteristic point i is therefore assigned a weighting coefficient c, representative of the "confidence" in the position of the point. If a point is invisible on the image, then its confidence coefficient is zero.

The determination of the pose of the individual on the image is then written as follows: 22 min ci lipi - (s R. vi + 011 s, ax, ay, az, ti = i The pose obtained for the template to the end of the minimization is the pose of the face of the individual on the image.

This optimization problem is solved with a two-step procedure, the first step 2211 being the linear search of a solution, and the second step being the nonlinear minimization 2212 to refine the estimate of the pose obtained with the first one. step. Linear estimation step 2211 will now be described.

This estimate assumes that the distance between the positions of the characteristic points and the modeling of their projection, called the "rear projection" error, is Gaussian with a zero mean and a standard deviation of -, and that if the error is independent for all points, so we can show that the solution of the previous equation is also the solution of the following linear system: With Ax = b CiVi oT 0 \ OT 0 civf c1 'A = c2vf C2 oT 0 - - - C2 C22 Vf2 C22 OT 0 OT C221212 C22 / XT = (sri1 sr12 sr13 tx sr21sr22 sr23 ty) = (ctPx, 1 C1Py, 1 --- C22Px, 22 C22 Py, 22 The resolution of this overdetermined linear system is standard linear algebra and is performed using the pseudo-inverse given by the singular value decomposition described in Golub et al., Matrix computations volume 3, Johns Hopkins Univ Pr, 1996. This first linear resolution step 2211 provides a good estimate of depar However, since the assumption previously adopted for the linear resolution is not based in practice, the estimation needs to be refined by the non-linear estimation step 2212. The linear step is refined by implementing a non-linear iterative step 2212, for which a preferred method is the Levenberg-Marquadt minimization. Reference may be made to Gill et al. Practical Optimization. Academic Press, London and New York, 1981. This step finally makes it possible to obtain a first estimate of the pose of the face of the individual on the image, this pose being then refined during the step 400 of estimation "flexible Of the pose and the shape of the face. It is therefore considered that at this stage a "draft" of the pose of the face has been determined. We will now describe the step 2220 of flexible estimation of the pose and the shape. This estimation is carried out using the three-dimensional face model obtained in step 2100. As indicated above, this model is written in the form of a linear combination of the template SO and the deviations of this template with respect to examples of individuals: S = S ° + aiSi The shape of any face can be obtained by choosing the coefficients ot1 of the linear combination. The flexible estimation of the shape and pose of the face of the individual on the image is therefore performed by the processing unit 10 by minimizing the difference between the projections of the characteristic points p, of the face of the individual. on the image, and the same projections of the model. To do this, it is iteratively modified the shape of the face obtained by the model (thanks to the coefficients al) and the parameters of pose of the face.

Mathematically, we therefore try to obtain the following minimum: 22 min / a, s, ax, ayaz, t pi- (sR (SP + t) i = 1 i = 1 2 However the resolution of this equation could lead to a form the deformed face model that no longer corresponds to a human face, because the characteristic points p of the face of the individual may be noisy or inaccessible, and the system would not be determined enough.

The al coefficients are then constrained to guarantee a realistic human face. For this, the standard of the second derivative of the deformation of the three-dimensional model is minimized, the deformed vertexes of the model being here defined as a function of the vector a comprising the al for j between 1 and M. The derivative of the deformation The three-dimensional model is obtained by multiplying the deformed model by a matrix A 'constructed in the same manner as the matrix A of previous differentiation. This minimization step corresponds to a hypothesis of continuity of the face, which is verified regardless of the individual and therefore allows the process to be as general as possible, that is to say applicable for any individual. We thus obtain the following equation: 22 min / Ci a, s, ax, ayaz, ti = 1 pi - (sR (SP + t) i = 1 2 + This equation is solved, in a similar way to the step of non-linear minimization, using the Levenberg-Marquardt minimization algorithm The initialisation of the pose is provided by the pose obtained at the end of the rigid estimation step The initial form used for the minimization is that of the template S ° of origin, ie the values of the initial al coefficients are null Once this estimate is implemented, we obtain a 3D shape corresponding to the shape of the face on the image, according to a first estimate 2000, and represented in Figure 6a.

Returning to FIG. 1, the method then comprises the refinement of the estimate of the shape of the face on the image, while leaving the pose unchanged. Refined estimate of the shape of the face As can be seen in FIG. 6b, the face shape obtained at the end of step 2000 is not exact because, in particular, at easily recognizable facial characteristic points such as the commissures lips, the ends of the eyebrows, etc., the vertex of the 3D shape corresponding to these points are not projected on the image exactly at the positions of the corresponding characteristic points.

Therefore, the method comprises a step 3000 refined estimation of the shape of the face, by constraining certain vertex coordinates of the shape of the face so that these coordinates are accurate. In particular, during step 3000, the first estimate of shape is refined by constraining the positions of certain vertexes, corresponding to points of the face, so that the projections of these vertexes in the plane of the image correspond exactly to at the positions of the corresponding points. In other words, at the end of step 3000, the coordinates of certain vertex projected in two dimensions on the image correspond exactly to those of the corresponding points of the face. Advantageously, the 3D vertices whose two-dimensional projections are thus aligned at corresponding points of the face are those corresponding to easily recognizable characteristic facial features, such as the 22 characteristic points of a human face (tip of the nose, commissures lips, ends of the eyebrows ...), as indicated above. However, the invention is not limited to this number and there may be more or less vertex involved. In the following, we call "vertex constraint" a vertex whose coordinates of the projection on the image plane are forced to particular values corresponding to the coordinates of the corresponding point of the face on the image. This refined estimation step comprises a first step 3100 for acquiring the positions of points of the image corresponding to the constrained vertexes. According to a first embodiment 3110, the positions of each point of the face corresponding to the vertex to be reset can be automatically identified by computer. In particular, these may be the positions already detected automatically during step 1000, in which case this step simply comprises a loading of the previously recorded coordinates of the points located in step 1000. In this case, these are advantageously all the points detected in step 1000 which constrain the projections of the corresponding vertices, because this allows a greater accuracy of the final form. According to a second embodiment 3120, which is more advantageous because it is more accurate, the exact positions of the points of the face are determined by an operator.

In this case, the implementation of step 3120 comprises a first step 3121 consisting in displaying a two-dimensional projection of the face shape on a man-machine interface, conferring the same pose on the shape as that of the face on the image (estimated at step 2200), and superimpose the projection to the image. Then, an operator can move the projections of a set of 3D vertex to superimpose exactly on the corresponding points of the face on the image. The implementation of this step by an operator makes it possible to guarantee that the positions of the incorrected vertices correspond exactly to those of the corresponding points of the face. Optionally, at the end of this step, the processing unit can again implement a step 2200 'of adaptation of the model, including an estimate of the pose and the shape of the face from the new coordinates. The method then comprises a step 3200, implemented by the processing unit, of refining the estimation of the shape of the face from the coordinates of the 3D forced vertexes. This refined estimate is implemented by again deforming the face shape obtained at the end of step 2200 or 2200 ', by forcing the coordinates of the projections of the vertexes forced to be those determined at step 3100. In the case where the registration of the vertices of step 2000 is implemented automatically by computer, this step 3200 - or, if appropriate, the succession of step 2200 'and then step 3000 - is advantageously done simultaneously, that is that is, the computer detects on the image the coordinates of the face points to be mapped to the vertexes and simultaneously realizes, from these coordinates, the registration of the constrained vertexes and the deformation of the remainder of the face. In the case where the vertex registration is implemented by an operator, step 3200 - or if appropriate step 2200 'and then step 3200 - can be implemented after the operator has moved all vertex projections to be repositioned.

Alternatively, the processing unit can recalculate the position of the set of 3D vertices according to step 3200 - or, if necessary, implement step 2200 'and then step 3200 - as soon as an operator has set the vertex coordinates by mapping a vertex to a point in the face at step 3100, not just after the operator has re-aligned all 3D vertices. Thus, the estimation of the shape of the face is refined with each new vertex fixed with respect to a corresponding point of the face. The positions of the vertices displaced during the deformation 3200 are constrained so that the deformation between the face shape obtained at the end of step 1200, and the final shape of the refined estimate is as continuous as possible. Since the vertex positions of the characteristic points are fixed on the image, the coordinates of these vertexes in 3D can always vary since the depth coordinate of the vertex with respect to the plane of the image can vary along a straight line. by the position of the characteristic point on the image and orthogonal to the plane of the image. The equation which projects a 3D vertex whose coordinates are (X, Y, Z) on a point in two dimensions (x, y) are the following: X) c, _ sR2,3 [-y1 + [t tx r ] where s is the scale factor, tx and ty is the translation in 2D and R is the rotation matrix, all these parameters resulting from the pose estimated during the step 1000 and being constant for the continuation of the process. R2,3 constitutes the first two rows of the 3D rotation matrix R. R2,3 is of rank 2 and its decomposition in singular values is: R2,3 [01 011 [01. 01 oi _I. V3,3 Where V3,3 is an orthonormal matrix (so its transpose is its inverse). The core of R2,3 is therefore the third column of V3,3, which means that the projection of the 3D point (X, Y, Z) on the plane of the image is unchanged if this point is moved along a straight line whose direction is provided by the third column of V3,3. We thus obtain, by the estimation of the pose of the face, the direction in which the depth of a vertex of a characteristic point can vary.

We thus obtain, for such a point, that we call X: R2,3.X = R2,3 (X + / 117:, 3) Ot.i V, 3 is the third column of V3,3 and A represents the variation of the position of the point along the line extending in the direction given by V, 3, for each vertex.

Since V is orthonormal, this equation can be reformulated as follows: 1 0 1 - tx = V3,3 [0 11 [X x] - 3 = A + b., S [Y t 0 0 y If a 3D vertex x satisfies this equation, it is projected on the two-dimensional point of coordinates (x, y). The 3D vectors a and b are constants, so À is a unique unknown for each constrained vertex.

The refined estimate step 3200 of the facial shape therefore comprises the determination, for each vertex constrained (ie, recalibrated to be superimposed on a characteristic point of the face), of the value of λ, and the determining the coordinates of the other deformed vertices for which the deformation from the first shape estimate is as continuous as possible.

These coordinates are determined by minimizing the norm of the second derivative of the deformation of each triangular facet of the form whose vertices are formed by vertexes. This corresponds to minimizing the term: Es = - vec (Aiii) 112 Where N is the set of adjacent triangles of the face shape, i, j, k, and I the vertex indices of two adjacent triangles, Aijk the matrix of deformation of a triangle whose vertices are the vertex i, j, and k, which corresponds to a sub-matrix of the differentiation matrix A calculated previously with reference to FIG. 4, with the concatenation operator of a matrix in a column vector, and cilia the constant wedbq - bM (the triangles q and t being formed by the vertex i, j, k and I, j, k), depending solely on the shape of the face before deformation.

The differentiation constant cilia is determined by calculating for each triangle t the derivative of the deformation of the triangle with respect to the neighboring triangles. It is a constant term that depends solely on the initial shape of the initial face. This term is also detailed in B. Amberg's thesis, Editing Faces in Videos, PhD Thesis, University of Basel, 2011. The matrix Aijk of deformation of a triangle is given by the following equation: Aiik. lek = X. Kiik The term eik is a 3 * 3 matrix whose first two columns are the edges connecting the vertex i to j and the vertex i to k, the third column being the normal to the triangle ijk of the initial form. The matrix X is formed by the position, after deformation, of the vertexes and normals to the triangles formed by these vertex arranged in columns. It therefore has 3 rows and V + T columns, where V is the number of vertexes and T is the number of triangles. The X matrix is the unknown of the problem of minimization. The Kijk matrix is the topology matrix of the triangle ijk, it is a hollow matrix, which, when multiplied by the matrix X, gives the edges of a triangle and the normal to the triangle as the matrix eik but for the final triangle, after deformation. This matrix therefore has V + T rows and three columns. It is zero everywhere except: - At line i, where the value of the first two columns is -1, - At line j, where the value of the first column is 1, - At line k, where the value of the second column is 1, and - At the line V + 1,, k where the value of the third column is 1. the is the index of the triangle ijk in the list of triangles. The objective being to minimize the term Es indicated above while respecting the condition on the constrained vertex, we can reformulate the last equation as follows: vec (ilijk) = ((Kijk.rek 1) T 0/3) vec (X) = Dijkvec (X) The operator 0 is the product of Kronecker and Duk is a matrix that has 9 rows and 3 (V + T) columns. The minimization of the term Es is therefore equivalent to minimizing the norm to the square of the vector es: es = Js.vec (X) The matrix Js is the vertical concatenation of the matrices Duk-Dul, it is a matrix column of dimension 9E * 3 (V + T), where E is the number of sides of the 3D surface of the face that are connected to two triangles, and is the number of terms in the sum of the equation of Es.

As some vertexes are free and others constrained, the exponent A denotes the free vertex and the exponent B the constrained vertex. The exponent Bi thus indicates the constrained vertex, so by separating the contributions of the free vertex and the constrained vertex the previous equation corresponds to: = vec (XA) Here vec (XA) is a column vector of 3 ( VA + T) lines, xBi is a column vector of 3 lines, g is a matrix of (9E, 3 (VA + T)) and is a matrix (9E, 3), but for a 3D vertex i constrained, as previously seen this vertex is bound by the following relation: xBi = Àiai + bi And so the term es can be formulated as follows: es = J. .vec (XA) +1 (JsBi. aiAi + JsBibi) Which corresponds, in noting F the number of vertexes constrained, and adopting the following notations: - Ls = / flat ... JsBFaFl - XT = [vec (XA) T À1 .-- AFI - bs = EiJsBl bi, To the term reformulated thus: es = Ls2T + bs Or, as indicated above, the matrix XA includes the position of the free vertexes and the direction of the normals of the 3D triangles obtained after deformation.The normals depend in a non-linear way of the vertexes. the matrix Ls also depends on the vertex positions, so that the last equation obtained for the term to be minimized is not linear with the vertex position.

To be able to implement this minimization, we introduce new variables, called "normal dummies" (in English "slack normals") h, which replace the true normals in the vector x. These normals make it possible to lift the non-linearity with respect to the vertex positions.

Moreover, in order to make the dummy normals as close as possible to the normals of the triangles obtained after deformation, we add a new cost function En defined by: En = 111fit ntii2 tE1 'The letter 7- denotes the set of triangles of the three-dimensional form of the face. This cost function Is is minimized iteratively, simultaneously with the function Es, so that during the determination of the vector x the dummy normals approach the maximum of normals. At each iteration i, we use as normal nt, the normals obtained at the previous iteration. Therefore, the minimization of the cost function En is equivalent to minimizing the quadratic norm of the vector in: en = in.x - ni-1 where ni-1 is a column-vector of 3T elements corresponding to the stacking of the norms at triangles obtained after deformation at the iteration i-1. The matrix Jn is a matrix with 3 rows and 3 (VA + T) + F columns. The columns that correspond to the vertexes to be determined (the first 3 VA first columns and the last F columns) are null, and the columns that correspond to the triangles (the middle 3T columns) are filled by the Identity matrix: fn = [0 1T 0] The minimization of the weighted sum of the two cost functions Es and En, Es + VTcEn, where k is a weighting factor, which is a positive integer, is equivalent to the minimization of the quadratic norm of the following vector: e = [s 2 [= J2 b [kin] We introduce here to simplify the notations of the Jacobian matrix J and the vector b. This system of linear equations is overdetermined. Its least squares solutions are provided by the following equation: x = (JTJ) -1-rb Therefore, the constrained deformation of the face shape of step 4000 is implemented by the processing unit by solving this last equation iteratively until convergence. The weighting coefficient k is advantageously chosen of a limited value so as not to implement a too large number of iterations, which would lengthen the calculation time. Thus, by way of nonlimiting example, the weighting coefficient k can be between 1 and 10, and for example be equal to 1. Once matrix 2 is obtained, the coefficients of free 3D vertexes are thus obtained, and the positions in depth (along the line orthogonal to the image plane) constrained vertex, which provides all the information on the shape of the face. We thus obtain the face shape shown in Figure 7a, the superposition to the face image corresponds exactly to the characteristic points, as shown in Figure 7b.

To further perfect the obtained face shape, the steps 3100 of vertex alignment on characteristic points and 3200 of constrained deformation of the shape of the face can be repeated one or more times (arrow F figure 1), step d vertex alignment is then implemented for vertex corresponding to characteristic points not yet aligned with the corresponding characteristic points of the face. With reference to FIG. 7c, the points identified by reference P are recalibrated, and at the end of the process, as in FIGS. 8a and 8b, a face shape is further refined with respect to the face of the image. .

Following the three-dimensional modeling of the face, the method may then include post-processing steps 4000 to exploit the three-dimensional modeling to perform identification or authentication of the face of the image. These steps include an estimation step 4100, from the shape and pose of the face, the texture of the face on the image. To do this, the processing unit 10 samples the original image at the positions of the points of the three-dimensional shape and extrapolates the texture for the points of the shape of the face not visible in the image.

The processing unit 10 finally generates during a step 4200, from the shape of the face, positioned from the front, and the representation of the texture of the face, a two-dimensional image of the front of the individual. This image is illustrated in Figure 8c. It can serve as a basis for a conventional identification or authentication method by face recognition 4300. Such a method is conventionally implemented in a system 1 shown in FIG. 9, comprising a management server 20 associated, if necessary, with a base. of data; and a control server of an individual to identify. The control server 10 and the management server 20 may be merged. The control server 10 is advantageously the processing unit adapted for implementing the three-dimensional modeling method according to the preceding description. The control server is advantageously associated with or equipped with image acquisition means 11. Thus, the control server 10 can acquire from an individual to control an image, which is the one from which the three-dimensional modeling of the face is performed, and then the generation of the front image of the individual. Once this image has been obtained, the control server can communicate it to the management server 20 which confronts it with one or more reference images, either stored in the database or acquired elsewhere, to determine whether the The individual of the image obtained by the control server 10 corresponds to the image or one of the images to which it is compared. The facial pattern modeling method increases the success rate of identification and authentication. 25

Claims (13)

REVENDICATIONS1. A method of three-dimensional modeling of a face represented on a two-dimensional image, comprising: an acquisition (1000) of coordinates of at least one point of the facial surface on the image, a first estimate (2200) of the shape of the face, the shape comprising an array of vertices each corresponding to a point on the surface of a face, and the estimation of the pose of the face on the image, the method being characterized in that it comprises in in addition to a step of deforming (3200) the shape of the face to obtain a refined estimate of the shape of the face on the image, the deformation being continuous with respect to the first estimate of shape of the face and the coordinates of the projection on the face. image of at least one vertex being constrained to correspond to the position of a corresponding point of the face on the image. 2. The modeling method as claimed in claim 1, further comprising a second step of acquiring (3100) a set of points of the face, the vertex coordinates corresponding to said points of the face being constrained during the deformation step ( 3200) to correspond to the coordinates of said detected points in said step (3100). The modeling method according to claim 2, wherein the second acquisition step (3100) comprises displaying a two-dimensional projection of the face shape, superimposed on the face on the image, and the location of the coordinates of one or more points of the face on the image by an operator by aligning one or more vertices with the corresponding points. 4. Method according to one of claims 2 or 3, further comprising the repetition of the steps of acquisition (3100) of a set of points of the face, and stress strain (3200) of the shape of the face, said step deformation being implemented by aligning new vertices corresponding to points of the face at the corresponding points of the face on the image. 5. Modeling method according to one of claims 1 to 4, wherein, during the deformation (3200) of the face shape, the coordinates of a vertex corresponding to a point of the face (xBi) are constrained to belong to a straight line passing through the corresponding characteristic point of the face on the image and orthogonal to the plane of the image. The modeling method according to claim 1 to 5, wherein the continuous deformation of the face shape (3200) comprises minimizing a derivative of the difference between the face shape obtained at the end of the first estimate. (2200) and the shape obtained at the end of the deformation. The modeling method according to claim 6, wherein the continuity constraint of the deformation of the shape of the face comprises the minimization of the standard of the second derivative of the transformation of each triangle whose vertices are formed by three vertices, said vertex standard being of the type Es = cipallvec (Aiik) - 12 (i, j, k, OEN where N is the set of adjacent triangles of the face shape, i, j, k, and I the vertex indices, Ail ( the deformation matrix of a triangle whose vertices are the vertex i, j, and k, with the operator of concatenation of a matrix into a column vector, and Ciiki a constant weighting the contribution of the triangles ijk and ijl, dependent only the shape of the face before deformation. The method of claim 7, further comprising approximating normals (nt) to triangles whose vertices are formed by three vertexes by additional variables (fit), and deforming the shape of the face further comprises minimizing of the difference (in) between the additional variables and the normals to the triangles of the shape of the face obtained after deformation. 9. Modeling method according to one of the preceding claims, further comprising the steps of: from the estimation of the pose and the shape of the face of the individual on the image, generating a representation of the texture of the individual's face (5100), and- generate a front view (5200) of the individual's face. 10. Processing unit (10), comprising processing means configured for implementing the method according to one of the preceding claims. 11. Computer program product, comprising code instructions for implementing the method according to one of claims 1 to 9 when the latter is implemented by processing means of a processing unit. 12. A method of identifying or authenticating an individual, characterized in that it comprises: - from a face image of the individual, the generation of a front view (5200) of the face of the individual by the implementation of the method according to claim 9, - the implementation of a face recognition treatment (5300) by comparing the front view of the face of the individual with one or more images of reference faces. 13. System (1) for authenticating or identifying an individual, comprising a control server (10) of an individual (I) to be identified, and at least one management server (20) of a base reference images of individuals listed, the control server (10) comprising acquisition means adapted to proceed to the acquisition of an image of the face of the individual (I), the identification system of characterized in that one of the control server (10) and the management server (20) has processing means adapted to implement the method of claim 9, and from a view facial face of a person obtained, implement a face recognition treatment by comparison with the reference images of the base, to identify the individual.