RU2582852C1 - Automatic construction of 3d model of face based on series of 2d images or movie - Google Patents

Abstract

FIELD: information technology.

SUBSTANCE: invention relates to personal recognition by image. Method for automatic construction of 3D model of person's face comprises steps of: obtaining at least two 2D images of first view, determination of facial area on each of said 2D images of first view; construction of at least one intermediate 3D model of facial area on basis of at least one of said 2D image of first view; determination of view of person on at least one intermediate 3D model; change of view area of face on above intermediate 3D model; conversion of obtained at least one 3D model of facial area in at least one 2D image of second view; determination of facial area on at least one 2D image of second view; determination of coordinates of conjugate points areas of face on at least two 2D images, at least one of which is 2D image of second view; implementation of conversion of said coordinates of conjugate points areas of face in two-dimensional coordinates of conjugated points areas of face on at least one 2D image of first view; calculation of obtained coordinates of conjugate points position and matrix calibrating source on at least one 2D image of first view of facial area in three-dimensional coordinate system and reconstruction of two-dimensional coordinates of conjugate points areas of face on at least one 2D image of first view in three-dimensional model of face with help of said matrices.

EFFECT: high accuracy of 3D reconstruction of face.

9 cl, 16 dwg

Description

FIELD OF TECHNOLOGY

The invention relates to the field of pattern recognition, and in particular to the task of recognizing a person’s identity by image, using the method, device and computer-readable medium for automatically building a 3D model of a face from a series of 2D images.

BACKGROUND

A known method of constructing a three-dimensional model of the face using a stereo camera in order to automatically recognize a person’s face (RU 2012101848, from 01/19/2012), in which two or more stereo cameras are located at some distance between each other, while each of the stereo cameras reconstructs its three-dimensional image of a part of the face, then the individual parts are combined into one common three-dimensional image, which is used to recognize a person.

The disadvantages of this method is the technical complexity of its implementation, and the frames must be shot at the same time, as well as the inability to work with ready-made images or a sequence of frames, for example, a video film.

Known methods for constructing a three-dimensional model of the face based on only one or more 2B face images. In these methods, obtaining a two-dimensional image can be carried out with just one video camera, however, this increases the complexity and decreases the accuracy of building a three-dimensional model of the face, which can negatively affect the accuracy of further recognition or identification of a person's face.

Among the mentioned methods, there are known methods for constructing a three-dimensional shape of a person’s face from one or more 2B images using a three-dimensional deformable face model (US 20070080967 A1, US 7853085 B2, US 7643685 B2, US 7643683 B2, US 7643671 B2, ANIMETRICS INC.). In known methods, a face image is input to the device input, and the parameters and positions of the three-dimensional face model are selected so that the projection of the model most closely matches the face image in a series of 2B images.

The disadvantages of the known methods are:

- limited reconstruction accuracy, since the three-dimensional shape of the face is described by some model with a finite number of parameters. Therefore, the restored face shape only approximately coincides with the original face. A simple model can result in poor results when a person’s race or age changes,

- the accuracy of the construction depends on the accuracy of determining on the two-dimensional image the coordinates of some selected points in the corners of the lips, eyes and other basic elements of the face, called anthropometric points. At these points, the coincidence of the projection of the three-dimensional model with the original image of the face is controlled;

- the initial information for constructing a three-dimensional model is the two-dimensional coordinates of anthropometric points, the resulting three-dimensional model does not contain additional information except that already present in the two-dimensional image, therefore, it may not give advantages when recognizing using the three-dimensional face comparison algorithm;

- there is a difficulty in reconstructing facial expressions of the facial shape, since the model cannot be trained to repeat all existing facial expressions;

- the more accurately a three-dimensional model can approximate various forms of a person’s face, the longer the time it takes to select the model’s parameters to find the three-dimensional shape of a person’s face.

Another possible method is the well-known method of reconstructing a three-dimensional scene, according to a series of images from different angles of view in the case when the element of the reconstructed scene is a rigid object. Although only with a certain approximation can the human face shape be considered unchanged, this method is also of interest. In typical conditions, the camera is mounted motionless, a person’s face moves in space in the control zone. In the coordinate system associated with the face, the face is stationary, and the camera moves relative to the subject, so the task is reduced to reconstructing a three-dimensional scene with several cameras, the relative position of which is not known in advance (see D. Forsyth, J. Pons, “Computer Vision. A Modern Approach ", Translated from English, Moscow: Publishing House" William ", 2004). This problem is solved by searching for conjugate points on different projections of the image of the three-dimensional scene, which correspond to the same points of the three-dimensional scene. These points on different projections are connected by the Longet-Higgins relation (see D. Forsyth, J. Pons, “Computer Vision. A Modern Approach”, Translated from English, Moscow: Williams Publishing House, 2004, p. 312 ) through the fundamental matrix, which contains information about the relative position of the cameras. Except for some degenerate cases, which are generally not of interest to the invention in question, the fundamental matrix can be calculated from the two-dimensional coordinates of the conjugate points. Factorization of the fundamental matrix leads to calibration matrices for the internal and external parameters of the camera in different positions relative to the subject of the three-dimensional scene. Such a procedure is well known and implemented, for example, in the OpenCV computer vision library. After the calibration matrices are determined for each camera position, the reconstruction of all points in the scene is performed by standard algorithms based on the triangulation of rays emerging from conjugate points on different projections.

The specified method is a prototype of the present invention.

The disadvantages of this method are the high sensitivity of reconstruction errors to the accuracy of determination of two-dimensional conjugate points, which leads to a distortion of three-dimensional reconstruction, as well as the difficulty of using this method, as applied to reconstruction of the shape of a person’s face, since the following factors must be taken into account:

1) a three-dimensional scene should be segmented into a face and a background, since reconstruction is possible only for a rigid body, in which the relative dimensions do not change, and when a person moves in space, then the person’s face and background are not simultaneously a rigid body;

2) under typical operating conditions, the CCTV camera covers a large area, and in the image of the camera the person’s face occupies a small area, while the angles of view of the various conjugate points found on the face in different frames are quite small. For example, typical conditions are recognition of a face having characteristic dimensions between face points of the order of 10 cm, both in depth and in width, from a distance of several meters from the camera. In this case, small errors in determining the coordinates of the conjugate points on different frames lead to large errors in determining the positions of the camera relative to the face at different points in time, which introduces an error in the accuracy of three-dimensional reconstruction;

3) to use this method, facial images must be obtained at different frames in different positions: the greater the distance between different positions of the face, the greater the stereo base and the better the accuracy of triangulation (see D. Forsyth, J. Pons, “Computer Vision. Modern Approach ”, Translated from English, M .: Publishing House“ Williams ”, 2004), at the same time, the greater the movement of the face, the longer the time between frames and the more likely that the image of the face will be obtained with a significant change in angle , and changing the angle negatively affects the accuracy of calculating the coordinates of anthropometric points.

There is also a method based on the use of the Ransac method (https://ru.wikipedia.org/wiki/RANSAC), in which among all the found conjugate points iteratively discarded points that introduce the greatest error in the result. However, even the use of this method in the case of insufficiently high accuracy in determining most of the conjugate points is often insufficient to obtain a high-quality reconstructed three-dimensional image of the face shape.

SUMMARY OF THE INVENTION

The objective of the present invention is to provide a method for obtaining a high-quality three-dimensional model of a person’s face, in order to increase the accuracy of face recognition based on a comparison of three-dimensional face shapes, which allows to determine a large number of conjugate points in a two-dimensional image of a person captured using only one source, such as a video camera, camera, sequence of frames, etc.

The technical result is to increase the accuracy of three-dimensional face reconstruction.

The claimed technical result is achieved due to the method of automatically building a 3D model of a person’s face, containing the stages in which:

- receive from the data source information containing at least two 2D images of the first type containing a display of a person’s face;

- determine the area of the person’s face on each of the above-mentioned 2D images of the first type;

characterized in that

- constructing at least one intermediate 3D model of the face region based on at least one of the aforementioned 2D images of the first kind;

- determine the angle of a person’s face in at least one intermediate 3D model;

- change the angle of the face on the said intermediate 3D model;

- convert the obtained at least one 3D model of the face into at least one 2D image of the second type;

- determine the region of the person’s face in at least one 2D image of the second type;

- determine the coordinates of the conjugate points of the areas of the face in at least two 2D images, of which at least one is a 2D image of the second type;

- carry out the conversion of the mentioned coordinates of the conjugate points of the regions of the face in two-dimensional coordinates of the conjugate points of the regions of the face in at least one 2D image of the first type;

- from the obtained coordinates of the conjugate points, the position and matrices of the source calibration are calculated on at least one 2D image of the first kind of the face region in a three-dimensional coordinate system,

and reconstructing the two-dimensional coordinates of the conjugate points of the regions of the face in at least one 2D image of the first kind into a three-dimensional model of the face using the aforementioned matrices.

In a particular embodiment, the data source is a video camera, or a camera, or a stereo camera, or a frame sequence.

In a particular embodiment, the frame sequence is a video file.

In a particular embodiment, the intermediate 3D model is rotated to change the angle.

In a private embodiment of the invention, the coordinates of the anthropometric points of a person’s face are determined on at least one 2D image of the first kind, which are used to change the angle of the 3D model.

In a private embodiment of the invention, the intermediate three-dimensional model is a deformable three-dimensional model of the face, the shape and orientation of which in three-dimensional space is selected so that the anthropometric points on the three-dimensional model have projections onto various 2D images of the first kind, as close as possible to the coordinates of two-dimensional anthropometric points defined on these 2D images.

In a particular embodiment, the coordinates of two-dimensional anthropometric points are determined using a two-dimensional search algorithm.

Also, the present invention is implemented using a device and a machine-readable medium, which will be disclosed further in the present application materials.

BRIEF DESCRIPTION OF THE DRAWINGS

In FIG. 1 shows the general sequence of steps of the method for automatically constructing a 3D model of a face.

In FIG. 2-3 are images of a person’s face, shot with the same camera at different angles of view.

In FIG. 4-8 show an intermediate three-dimensional model of the face region.

In FIG. Figures 9-10 show face-aligned images of the face on which the conjugate points are calculated.

In FIG. Figures 11-12 show the conjugate points in the original images calculated by the coordinates of the conjugate points aligned with the angle of the face images.

In FIG. 13-15 presents a three-dimensional reconstruction of the visible part of the face.

In FIG. 16 shows a General view of a device that implements the claimed method.

DETAILED DESCRIPTION OF THE INVENTION

To eliminate the disadvantages of the method described in the prototype, the present invention proposes to use a preliminary conversion of images of a person’s face obtained from a series of 2D images, or from a sequence of frames, for example, video recording (video). In the present method, the method of changing the angle of the face of a person in the source images is used, for which purpose an intermediate three-dimensional model of the person’s face is built and its subsequent transformation to the required angle.

In this case, the search for conjugate points is carried out previously on the face-aligned face images in each of the frames of the 2D image. In this method, the reversibility and continuity of the transformation function for alignment of the angle is important. If you use a three-dimensional model of the face to rotate it, you can specify a one-to-one reversible correspondence between the source points on the face image on the source frames and the points on the aligned face image, provided that these points are visible on the images. Reversibility and continuity of alignment is not trivial in the general case for an arbitrary method of alignment to the desired angle.

The latter circumstance means that the conjugate points can be initially obtained not from the original images, but from the aligned ones, and after that they are recounted quite accurately to the original images. Indeed, the projective transformation and the transformation of rotation and spatial transfer of the coordinate system of a three-dimensional face model are continuous functions, therefore, the relation is true:

lim (ε ′) = 0 as ε → 0,

where ε is the error in determining the conjugate point in the aligned face image, ε ′ is the error in determining the conjugate point in the original face image.

In FIG. 1 shows the sequence of steps of the claimed method (100) for automatically constructing a 3D model of a face, which are performed in a preferred embodiment of the claimed invention.

At step (101), information is received from the data source containing at least two 2D images of the first kind containing a display of a person’s face. Original images - images of the first kind, are obtained from a data source, such as: a camera, video camera, stereo camera or an already prepared sequence of frames / images, for example, a video film. In FIG. 2-3, images obtained from a data source are represented, which are 2D images taken by a single camera at different angles of view.

Next, at step (102), a person’s face region is determined on each of the 2D images received from the source.

At step (103), at least one intermediate 3D model of the human face region is constructed, which is constructed from the original images, said intermediate model being constructed from at least one 2D image obtained from the source. In FIG. Figure 4-8 shows the process of constructing a 3D model of the face region from the original 2D images. The resulting model allows you to freely operate with its position in space, regardless of the angle of the original 2D images.

Then, at the stage (104), the angle of a person’s face is determined on the obtained intermediate 3D model, and one or many intermediate 3D models can be used. If the angle of the mentioned intermediate 3D model is very different from the angle of the face area of a person, then they are changed at step (105), for which they rotate the model of the face area to the required position. In FIG. 9-10, camera-aligned source images using a three-dimensional face model are shown, and conjugate points are determined on them.

At step (106), the obtained at least one 3D model of the face region is converted into at least one 2D image — a second view image that will have the necessary view of the display of the human face region. Next, at step (107), a person’s face region is determined in at least one 2D image of the second type, and at step (108), the coordinates of the conjugate points of the face regions are determined. For this, at least two 2D images are used, of which at least one is a 2D image of a second kind. The view must be changed so that a group of at least two 2D images with similar angles is obtained, from which you can determine the conjugate points with a high degree of accuracy, and this group can include only 2D images of the second type, or one 2D an image of a first view and one or more 2D images of a second view.

At step (109), said coordinates of the conjugate points of the face regions are converted into two-dimensional coordinates of the conjugate points of the face regions in at least one 2D image of the first kind, as shown in FIG. 11-12. Then, at step (110), from the obtained coordinates of the conjugate points, the position and matrices of the source calibration in the at least one 2D image of the first kind of the face region in the three-dimensional coordinate system are calculated. At step (111), the two-dimensional coordinates of the conjugate points of the face regions in the at least one 2D image of the first kind are reconstructed into a three-dimensional face model using the mentioned matrices. The result of constructing a model that has the correct proportions and depth is shown in FIG. 13-15.

In a particular embodiment of the claimed invention, in order to rotate a 3D model of a face region, the coordinates of the anthropometric points of a person’s face are determined on at least one 2D image of the first kind, which are used to perform the said rotation of the 2D image of the first kind. An intermediate three-dimensional model of a face can represent a deformable three-dimensional model, the shape and orientation of which in three-dimensional space is selected so that the anthropometric points on the three-dimensional model have projections onto various 2D images of the first kind, as close as possible to the coordinates of two-dimensional anthropometric points defined on these 2D images. The coordinates of two-dimensional anthropometric points are determined using a two-dimensional search algorithm, for example, described in: Wibowo, Moh Edi & Tjondronegoro, Dian W., “Gradual training of cascaded shape regression for facial landmark localization and pose estimation”, In International Conference on Multimedia and Expo 2013, July 15-19, 2013, San Jose, CA.

In FIG. 16 shows a general view of a device (200) for automatically building a 3D model of a person’s face. The device (200) can be a different type of software and hardware systems suitable for the tasks of automatically building a 3D model of a person’s face, containing processing tools in the form of one or more processors (210) and memory (220) for storing machine-readable instructions for automatically building 3D- models of a person’s face performed by at least one processor (210), and when executed, the above steps are taken to automatically build a 3D model of a person’s face disclosed in these materials Lah application.

As the device (200), for example, one or more stationary and / or mobile computers connected via a wired and / or wireless type data network, a handheld computer (PDA), tablet, smartphone, phablet, mainframe, etc., can be used . The memory (220) depending on the type of device (200) can be, but not limited to: random access memory (RAM; RAM), read-only memory (ROM; ROM), cache memory, flash memory, hard disk, solid state drive or any other suitable memory component.

The claimed invention is also embodied on a computer-readable storage medium that contains computer-readable instructions for automatically building a 3D model of a person’s face, which, when processed by one or more processors, perform the above steps of the method for automatically building a 3D model of a person’s face. The storage medium may be suitable for use in the aforementioned device for automatically constructing a 3D model of a person’s face, and may be, but not limited to: a USB flash drive, SD-, Micro SD-, Mini SD-, XD-, MMC memory card, CD-, DVD-, BlueRay-disk, HDD-, SSD - hard disk or any other suitable storage medium.

The information on the implementation of the claimed invention set forth in these materials of the application should not be interpreted as limiting other, private embodiments of the claimed invention that do not go beyond the disclosure of information of the application, and which should be obvious to a person skilled in the art having the usual qualifications, which calculated the claimed technical solution.

The proposed method is recommended for use at airports, train stations, and the subway for use in personal identification systems, statistical analysis, control of ticketless passage of turnstiles, in law enforcement agencies to search for offenders in the flow of people, to identify computer users, and also to identify individuals in enterprises with bandwidth mode.

Claims (9)

1. A method for automatically building a 3D model of a person’s face, comprising the steps of:
- receive from the data source information containing at least two 2D images of the first type containing a display of a person's face;
- determine the area of the person’s face on each of the above-mentioned 2D images of the first type;
characterized in that
- constructing at least one intermediate 3D model of the face region based on at least one of the aforementioned 2D images of the first kind;
- determine the angle of a person’s face in at least one intermediate 3D model;
- change the angle of the face on the said intermediate 3D model;
- convert the obtained at least one 3D model of the face into at least one 2D image of the second type;
- determine the area of the person’s face in at least one 2D image of the second type;
- determine the coordinates of the conjugate points of the areas of the face in at least two 2D images, of which at least one is a 2D image of the second type;
- carry out the conversion of the mentioned coordinates of the conjugate points of the regions of the face into two-dimensional coordinates of the conjugate points of the regions of the face in at least one 2D image of the first type;
- from the obtained coordinates of the conjugate points, the position and matrices of the source calibration in the at least one 2D image of the first kind of the face region in the three-dimensional coordinate system are calculated
and reconstructing the two-dimensional coordinates of the conjugate points of the regions of the face in at least one 2D image of the first kind into a three-dimensional model of the face using the aforementioned matrices. 2. The method according to p. 1, characterized in that the data source is a video camera, or a camera, or a stereo camera, or a sequence of frames. 3. The method according to p. 2, characterized in that the sequence of frames is a video file. 4. The method according to p. 1, characterized in that they perform the rotation of the intermediate 3D model to change the angle. 5. The method according to p. 4, characterized in that they determine the coordinates of the anthropometric points of the person’s face in at least one 2D image of the first kind, which is used to change the angle of the 3D model. 6. The method according to p. 5, characterized in that the intermediate three-dimensional model is a deformable three-dimensional model of the face, the shape and orientation of which in three-dimensional space is selected so that the anthropometric points on the three-dimensional model have projections onto various 2D images of the first kind, as close as possible to the coordinates of the two-dimensional anthropometric points defined on these 2D images. 7. The method according to p. 6, characterized in that the coordinates of two-dimensional anthropometric points are determined using a two-dimensional search algorithm. 8. A device for automatically building a 3D model of a person’s face, comprising:
- one or more processors;
- memory for storing machine-readable instructions;
wherein said instructions, when executed by at least one processor, implement the method according to any one of claims. 1-7. 9. A machine-readable storage medium containing machine-readable instructions for automatically building a 3D model of a person’s face, which, when processed by one or more processors, performs the method according to any one of claims. 1-7.

Images