KR20170019779A - Method and Apparatus for detection of 3D Face Model Using Portable Camera - Google Patents

Abstract

The present invention provides a 3D face model obtaining method and device which use a portable camera. The 3D face model obtaining device using a portable camera comprises: a face image capturing unit for capturing an image of users face; a face image processing unit for extracting a series of face images from the captured user face image; a 3D face mesh generating unit for analyzing the relationships between the face images and generating a 3D user face mesh; a standard face model matching unit for matching a 3D face standard model to the 3D face mesh made by the mesh generating unit, and ultimately making a high-quality 3D user face model; a face skin texture composing unit for generating a face skin texture from the captured image; and a realistic face rendering unit for using the high-quality 3D face model and skin texture of the user to carry out realistic rendering.

Description

TECHNICAL FIELD The present invention relates to a method and apparatus for acquiring a three-dimensional face model using a portable camera,

The present invention relates to a method and apparatus for generating a three-dimensional face model using a camera, and more particularly to a method for generating a high-quality photorealistic three-dimensional face model from a series of successive images photographed from a camera, ≪ / RTI >

A method of generating a three-dimensional face model using a camera includes a method using a plurality of synchronized cameras and a method using a single camera. When using multiple synchronized cameras, several high-resolution cameras are placed in fixed positions, where the camera can be configured as a depth camera as well as a color camera. Thus, a plurality of calibrated images are acquired at a time to generate a three-dimensional face model.

This approach can capture facial photographs in a single moment and use the already synchronized camera information to create a three-dimensional facial model in a short time. However, in general, expensive equipment is required, and since it takes up a considerable amount of space to place the camera in a fixed position, the installation cost is high, and there is a limitation in cost and space for general users to use.

To create a three-dimensional face model using a single camera, you have to face multiple times to get more than one image. First, the feature points (eye, nose, and mouth points) of the face are found, and the correlation of each feature point is analyzed to obtain coordinates in three-dimensional space. Dimensional model of the user is obtained by transforming corresponding points of the three-dimensional model already created. However, since the number of generated feature points is several tens, there is a limitation in accurately restoring the various facial shapes of people to a level that allows virtual molding in three dimensions.

SUMMARY OF THE INVENTION It is an object of the present invention to overcome the above-mentioned problems by providing a method and apparatus for generating high-quality realistic three-dimensional face models from a series of successive images taken from a camera.

The technical objects of the present invention are not limited to the technical matters mentioned above, and other technical subjects not mentioned can be clearly understood by those skilled in the art from the following description.

In one aspect, a three-dimensional face model acquisition apparatus using a portable camera proposed in the present invention includes a face image photographing unit for photographing a user's face according to a photographing guide UI using a portable camera and storing a moving image and an image sequence, A face image processing unit for verifying whether a three-dimensional face model can be generated from a photographed image by using sensors mounted on a camera and extracting a series of face images from the photographed image; A three-dimensional face mesh generation unit for generating a three-dimensional face mesh, extracting a mask area including the eyes, nose, mouth, jaw and forehead of the user's face in the generated three-dimensional face mesh, A three-dimensional face model is matched to the three-dimensional face mesh to obtain an optimal three-dimensional face model A facial skin texture synthesizer for synthesizing a facial skin texture from the photographed image using the sensors mounted on the portable camera, and a facial skin texture synthesizer using the optimal three-dimensional facial model and the facial skin texture And a realistic face rendering unit that performs realistic rendering.

The 3D face image processing unit may select an image at a predetermined frame interval from the set of the face images.

The three-dimensional face mesh generation unit extracts Key points from the series of face images optimized through the face image processing unit, analyzes the spatial relationship between the key points to create a three-dimensional point cloud, The three-dimensional face mesh can be generated by connecting points on the space of the point cloud.

The standard face model matching unit may generate a three-dimensional face model that matches the three-dimensional face mesh using a predetermined three-dimensional face standard model and is capable of final face modification.

The facial skin texture synthesis unit may convert the optimal 3D facial model into UV space, and synthesize user images to generate facial skin texture maps for rendering.

The realistic face rendering unit may perform realistic face rendering using a technique of mapping a face skin texture to the optimal three-dimensional face model and simulating a skin effect.

According to another aspect of the present invention, there is provided a method for acquiring a three-dimensional face model using a portable camera, comprising the steps of: capturing a user's face in accordance with a photographing guide UI using a portable camera to store a moving image and an image sequence; Dimensional face model from the photographed image using sensors mounted on the camera, selecting an image at a predetermined frame interval from the photographed image, extracting key points from the photographed image, Generating three-dimensional point clouds by analyzing a spatial relationship between the Key points, generating a mesh by connecting points of the three-dimensional point cloud, generating a mesh of eyes, nose, mouth, The mask area containing the jaw and forehead is extracted and the rest is removed. Modeling an optimal three-dimensional face model by matching a predetermined three-dimensional face standard model with the three-dimensional face mesh, and extracting a face skin texture map for mapping the optimal three-dimensional face model from the photographed image And synthesizing.

In the step of selecting an image at a predetermined frame interval from the photographed image, a minimum number of images can be selected from the photographed image at regular intervals.

The step of extracting the key points from the photographed image and analyzing the spatial relationship between the key points to extract the key points from the photographed image, The key points are connected, the Sfm algorithm using factorization is applied, the structure of the key points between the images is analyzed, the camera attributes in the captured images are obtained, You can apply triangulation techniques to points and generate coordinate values in three-dimensional space for each key point.

The step of connecting the points of the three-dimensional point clouds to generate a mesh may include generating an additional point cloud using an optical flow analysis technique and applying a triangulation technique to the generated point cloud to generate a three- .

Wherein the step of synthesizing the facial skin texture map for mapping the optimal three-dimensional facial model from the photographed image maps the generated optimal three-dimensional facial model to the UV space, and the user corresponding to the coordinates of the UV space And generates an additional face skin texture map by using the generated face skin texture map and the generated optimal three-dimensional face model as a normal vector and a light source position .

According to the method and apparatus for acquiring three-dimensional face models using the portable camera described above, it is possible to provide a three-dimensional face model acquiring method and apparatus using a portable camera without using a special purpose scanner device or a separate expensive camera, Using the same portable camera, an individual can easily model a user's three-dimensional face at low cost.

According to the embodiment of the present invention, instead of generating a low-quality three-dimensional face used for a character of a game or an avatar on a cyber by utilizing only a limited number of feature points in a face of a photograph, a high-quality realistic three- can do.

In addition, according to the embodiment of the present invention, a high-quality three-dimensional user face is secured at a low cost, thereby realizing a virtual virtual molding service more easily. In addition, it is possible to provide a variety of virtual simulation services related to the face such as virtual wearer services such as glasses, earrings, virtual hair styling service, and 3D skin makeup service by utilizing the user's three-dimensional face model, .

1 is a block diagram of a 3D face model acquisition apparatus using a portable camera according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating a process of acquiring a three-dimensional face model using a portable camera according to an exemplary embodiment of the present invention.
3 is a diagram showing intermediate results of a 3D face mesh restored from a series of face images obtained from a portable camera according to an embodiment of the present invention.
4 is a view showing a screen of a UI guide provided for capturing a face image of a user in a portable camera according to an embodiment of the present invention.
FIG. 5 is a view showing a result of extracting only a face mask portion including only eyes, nose, mouth, jaw and forehead in a 3D face mesh restored from a series of face images obtained from a portable camera according to an embodiment of the present invention.
FIG. 6 is a diagram illustrating a final three-dimensional face model generated by matching a three-dimensional face mask mesh of a user and a three-dimensional face standard model according to an embodiment of the present invention.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail.

It is to be understood, however, that the invention is not to be limited to the specific embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the relevant art and are to be interpreted in an ideal or overly formal sense unless explicitly defined in the present application Do not.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the present invention, the same reference numerals are used for the same constituent elements in the drawings and redundant explanations for the same constituent elements are omitted.

1 is a block diagram of an apparatus for acquiring a three-dimensional face model using a portable camera according to an embodiment of the present invention.

1, an apparatus for acquiring a three-dimensional face model using a portable camera according to an embodiment of the present invention includes a face image capturing unit 100, a face image processing unit 200, a three-dimensional face mesh generating unit 300, A facial model matching unit 400, a facial skin texture synthesis unit 500, and a realistic facial rendering unit 600.

The face image capturing unit 100 captures a user's face using a portable camera. At this time, a UI for a photographing guide is provided to obtain images suitable for subsequent processing. Then, the face image processing unit 200 verifies whether the moving image captured by the user can be successfully generated using the three-dimensional face model, using the sensor information of the smartphone, and transmits the verified moving image to optimize the subsequent processing performance And selects images at regular intervals in the set of consecutive images.

Then, the three-dimensional face mesh generation unit 300 extracts Key points from the optimized series of images, analyzes the relationship between them, creates a three-dimensional point cloud, connects the points on the three- Create a mesh. In this case, the 3D face mask includes a function of extracting only the region of interest of the face in the reconstructed 3D face mesh including a part of the background to generate a 3D face mask.

Thereafter, the standard face model matching unit 400 matches the 3-dimensional face mesh using the standard model, which is a high-quality standard model in which the control points are defined in advance so as to be deformable to the face, Dimensional face model. Then, the facial skin texture synthesis unit 500 converts the 3D facial model into UV space, and synthesizes user images to generate texture maps for facial skin rendering. Then, the realistic face rendering unit 600 performs a realistic face rendering using a technique of mapping a face skin texture to a three-dimensional face model and simulating a skin effect.

2 is a view illustrating a method of acquiring a 3D face model using a portable camera according to an embodiment of the present invention

A method for acquiring a three-dimensional face model using a portable camera proposed in the present invention includes the steps of photographing a user's face according to a photographing guide UI using a portable camera and storing a moving image and an image sequence, The method of claim 1, further comprising: verifying whether a three-dimensional face model can be generated from the photographed image; selecting an image at a predetermined frame interval from the photographed image; extracting key points from the photographed image; Analyzing the relationship to generate three-dimensional point clouds; connecting the points of the three-dimensional point cloud to generate a mesh; generating a mask including a face's eyes, nose, mouth, Extracting the area and removing the rest, a predetermined three-dimensional face The method of claim 1, further comprising: modeling an optimal three-dimensional face model by matching a standard model with the three-dimensional face mesh; and analyzing the facial skin to map the optimal three-dimensional face model from the photographed images using the sensors mounted on the portable camera And compositing the texture map.

2, the user's face according to an embodiment of the present invention does not move, but moves the camera from the left side of the face to the right side of the face, or moves the camera from the right side of the face to the left side of the face , The face is rotated from the left side to the right side while the camera is fixed, or from the right side to the left side (S100). At this time, the photographed image can be compressed or stored as a moving image or can be stored as a continuous image form.

Thereafter, when the brightness of the photographed image is not equal to or greater than a predetermined LUX value by using the illuminance sensor information of the smartphone with respect to the photographed moving image, or by using the acceleration sensor of the smartphone, In the case where the change amount of the speed changes abruptly from + to - or from - to + by a constant change amount, the user is guided to retake (S101, S102), and the successfully photographed image is transmitted (S103) a process of selecting a minimum number of images at regular intervals in a moving image or a continuous image set.

Then, the key points are extracted from the facial images transmitted in the image processing, and the corresponding key points are found in the adjacent images to connect the key points (S104). In this case, the algorithm that extracts the key points between the images and links the key points between the images can refer to the public technique [1].

[1] Stefan Leutenegger, Margarita Chli and Roland Siegwart, "BRISK: Binary Robust Invariant Scalable Keypoints", Proceedings of the IEEE International Conference on Computer Vision, 2011.

Then, by applying the Sfm algorithm using factorization using mapped key points, the direction of movement of the key points between each images, the moving distance. Speed, etc., which can be used to obtain the camera properties, such as focal length, angle, etc., of the captured images. By applying this information to the key points, (S105). ≪ / RTI > This process creates a three-dimensional face sparse point cloud.

Since a sparse point cloud generated in the above step is difficult to generate a 3D mesh, an additional point cloud is generated by using an optical flow technique to obtain a denser point cloud, (S106).

Thereafter, the 3D mesh is generated by applying a triangulation technique to the Dense point cloud generated in the above step (S107).

The generated three-dimensional face mesh may include a background portion other than the face, and it is difficult to completely restore the part such as the hair or the ear due to the photographing conditions. Therefore, these portions are removed and the face, eyes, nose, A function of extracting only the mask portion including the region is required (S108, S109, S110). First, coordinates are obtained on the three-dimensional nose region using the two-dimensional coordinate information obtained by finding the nose region in the input images and the camera position information on each image on the three-dimensional space (S108).

Then, starting from the nose region, the face region is expanded based on the skin color information using the human face color distribution in the CIH Lab color space (S109). In this way, a 3D face mask area excluding the head hair, hair, and ear is finally extracted (S110).

The generated three-dimensional facial mask mesh does not include the incomplete three-dimensional facial parts such as the head face hair, the hair, and the ears, and the irregular mesh may also be generated by the noise on the face surface. Further, the structure of the mesh is also irregular, so that it may be difficult to deform. Therefore, in the present invention, a high-quality three-dimensional standard face model in which important feature points of a face are set in advance is previously stored and matched to the generated mesh to generate a final three-dimensional user face model.

For this purpose, landmarks are set in the three-dimensional face mask mesh generated in the above step using the landmarks of the three-dimensional standard model, and two models are sorted using ICP (Iterative Closest Points) (S111). Then, the landmarks are firstly repositioned into the landmarks of the three-dimensional face mesh in the three-dimensional face standard model, and the remaining vertices of the three-dimensional face standard model are transformed into the three- And matched to the vertices (S112). A concrete algorithm for such matching can be implemented by referring to [2].

[2] David C. Schneider, and Peter Eisert, "Algorithms For Automatic And Robust Registration Of 3D Head Scans." JVRB - Journal of Virtual Reality and Broadcasting.

Thereafter, the generated three-dimensional face model is mapped to the UV space (S113), the pixel values of the user's images corresponding to the respective UV coordinates are found, and a user's face skin color texture map is generated (S114). In order to perform realistic rendering, an additional texture map such as a normal map, a specular map, and the like can be generated using the generated color map and the generated three-dimensional face model using a normal vector and a light source position (S115). Sub-surface scattering is then simulated using the completed 3D face model and automatically generated skin texture maps (S116).

3 is a diagram showing intermediate results of a 3D face mesh restored from a series of face images obtained from a portable camera according to an embodiment of the present invention.

Referring to FIG. 3, reference numeral 300 according to an embodiment of the present invention is a continuous image used for obtaining a three-dimensional model in an image taken by a user, and reference numeral 301 denotes a sparse cloud generated in step S105 of FIG. Point and the camera position information at this time. Reference numeral 302 denotes a dense cloud point obtained by performing step S106 in FIG. 2, and reference numeral 303 denotes a three-dimensional face mesh obtained by performing step S107 in FIG.

FIG. 4 is a diagram showing a UI for guiding photographing when photographing a user's face with a smartphone according to an embodiment of the present invention.

Referring to FIG. 4, reference numeral 400 in FIG. 4 denotes a display when the user's face is maximally centered and guides the user to include the entire face, In order to control the case where the 3D face can not be restored normally because the progress speed is too fast when proceeding to the right side, a mark passing the face of the person at a constant speed is provided to guide the photographing of the face image successfully.

FIG. 5 is a diagram showing a first-reconstructed 3D face mesh according to an embodiment of the present invention and a mesh obtained by extracting only a face mask of a user in the entire mesh.

Referring to FIG. 5, 500 according to an embodiment of the present invention is a three-dimensional face mesh generated through S106 of FIG. The generated three-dimensional face mesh includes the background around the face, and includes incorrectly restored ear, head hair, and hair. A face mask area including only eyes, nose, mouth, and chin is extracted (501) through the steps S108, S109, and S110 of FIG. 2 to obtain a three-dimensional face mask model.

FIG. 6 is a diagram illustrating a final three-dimensional face model generated by matching a three-dimensional face standard model according to an embodiment of the present invention and a three-dimensional face mask of a user.

Referring to FIG. 6, reference numeral 601 according to an embodiment of the present invention is a three-dimensional face standard model prepared in advance. As shown in (600), a third face It has a landmark that is used to match the mask mesh, and is structured so that each area of the face can be divided and controlled so that deformation can be done well. Therefore, the final three-dimensional face model 602 matched through (S111, S112) in FIG. 2 also has the same mesh structure as the three-dimensional face standard model, so that deformation is facilitated.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, And various modifications may be made by those skilled in the art without departing from the scope of the present invention.

100: Face image capturing unit
200: facial image processing unit
300: 3D face mesh generation unit
400: Standard face model matching unit
500: facial skin texture synthesis unit
600: Realistic facial rendering unit

Claims (10)

A three-dimensional face model acquisition apparatus using a portable camera,
A face image photographing unit for photographing a user's face according to a photographing guide UI using a portable camera and storing a moving image and an image sequence,
A face image processing unit for verifying whether a three-dimensional face model can be generated from the photographed image using the sensors mounted on the portable camera, and extracting a series of face images from the photographed image;
Extracting a mask area including the eye, nose, mouth, jaw and forehead of the user's face from the generated three-dimensional face mesh, Dimensional face mesh generation unit,
A standard face model matching unit for generating an optimal three-dimensional face model by matching a predetermined three-dimensional face standard model with the three-dimensional face mesh,
A facial skin texture synthesizer for synthesizing a facial skin texture from a photographed image using sensors mounted on the portable camera,
And a realistic face rendering unit that performs realistic rendering using the optimal three-dimensional face model and the face skin texture,
Dimensional face model acquisition device using the portable camera.
The method according to claim 1,
Wherein the 3D face image processing unit comprises:
And selecting an image at a predetermined frame interval from the set of the set of face images. The method according to claim 1,
Wherein the standard face model matching unit comprises:
Wherein the three-dimensional facial model is matched with the three-dimensional facial mesh using a predetermined three-dimensional facial standard model to generate a three-dimensional facial model capable of final facial deformation. The method according to claim 1,
The facial skin texture synthesizer may include:
Wherein the optimal three-dimensional face model is converted into UV space, and the user's images are synthesized to generate facial skin texture maps for rendering. The method according to claim 1,
The realistic face rendering unit,
Wherein the realistic face rendering is performed using a technique of mapping a face skin texture to the optimal three-dimensional face model and simulating a skin effect. A three-dimensional face model acquisition method using a three-dimensional face model acquisition apparatus using a portable camera,
Storing a moving image and an image sequence by photographing a user's face according to a photographing guide UI using a portable camera;
Verifying whether a three-dimensional face model can be generated from a photographed image using sensors mounted on the portable camera;
Selecting an image at a predetermined frame interval from the photographed image;
Extracting Key points from the photographed image and analyzing a spatial relationship between the Key points to generate three-dimensional point clouds;
Connecting each point of the three-dimensional point clouds to generate a mesh;
Extracting a mask area including eyes, nose, mouth, jaw and forehead of the face from the generated mesh and removing the rest;
Modeling an optimal three-dimensional face model by matching a predetermined three-dimensional face standard model with the three-dimensional face mesh; And
Synthesizing a facial skin texture map for mapping the optimal three-dimensional facial model from a photographed image using sensors mounted on the portable camera
A method for acquiring a three-dimensional face model using a portable camera. The method according to claim 6,
Wherein the step of selecting an image at a predetermined frame interval from the photographed image comprises:
And selecting a minimum number of images at a predetermined interval from the photographed image. The method according to claim 6,
The step of modeling an optimal three-dimensional face model by matching the predetermined three-dimensional face standard model with a face mask model extracted as a mask region including an eye, nose, mouth, jaw and forehead of the face,
Characterized in that the landmarks are set in a face mask model extracted as a mask area including the eyes, nose, mouth, jaw and forehead of the face using landmarks of a predetermined three-dimensional face standard model A Method for Acquiring 3D Face Model Using. 9. The method of claim 8,
Aligning the two models using an ICP (Iterative Closest Points) algorithm based on the landmarks, changing the landmarks to the positions of the landmarks of the three-dimensional face mesh in the predetermined three-dimensional face standard model, Wherein the remaining vertices of the 3D face model are matched to the vertices of the 3D face mesh through nonlinear deformation. The method according to claim 6,
The step of synthesizing a facial skin texture map for mapping the optimal three-dimensional facial model from a photographed image using sensors mounted on the portable camera,
Maps the generated optimal three-dimensional face model to the UV space, generates a face skin color texture map by searching for pixel values of the user's images corresponding to the coordinates of the UV space, Dimensional face model, and an additional face skin texture map is generated using the normal vector and the light source position of the optimized three-dimensional face model.

Images