KR101212110B1 - Method for 2d and 3d virtual face modeling - Google Patents

Abstract

PURPOSE: A method for 2D and 3D virtual face modeling is to give a 3D effect to human faces with different textures or structures while maintaining consistency. CONSTITUTION: A first and second face image is input(S101). The input image is arranged(S102). A local characteristic of a first face image is detected(S103). A patch library including all patches of a second face image is generated(S104). An output face image for the second face image is initialized(S105). The patch corresponding to each pixel point of the output face image is detected from the generated patch library. The detected patches are reunited(S106). The detected local characteristic is delivered to the output face image(S107). [Reference numerals] (AA) Start; (BB) End; (S101) A first and second face image is input; (S102) The input image is arranged; (S103) Local characteristic of a first face image is detected; (S104) Patch library including patch of second face image is generated; (S105) An output face image is initialized; (S106) The detected patches are reunited; (S107) The detected local characteristic is delivered to the output face image

Description

2D and 3D virtual face modeling method {METHOD FOR 2D AND 3D VIRTUAL FACE MODELING}

The present invention relates to a 2D and 3D virtual face modeling method, and more particularly, by comparing the two-dimensional texture of any face having three-dimensional information by using the two-dimensional texture of the input face and then passing the patch unit A 2D and 3D virtual face modeling method for reconstructing three-dimensional information.

In general, 2D face compositing creates a new face image by combining two or several portraits, and may create a virtual face that does not correspond to a specific person.

In particular, 2D facial synthesis can be applied in industries for hiding someone's identity or for pure entertainment purposes.

For 2D face compositing, “D. Bitouk, N. Kumar, S. Dhillon, PN Belhumeur, and SK Nayar, "Face swapping: automatically replacing faces in photographs," ACM Transactions on Graphics (SIGGRAPH'08), vol. 27, no. 3, 2008. ”, Bitouk et al. Introduced an automatic system that naturally synthesizes the faces of two different people.

In addition, "U. Mohammed, SJD Prince, and J. Kautz, "Visiolization: Generating novel facial images," ACM Transactions on Graphics (SIGGRAPH'09), vol. 28, no.3, pp.57: 1-57: 8, 2009. ”synthesizes texture-based image quilting models by generating approximate correlation images from a given set of face data.

However, these existing methods are limited only to human faces.

3D face modeling is to restore a person's face in three dimensions in a given image of one or more people. In particular, such techniques are required for special effects, for example in the cinematography industry.

In recent years, a lot of research has been actively conducted regarding 3D face modeling. These studies are described in D. Bradley, W. Heidrich, T. Poap, and A. Sheffer, "High resolution passive facial performance capture," ACM Transactions on Graphics (SIGGRAPH'10), vol. 29, no. 3, 2010. '' or `` T. Beeler, B. Bickel, P. Beardsley, R. Sumner, and M. Gross, "High-quality single-shot capture of facial geometry," ACM Transactions on Graphics (SIGGRAPH'10), vol. 29, no. 3, 2010. ”, the focus is on building hardware systems, so that 3D models obtained in general can be very precise.

However, these systems are usually expensive and require a variety of input images.

Alternatively, Sung Won Park, Jingu Heo and Marios savvides, "3d face reconstruction from a single 2d face image," in Computer Vision and Pattern Recongnition workshops, 2008., and Arthur Niswar, Ee Ping Ong, and Zhiyong Huang , "Pose-invariant 3d face reconstruction from a single image," in SIGGRAPH-Asia, 2010. "

However, while these results yield satisfactory results, most of these methods require interactions that must be entered by the user.

The technical problem of the present invention devised to solve the above problems is not limited to a human face, but to provide a three-dimensional effect to a person and a statue face having different textures and structures, taking into account the structure of the human face An object of the present invention is to provide a 2D and 3D virtual face modeling method capable of maintaining the consistency of color, texture, and structure of (or human face).

In addition, according to an embodiment of the present invention, it is an object of the present invention to provide a 2D and 3D virtual face modeling method that can automatically achieve the task of 2D face synthesis and 3D face modeling in a unified framework.

In addition, according to an embodiment of the present invention, 2D that can restore the three-dimensional information of the input face through the patch unit transfer after comparison with the two-dimensional texture of any face having three-dimensional information using the two-dimensional texture of the input face And a 3D virtual face modeling method.

However, the objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned can be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, a virtual face modeling method according to an embodiment of the present invention includes a first step of receiving a first face image and a second face image; A second step of aligning the received first and second face images to detect local features of the first face image; Generating a patch library including all patches of the second face image; A fourth step of initializing an output face image having the same size as the first face image; A fifth step of detecting and recombining a patch corresponding to each pixel point of the output face image in the patch library; And a sixth step of transferring the local feature detected in the second step to the output face image.

The virtual face modeling method according to an embodiment of the present invention, in the fifth step, the recombination of the patches included in the patch library is similar to the similarity constraint (similarity constraint) and the patch effect on the structure of the first face image ( According to the condition of the smoothness constraint (smoothness constraint) for the patch effect.

In addition, the virtual face modeling method according to an embodiment of the present invention, when the patch is detected in the fifth step, the similarity degree for each patch is characterized by the following equation.

[Mathematical Expression]

는 패치 P(x)에 대한 평균 변화율이며,

는 패치 P(l_x)에 대한 평균 변화율이다.)
Where G (P) is the rate of change of patch P,

Is the average rate of change for patch P (x),

Is the average rate of change for patch P (l _x ).)

In addition, in the virtual face modeling method according to an embodiment of the present invention, the first face image is a human face, and the second face image is a statue face.

According to another aspect of the present invention, there is provided a virtual face modeling method comprising: a first step of receiving a second face image including a first face image and 3D information; A second step of comparing texture information between the first face image and the second face image; Generating a depth patch library including a depth value for each pixel of the second face image; A fourth step of transferring texture information of the second face image to the first face image; And a fifth step of generating a virtual 3D face image of the first face image by combining the color information of the first face image and the depth patch included in the depth patch library.

In addition, the virtual face modeling method according to another embodiment of the present invention, characterized in that for transmitting the 3D information of the second face image to the first face image in the fourth step.

According to the virtual face modeling method of the present invention, it is possible to maintain the consistency of the color, texture, and structure of the input sculpture by considering the structure of the human face by providing a three-dimensional effect to the person and the statue face having different textures and structures.

In addition, according to the virtual face modeling method of the present invention, there is a feature that can automatically achieve the work of 2D face synthesis and 3D face modeling in a unified framework.

In addition, according to the virtual face modeling method of the present invention, by using the two-dimensional texture of the input face compared with the two-dimensional texture of any face having three-dimensional information, it is possible to restore the three-dimensional information of the input face through patch unit transfer It has an effect.

1 is a flowchart illustrating a 2D virtual face modeling method according to an embodiment of the present invention.
2 is a diagram illustrating a synthesis of a human face and a statue face according to an embodiment of the present invention.
3 is an exemplary diagram illustrating 2D virtual face modeling according to an embodiment of the present invention.
4 is a flowchart illustrating a 3D virtual face modeling method according to an embodiment of the present invention.
5 is an exemplary diagram illustrating 3D virtual face modeling according to an embodiment of the present invention.
6 is an exemplary view showing an example of a stereoscopic image for a 3D display according to an embodiment of the present invention.
7 shows subjective test results for a 3D display.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a detailed description of preferred embodiments of the present invention will be given with reference to the accompanying drawings. In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear.

1 is a flowchart illustrating a 2D virtual face modeling method according to an embodiment of the present invention.

As shown, the first face image and the second face image are received for the 2D virtual face modeling (S101). The input first facial image and the second facial image are aligned (S102), and local features of the first facial image are detected (S103).

Thereafter, a patch library including all patches of the second face image is generated (S104), and an output face image for the second face image is initialized to have the same size as the first face image (S105).

In the patch library generated in step S104, after detecting a patch corresponding to each pixel point of the output face image, the detected patches are recombined (S106).

Next, the local feature detected in step S103 is transferred to the output face image (S107) to generate a virtual face for the first face image and the second face image.

The 2D and 3D virtual face modeling of the present invention can be in two ways. First, the present invention provides a mathematical formatting of 2D face modeling for human-sculptural face synthesis, and second, generalizes this approach to 3D face modeling applications.

[2D face modeling]

The present invention synthesizes a virtual statue having a shape of an input statue and looking like an input human face, as shown in FIG.

FIG. 1 shows 2D face modeling, FIG. 1A shows the input (fragmentary face and human face), and FIG. 1B shows the output (virtual statue) synthesized by the 2D face modeling of the present invention.

The virtual statue must be (1) the color and texture consistency of the original statue (input statue) must be verified, and (2) the structure of the human face must not be modified, otherwise the human will not be able to recognize it.

To carry out this method, the present invention clarifies tasks such as the graphical labeling problem described briefly in the next step.

Align two faces with respect to facial features (eyes, nose, mouth, etc.)

Generating a patch library containing all patches of the input statue face

Initializing an empty output face having the same size as the human face; All facial areas are unknown at the initial stage.

Finding the best patch from the patch library for each pixel point of the output face to minimize the energy function

Each of these steps is described in more detail below.

[Face Alignment and Patch Library]

This step represents the two preliminary steps in the present invention: the face alignment and the creation of a patch library. An interesting observation is that although faces of people and statues have different textures and structures, they have similar local areas around anchor points such as eyes and mouth.

Thus, the present invention can automatically align the two faces from these anchor points. This makes it easier to compare patches in the next step.

Face alignment can be easily done using the Active Shape Model (ASM) algorithm. ASM is a common algorithm that detects local features that describe the shape of an object through learning.

In the present invention, S.Milborrow and F.Nicolls, "Locating facial features with an extended active shape model," European Conference on Computer Vision (ECCV'08), 2008, to detect all important facial features in both faces. Reference is made. The present invention then performed a warping algorithm to align these points to obtain an aligned person and statue face.

ASM also provides a mask that represents the boundary of the face in the image. Thus, detection of the facial area in the image is possible.

Subsequently, the area of the face for every pixel point is divided by a small patch (the patch size is set by the user and is generally 7x7). These patches are then stored in a patch library that forms a list of candidate patches for each pixel point in the output face.

[Energy Function]

In the present invention, patches included in a patch library (eg, from an original face) are recombined in a suitable form to create a virtual face image.

This form must (1) conform to the structure of a given human face (called a similarity constraint) and (2) minimize the "patch effect" (called a smoothness constraint). do.

로 표시된 그래프의 노드는, 출력 영상의 M 픽셀의 세트이다. 위치 x를 포함하는

의 각 픽셀 p에 대해, 라이브러리로부터 l_x로 표시된 최선의 라벨(패치)을 찾는 것이 본 발명의 목적이다.
To clarify this task as a graph labeling problem, each patch is associated with a label.

The node of the graph denoted by is a set of M pixels of the output video. Containing position x

For each pixel p of, it is an object of the present invention to find the best label (patch) represented by l _x from the library.

The present invention defines the energy associated with the sum of the two equations, as shown in the following equation (1).

)의 최선의 조합은 다음의 수학식(2)에서 나타낸 바와 같이, 에너지 E(L)를 최소화함으로써 얻어진다.
Where E _sim and E _smooth are the respective energies for similarity and smoothness constraints. α is the mutual weight that balances these two similarity constraint energies and smoothness constraint energies. The symbols <p, q> indicate that the pixels p and q are adjacent to each other. label(

The best combination of) is obtained by minimizing the energy E (L), as shown in equation (2) below.

The similarity formula is for finding patches with similar structure. Since the face of the person and the statue has a very different appearance, as shown in Fig. 1, the color is not an appropriate measurement, and thus referred to gradient information.

As shown in Equation (3), the similarity was confirmed as a normalized sum of squared difference (NSSD).

는 패치 P(x)에 대한 평균 변화율이며,

는 패치 P(l_x)에 대한 평균 변화율이다. E_data(l_x)는 「V.Kolmogorov, A. Criminisi, A. Blake, G. Cross, and C. Rother, "Bi-layer segmentation of binocular stereo video," IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR'05), vol. 2, pp.407-414, 2005.」에 나타낸 바와 같이, [0, 1]에서 실수 값을 리턴하는 정규화 식이다. 만약, G(P(x)) 및 G(P(l_x))가 동일하면, 리턴 값은 0이다.
Where G (P) is the rate of change of patch P,

Is the average rate of change for patch P (x),

Is the average rate of change for patch P (l _x ). E _data (l _x ) is described in V. Kolmogorov, A. Criminisi, A. Blake, G. Cross, and C. Rother, “Bi-layer segmentation of binocular stereo video,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition. (CVPR'05), vol. 2, pp.407-414, 2005. ", it is a normalization expression which returns a real value in [0, 1]. If G (P (x)) and G (P (l _x )) are equal, the return value is zero.

The smoothness energy encodes the overlapping coherence between two adjacent patches P (l _x ) and P (l _y ). This energy equation is defined as NSSD in their overlapping region. This equation can also be considered an extension of Li-Yi Wei and Marc Levoy, "Fast texture synthesis using tree-structured vector quntization," is SIGGRAPH '00, 2000, pp. 479-488.

In order to find the labeling that minimizes the MRF energy of Equation (2), in the present invention, "PFFelzenszwalb and DPHuttenlocher," Efficient belief propagation for early vision, "International Journal of Computer Vision (IJCV'06), vol. 70,` no .1, pp. 41-45, 2006. '' and J.Sun, L.Yuan, J.Jia, and H.-Y.Shum, "Image completion with structure propagation," in SIGGRAPH'05, 2005, pp .861-868. ”Apply BP (Belief Propagation).

BP is a popular iterative algorithm that treats the energy cost for each pixel point as potential and increases it through the graph to find the best label for each point.

By using the anchor points detected in the face alignment and patch library, the present invention applies a warping step to deliver the pixels of the output image to the statue face. Thus, it is possible to drastically reduce the number of candidate patches (eg, search spaces) and to use some deductive geometric information about the face.

[Results and Analysis]

The present invention applies a 2D face modeling framework to various photographs. Representative results are shown in FIG. 2. In addition, in the present invention for comparison, "Patrick Perez, Michel Gangnet, and Andrew Blake," Poisson image editing, "ACM Transactions on Graphics (SIGGRAPH'03), vol. 22, no.3, pp. 313-318, 2003. ”was applied to the Poisson Image Editing method.

The first column of FIG. 2 represents an input human face, the second column represents an input statue face, the third column is an output face using the Poisson Image Editing method, and the fourth column is a 2D face modeling of the present invention. Output face according to the method.

Although good results can be considered Poission outputs, it is important that they contain inappropriate colors (eg, colors not present on the input fragment) and therefore cannot be considered as virtual statues. In contrast, the method according to the invention makes it possible to maintain the consistency of the color, texture and structure of the input sculpture, and in particular, to restore small features such as ball wrinkles, hair lines and glasses.

4 is a flowchart illustrating a 3D virtual face modeling method according to an embodiment of the present invention. As shown in the figure, first, a first face image and a second face image including 3D information are input for virtual face modeling (S201). Subsequently, texture information between the input first face image and the second face image is compared (S202), and a depth patch library including a depth value for each pixel of the second face image is generated (S203). ).

Next, texture information of the second face image is transferred to the first face image (S203), and 3D information included in the second face image is also transferred.

Thereafter, the virtual 3D face image of the first face image is generated by combining the color information of the first face image and the depth patch included in the depth patch library (S205).

[3D face modeling]

As described above, the 2D face modeling of the present invention has shown how to deliver 2D color patches to create a virtual statue that looks like an input human face and verifies the texture / color consistency of the input statue face.

Also, knowing the 3D information of the face of the input statue, one might ask, "How does the output face look like 3D by directly conveying the depth value associated with the selected patch?" The motivation for this extension is simple.

If an input statue face is associated with a 3D model (obtained by some method), each pixel of the statue face has a depth value, thus making it possible to create a “depth patch library”. By combining the color information and the depth patch, the resulting virtual 3D face can be output. This approach allows us to explore alternatives to 3D face modeling problems. In the following, the generalization of 2D patch delivery based on the framework of the present invention towards 3D face modeling is shown.

[Energy Formula Generalization]

In the present invention, an energy equation generalization for processing depth information is shown. It does not change the similarity energy because it targets the output face to depict the same structure as the input human face. For smoothness energy, add one or more equations that encode depth consistency within the local region of the output face. This consistency is defined as the NSSD of the depth value outside of the overlapping area.

This additional equation allows to compensate for the smoothness around the neighbor's depth values in addition to the color values to make a satisfactory 3D face.

By changing the depth information, it is possible to output a virtual 3D face in association with the depth map, synthesize a virtual viewpoint as shown in FIG. 3, and also as shown in FIG. 4, a stereoscopic screen for 3D display. Can be generated.

[Comparative analysis with other input 3D models]

In the present invention, the influence of the input 3D model on the output 3D face is considered. As mentioned above, focusing on face modeling from statues and human faces, in particular two input human faces are considered to explore the maximum possibilities.

For experiments on input human 3D faces, see J.Sun, L.Yuan, J.Jia, and H.-Y.Shum, "Image completion with structure propagation," in SIGGRAPH'05, 2005, pp.861- 868. ”A dataset consisting of 3D models of various people was used at some orientations, and 3D faces generated from statues and human faces were compared.

Subjective tests were performed to vote for the most satisfying 3D face made using another input 3D face (called the reference face). To this end, a virtual stereoscopic screen for displaying a 3D face was generated using the Nvidia 3D Vision Kit. Through this, 3D information can be obtained. The virtual left screen of the stereoscopic image pair can be continuously generated, and the right screen is simply an original human image. In addition to the stereoscopic screen, the virtual left screen is shown in FIG. The first column of FIG. 4 is an original face, the second column is a virtual left screen automatically generated by the present invention, and the third column represents a stereoscopic image generated for the 3D display.

Subjective tests included four types of reference faces: a) statue face, b) random selection of human faces from the 3D face database, c) manual selection of human faces with appearances similar to target faces, and d) automatic selection of human faces (PCA faces). By recognition). The test results are shown in FIG. From the results, it is possible to obtain satisfactory results in which the 3D face modeled from the reference face whose appearance is similar to the target face is seen on the 3D display.

In addition, similar appearances of 2D images may involve similar 3D structures. With these reference faces, 3D displays capable of accurately conveying 3D information in accordance with the present invention have proven satisfactory by the testers.

[conclusion]

The present invention proposes a unified framework based on patch delivery for two tasks of 2D and 3D face modeling. In the case of 2D, structural information of color and constraints of smoothness are considered. A virtual statue face was created by combining human and statue faces. The formalization was generalized by adding depth smoothness conditions. By delivering depth as well as appearance, a 3D face model could be obtained from a single screen.

In addition, the present invention may generate a stereoscopic image screen for a 3D display application. Experimental results and subjective tests confirmed the inventive solution.

Claims (6)

In the virtual face modeling method,
A first step of receiving a first face image and a second face image;
A second step of aligning the received first and second face images and detecting a local feature of the first face image;
Generating a patch library including all patches of the second face image;
A fourth step of initializing an output face image for the second face image having the same size as the first face image;
A fifth step of detecting and recombining a patch corresponding to each pixel point of the output face image in the patch library; And
And a sixth step of transferring the local feature detected in the second step to the output face image.
The method of claim 1,
In the fifth step, the recombination of the patches included in the patch library depends on conditions of similarity constraints on the structure of the first face image and smoothness constraints on patch effects. Virtual face modeling method, characterized in that.
The method of claim 1,
When the patch is detected in the fifth step, the similarity degree for each patch is defined by the following equation.
[Mathematical Expression]

Where G (P) is the rate of change of patch P,

Is the average rate of change for patch P (x),

Is the average rate of change for patch P (l _x ).
E _data (l _x ) is described in V. Kolmogorov, A. Criminisi, A. Blake, G. Cross, and C. Rother, “Bi-layer segmentation of binocular stereo video,” IEEE Computer Society Conference on Computer Vision and Pattern Recognition. (CVPR'05), vol. 2, pp.407-414, 2005. ", it is a normalization expression which returns a real value in [0, 1]. If G (P (x)) and G (P (l _x )) are equal, the return value is zero.
The method of claim 1,
And wherein the first face image is a human face and the second face image is a statue face.
In the virtual face modeling method,
A first step of receiving a second face image including a first face image and 3D information;
A second step of comparing texture information between the first face image and the second face image;
Generating a depth patch library including a depth value for each pixel of the second face image;
A fourth step of transferring texture information of the second face image to the first face image; And
And a fifth step of generating a virtual 3D face image of the first face image by combining the color information of the first face image and the depth patch included in the depth patch library. Way.
The method of claim 5, wherein
And transmitting the 3D information of the second face image to the first face image in the fourth step.

Images