KR20210147626A - Apparatus and method for synthesizing 3d face image using competitive learning - Google Patents

Abstract

The present invention provides a three-dimensional (3D) face image synthesizing apparatus using competitive learning and a method thereof. According to the present invention, the 3D face image synthesizing apparatus comprises: a face model alignment unit receiving a two-dimensional (2D) face image to be synthesized into a 3D face image, extracting matching parameters from a pattern of the 2D face image according to a pre-learned pattern estimation method, and aligning a predetermined 3D face model to the 2D face image to acquire a UV map; and a UV map completion unit estimating and filling a hole area generated in the UV map by an unscanned area of the 2D face image according to the pre-learned pattern estimation method on the basis of the horizontal symmetry of the face to acquire a compensated UV map, and extracting fine features of the compensated UV map and restoring the UV map reflecting the fine features to acquire a completed UV map. Since the face model alignment unit and the UV map completion unit are trained in a competitive learning method to reduce generation loss and discrimination loss, even if a 2D face image captured for the entire face is not provided, the apparatus is learned on the basis of the energy of the completed UV map to finely and accurately restore even an uncaptured occlusion area.

Description

Apparatus and method for synthesizing a three-dimensional face image using competitive learning

The present invention relates to an apparatus and method for synthesizing a three-dimensional face image, and to an apparatus and method for synthesizing a three-dimensional face image using competitive learning between face alignment and color completion.

Face models are used in a variety of applications such as face compositing, 3D animation, and image editing. Building a human-like 3D face is therefore an important issue in computer vision and computer graphics.

3D Morphable Model (3DMM), a statistical model of 3D faces and textures, is the most widely used model for obtaining 3D face images from objects of 2D face images. With the advent of 3D sensors such as stereo cameras, it is possible to collect accurate and large number of 3D face data sets, so 3DMM has a powerful ability to express various natural facial shapes.

However, since the 2D face image is captured in a limited environment, it is difficult to accurately align the 3D face model to the image. In addition, since the 3D face model expresses a face in a mesh type, there is a limitation in that it is difficult to express an area that is not captured in the 2D face image even if it is aligned with the 3DMM face model image. In addition, the expression method of 3DMM is insufficient to accurately express facial textures such as beards, wrinkles, and dots.

Therefore, there is a need for a 3D face image synthesis technique that can accurately align a 3D face model to a face image, can restore natural areas that are not captured in a face image, and can express detailed textures.

Korean Patent Publication No. 10-2018-0082170 (published on July 18, 2018)

An object of the present invention is to obtain a UV map by accurately aligning a face model to a face image, and to provide an apparatus and method for synthesizing a three-dimensional face image that can be used to accurately express even areas not captured in a 2D face image using the obtained UV map. is to provide

Another object of the present invention is that it is not necessary to set conditions such as landmarks or feature points in the 2D face image to align the 3D face model to the 2D face image, and the 2D face images captured for all face regions are not provided for learning. An object of the present invention is to provide an apparatus and method for synthesizing a three-dimensional face image that can be easily learned and can restore facial texture.

In order to achieve the above object, a three-dimensional face image synthesizing apparatus according to an embodiment of the present invention receives a 2D face image to be synthesized into a 3D face image and matches from the pattern of the 2D face image according to a pre-learned pattern estimation method a face model aligning unit that extracts parameters and aligns a predetermined 3D face model with the 2D face image to obtain a UV map; and a hole area generated in the UV map by an unscanned area of the 2D face image according to a pre-learned pattern estimation method is estimated based on the horizontal symmetry of the face to obtain a compensation UV map, and a compensation UV map is obtained. and a UV map completion unit that extracts fine features and obtains a UV complete map by restoring the UV map reflecting the fine features.

The face model aligning unit is implemented as an artificial neural network in which a pattern estimation method has been previously learned, and represents the shape change, position, and orientation direction required to align the 3D face model to the 2D face image from the pattern of the 2D face image. UV encoder to obtain matching parameters; According to the matching parameter, the 3D face model is aligned to the 2D face image, the 2D face image is mapped to the 3D face model, and the mapped 3D face model is developed on a predetermined two-dimensional UV space to develop the UV mapping unit to obtain a UV map; and a mask acquirer configured to detect a hole area generated by an unscanned area of the 2D face image in the UV map and generate a mask corresponding to the hole area.

The UV map completion unit combines the UV map and the mask, and estimates the reflection coefficient based on the horizontal symmetry of the face from the pattern of the UV map to which the mask is combined according to a pre-learned pattern estimation method. a UV map rough estimation unit that fills in the color of the hole area and extracts and fills the light and shade according to the illumination of the UV map including the hole area from the pattern of the UV map to which the mask is combined to obtain the compensated UV map; and a UV map detailed estimator that combines the compensation UV map and the mask, and obtains the UV complete map by extracting and restoring fine features of the compensation UV map to which the mask is combined according to a pre-learned pattern estimation method. have.

The UV map completion unit uses the training data included by matching the 2D face image for training and the reference UV map that is a pre-generated UV map to correspond to the 2D face image for training, and the face model aligning unit converts the 3D face model to the 2D face Whether or not it is correctly aligned to the image is obtained as a generation loss from energy calculated in a predetermined manner for a learning UV completion map, which is a UV completion map obtained from the 2D face image for training, and the UV map completion unit of the training UV completion map The face model aligning unit and the UV to reduce the generation loss and the discrimination loss by calculating the difference between the UV completion map and the learning UV completion map in consideration of whether the hole area is normally filled or not It may further include an energy determination unit for learning the map completion unit in a competitive learning method.

The UV map rough estimation unit may include: a first mask combining unit for outputting the UV map and the mask by combining them in a predetermined manner; It is implemented as an auto-encoder including an encoder and a decoder implemented as an artificial neural network, and according to a pattern estimation method learned in advance, the reflection coefficient based on the horizontal symmetry of the face is estimated from the pattern of the UV map combined with the mask. a reflection estimator for obtaining an albedo map, which is a UV map from which the color of the region is estimated; The reflection estimator receives a feature map output from a layer corresponding to each of the encoder and decoder, and estimates the illumination coefficient of the UV map according to a pre-learned pattern estimation method based on the feature map applied from each encoder and decoder. a lighting estimator to obtain a lighting map that is a UV map in which the color of the entire area including the hole area is estimated; and a compensation UV map acquisition unit configured to obtain the compensation UV map by elementally multiplying the albedo map and the illumination map.

The UV map detailed estimator comprises: a second mask combiner for outputting the compensated UV map and the mask by combining the mask in a predetermined manner; And it is implemented as an auto encoder including an encoder and a decoder implemented as an artificial neural network and extracts and restores fine features from a pattern of a compensated UV map combined with a mask according to a pre-learned pattern estimation method to obtain the UV complete map. It may include an estimator.

_{The energy determining unit calculates the UV map image energy (E uv} (T,M)) representing the pixel-unit image loss in the effective area except for the hole area designated by the mask M in the UV map T

(Where Tc and Tf represent the compensation UV map and the UV completion map, respectively, and ⊙ is the elemental multiplication operator.), whether the albedo map (A) normally fills the hole area according to the horizontal symmetry of the face The UV map symmetry energy (E _f (T,M)) representing

이진 논리합(binary OR) 연산자이다.)에 따라 계산하며, 상기 UV 완성 맵의 세밀도를 나타내는 UV 맵 지각 에너지(E_p(T,M))를 수학식 (where flip() is a left-right flip function,

It is calculated according to the binary OR operator), and the UV map perception energy (E _p (T,M)) representing the granularity of the UV complete map is calculated according to the equation

(Where F is the pre-trained Visual Geometry Group (VGG) face identifier of the nth selected layer), the total UV energy

(where λ _uv , λ _p and λ _f are energy weights).

The energy determination unit is the total learning UV energy calculated for the _{training UV map (T e} ) and the training mask (M _e ) that are the UV map and mask obtained by the face model aligning unit aligning the 3D face model from the 2D face image for training E(T _e , M _e ) as the production loss (L _G )

Obtained as and the standard UV map (T _ref) and the reference UV map (T _ref) Total reference is calculated for the reference mask (M _ref) corresponding to the UV energy E (T _ref, M _ref) and learn the total Using the UV energy E (T _e , M _e ) and the margin, the discriminant loss (L _D ) is calculated as

can be obtained as

The energy determining section the generation loss (L _G) and the determined loss (L _D), respectively, or the generation loss (L _G) and the sum is a group propagation repeatedly so that the reference loss below a specified station in the determined loss (L _D) Alternatively, the learning may be performed by repeating and backpropagating a predetermined number of times.

In a 3D face image synthesis method according to another embodiment of the present invention for achieving the above other object, a 2D face image to be synthesized into a 3D face image is received, and the pattern of the 2D face image is obtained according to a pre-learned pattern estimation method. obtaining a UV map by extracting matching parameters and aligning a predetermined 3D face model with the 2D face image; and a hole area generated in the UV map by an unscanned area of the 2D face image according to a pre-learned pattern estimation method is estimated based on the horizontal symmetry of the face to obtain a compensation UV map, and a compensation UV map is obtained. and extracting the micro-features to obtain a UV complete map by restoring the UV map reflecting the micro-features.

Therefore, the 3D face image synthesizing apparatus and method according to an embodiment of the present invention can obtain a UV map by accurately aligning a 3D face model to a 2D face image without limiting conditions such as landmarks or feature points, and for the entire face Even if the captured 2D face image is not provided, it can be learned based on the energy of the completed UV map and accurately and accurately restored even the occluded area that was not captured.

1 shows a schematic structure of an apparatus for synthesizing a 3D face image according to an embodiment of the present invention.
FIG. 2 is a diagram for explaining an operation of the 3D face image synthesizing apparatus of FIG. 1 .
3 shows an implementation example of the UV map completion unit of FIG. 1 using an artificial neural network.
4 is a view showing the comparison of the performance of the face model aligning unit and the UV map completion unit of FIG. 1 by competitive learning.
FIG. 5 shows an example of a process in which a 2D face image input to the 3D face image synthesizing apparatus of FIG. 1 is changed while being synthesized into a 3D face image.
6 shows a 3D face image synthesis method according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it does not exclude other components, unless otherwise stated, meaning that other components may be further included. In addition, terms such as "...unit", "...group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

1 shows a schematic structure of a 3D facial image synthesizing apparatus according to an embodiment of the present invention, FIG. 2 is a diagram for explaining the operation of the 3D facial image synthesizing apparatus of FIG. 1 , and FIG. 3 is a diagram using an artificial neural network An example of implementation of the UV map completion part of 1 is shown.

Referring to FIG. 1 , the 3D face image synthesizing apparatus according to the present embodiment includes a face image acquisition unit 100 , a face model aligning unit 200 , a UV map completion unit 300 , and a 3D image synthesis unit 400 . can do.

The face image acquisition unit 100 acquires a 2D face image I for generating a 3D face image. That is, the face image acquisition unit 100 is configured to acquire a 2D face image I to be converted into a 3D face image, and is implemented as a video image capturing device such as a camera, for example, or a storage device in which a video image acquired in advance is stored. Alternatively, it may be implemented as a communication unit that receives a video image from another device through a network.

The face model aligning unit 200 aligns and maps the 3D face model with respect to the 2D face image (I) obtained by the face image acquisition unit 100 according to the pre-learned pattern estimation method to obtain a 3D face image, and obtain The 3D face image is projected onto a two-dimensional UV space to obtain a UV map. 3D Here, a 3D face model may be obtained based on 3DMM.

The 2D face image I acquired by the face image acquisition unit 100 is an image obtained by capturing the subject's face in a specific direction and location using a camera or the like. In addition, the face shape of the subject is also different from each other.

Accordingly, the face model aligning unit 200 first transforms the 3D face model into a shape corresponding to the subject's face shape in order to map the 2D face image (I) to the 3D face model and convert it into a 3D face image, and then transform the 3D face The model is rotated and moved to correspond to the direction and position in which the 2D face image was captured so that the 3D face model can be aligned corresponding to the 2D face image (I).

The face model alignment unit 200 may include a UV encoder 210 and a UV mapping unit 220 . The UV encoder 210 is implemented as an artificial neural network to extract matching parameters for aligning the 3D face model and the 2D face image (I).

As an example, the UV encoder 210 encodes a 2D face image (I) according to a pre-learned pattern estimation method to adjust a shape parameter (f _s ) for transforming a shape of a 3D face model and a 3D face model direction and position. It is possible to extract the location parameter (f _{c ) for the matching parameter.} Here, the shape parameter f _s is a parameter for transforming the shape of the 3D face model to correspond to the face shape of the subject included in the 2D face image I. And the position parameter (f _c ) corresponds to the rotation (R), translation (t) and focal length (f) of the camera, which are camera parameters applied to obtain the 2D face image (I), the position of the 3D face model and This is a parameter for adjusting the orientation direction.

The UV mapping unit 220 transforms the appearance of the 3D face model according to the _{shape parameter f s} among the matching parameters obtained from the UV encoder 210 , and moves the 3D face model according to the _{position parameter f c to 3D} A face model is aligned to a 2D face image (I), and the 2D face image is converted into a 3D face image by mapping the aligned 2D face image to the 3D face model.

Then, the converted 3D face image is unwrapped in a two-dimensional UV space to obtain a two-dimensional UV map T for the 3D face image. At this time, the UV map T is obtained by developing a 3D face image in a two-dimensional UV space based on the three-dimensional coordinates of the 3D face model.

Here, the UV map (T) applies spherical unwrap to the three-dimensional coordinates (X, Y, Z) of each vertex (F) in the 3D face model of the mesh structure according to Equation 1 in the UV coordinate system (v _uv = (u, v)).

이다.where r is the radius of the sphere when unfolding the sphere

am.

However, as described above, in the 2D face image I, only the captured area is expressed due to the characteristics of the 2D image, and not all areas of the subject's face are expressed. Therefore, there is an area that is not captured on the subject's face depending on the captured direction, and the uncaptured occluded area is not mapped to the 3D face model even if the 2D face image (I) is mapped to the 3D face model. So converted When the UV map T is obtained from the 3D face image, the occluded area appears as a hole area.

Accordingly, the mask acquirer 230 may generate a mask M indicating a hole area by sampling the facial visibility of each pixel of the 2D face image I from the UV map T, and the face visibility can be determined by projecting the 3D face model onto the image plane through rasterization. The mask M is a binary mask, and may be created such that, for example, the hole region is filled with 0 and the remaining region is filled with 1.

Here, the mask acquisition unit 230 generates the mask M is generated in the hole region when the UV map completion unit 300 fills the hole region of the UV map with a texture corresponding to the 2D face image I. in order not to be affected by

Referring to the operation of the face model alignment unit 200 with reference to FIG. 2 , in FIG. 2 , for convenience of understanding, a case in which a 3D face model is correctly aligned with a 2D face image (I) and a case in which it is misaligned are (a) and (b).

Accordingly, when the UV encoder 210 is normally trained to accurately estimate the pattern of the 2D face image (I), if the 3D face model is transformed according to the _{matching parameters (f s} , f _{c ) extracted from the UV encoder 210, (} As shown in a), the 3D face model can be precisely aligned with the face shape and orientation direction of the subject shown in the 2D face image (I). On the other hand, if the UV encoder 210 is abnormally trained, if _{the 3D face model is deformed according to the matching parameters (f s} , f _c ), as shown in (b), the face shape or orientation direction of the 3D face model It is aligned differently with the subject of this 2D face image (I).

And in (a) and the normally aligned UV map (T), the face shape and hole area are displayed at the correct location, whereas in (b) and the abnormally aligned UV map (T), the 2D face image is displayed in the 3D face model. It can be seen that not only the face shape was abnormally expressed on the UV map T as it was mapped to the wrong position, but also the position of the hole area was different from the case where it was normally aligned.

The face model aligning unit 200 that obtains the UV map T by aligning the 3D face model and the 2D face image I, including the UV encoder 210 implemented as an artificial neural network, is also referred to as an alignment network. can

The UV map completion unit 300 receives the UV map T obtained from the face model alignment unit 200, estimates the pattern of the UV map T according to a pre-learned pattern estimation method, and applies a texture to the hole area. Fill in to complete the UV map.

The UV map completion unit 300 may include a UV map rough estimation unit 310 and a UV map detailed estimation unit 320 . The UV map rough estimation unit 310 receives the UV map T obtained from the face model aligning unit 200, and from the pattern of the applied UV map T, the reflection coefficient (albedo) and illumination coefficient for the hole area _{A compensation UV map (T c} ) is obtained by estimating a lighting coefficient to roughly fill the color and texture of the hole region in the UV map (T).

The rough UV map estimator 310 may include a first mask combiner 311 , a reflection estimator 312 , an illumination estimator 313 , and a compensated UV map acquirer 314 .

The first mask combining unit 311 combines the UV map T obtained by the UV mapping unit 22 and the mask M obtained by the mask obtaining unit 230 and transmits it to the reflection estimation unit 312 . At this time, the first mask combiner 311 simply concatenates the UV map T and the mask M, or performs elemental multiplication operation on the UV map T and the mask M to deliver it to the reflection estimator 312 . can

The reflection estimator 312 is implemented as an artificial neural network in which the pattern estimation method has been learned in advance, and the first mask combiner 311 estimates the pattern of the UV map T applied in combination with the mask M, By estimating and filling the reflection coefficient of the hole region of the map T, the albedo map A is obtained.

As shown in FIG. 3 , the reflection estimator 312 may have an auto-encoder structure in which an encoder en1 and a decoder de1 including a plurality of layers having a structure symmetric to each other are combined. have. The encoder en1 obtains a feature map by extracting features from the remaining regions except for the hole region designated by the mask M, and the decoder de1 upsamples the hole from the feature map obtained from the encoder en1 Fill in the area to obtain an albedo map (A). Here, the albedo map (A) is a 2D image that fills the hole area by estimating the reflectance according to each position of the UV map in consideration of the 3D face model, and may be extracted as an image in which the effect of lighting is removed. Here, the reflection estimator 312 may be trained to estimate the hole area according to the horizontal symmetry of the human face in the UV map T. That is, since the human face is basically horizontally symmetrical, the hole region, which is an unscanned face region, may be symmetrically estimated and filled from the scanned face region.

In this case, the plurality of layers of the encoder en1 may be configured as a gated convolution layer, and the decoder de1 may have a structure symmetrical to the encoder en1 .

On the other hand, the illumination estimator 313 receives a feature map from among the plurality of layers of the encoder en1 and the decoder de1 of the reflection estimator 312 and corresponds to each other specified in advance, and receives a feature map between the applied feature maps. By extracting the lighting features of the UV map based on the pattern difference of , the lighting map S is obtained.

As the lighting estimator 313 generates the lighting map S by extracting the lighting features from the UV map T separately from the reflection estimating part 312, the lighting map S further increases the difference in features due to lighting. By emphasizing it, the hole region of the UV map T can be filled, and the reflection estimator 312 can extract the albedo-focused feature to fill the hole region of the UV map T.

Here, the albedo map (A) can be viewed as an image filled with the color of the hole area of the UV map (T), and the illumination map (S) can be viewed as an image filled with the contrast of the hole area of the UV map (T) .

The compensated UV map acquisition unit 314 combines the albedo map A estimated by the reflection estimator 312 and the illumination map S estimated by the illumination estimator 313 to obtain _{a compensation UV map T c .} do. _{The compensation UV map acquisition unit 314 may acquire the compensation UV map T c} by elemental product between the pixels of the albedo map A and the illumination map S as shown in Equation 2, for example.

Here, ⊙ denotes an elemental multiplication operator.

In general, the texture of a person's face can be expressed by albedo and light and dark, and the contrast can be determined by lighting and the normal of the shape of the face. Accordingly, if albedo and contrast are applied to each pixel position of the hole region in the UV map T, a compensation UV map T _c in which the texture for the hole region is coarsely filled can be obtained.

A UV map rough estimation unit 310 for obtaining a _{compensation UV map (T c} ) in which the texture for the hole area is roughly filled, including the reflection estimation unit 312 and the lighting estimation unit 313 implemented as an artificial neural network. It can be called a coarse network.

The UV map detailed estimator 320 receives the compensation UV map T _c , which is roughly filled in from the UV map coarse estimator 310 , and fine features _{of the compensation UV map T c} according to a pre-learned pattern estimation method. By extracting and reconstructing the extracted fine features, a UV completion map (T _f ) with _{a more precise texture than the compensation UV map (T c ) is obtained.}

3 , the UV map detailed estimator 320 may include a second mask combiner 321 and a fine estimator 322 .

The second mask coupling unit 321 receives and combines the compensation UV map T _c and the mask M. In this case, the second mask coupling unit 321 may _{simply combine the compensation UV map T c} and the mask M or combine the UV map T and the mask M by elemental multiplication operation.

Similar to the reflection estimator 312 of the UV map coarse estimator 310 , the fine estimator 322 is an auto-encoder in which an encoder en2 and a decoder de2 including a plurality of layers having a structure symmetric to each other are combined. may be configured, but may be configured to include more layers to extract and restore more precise features compared to the UV map rough estimation unit 310 . In addition, a plurality of layers of the encoder en2 may also be configured as gate convolutional layers, and the decoder de2 may have a symmetrical structure to the encoder en2 . However, in the UV map detailed estimator 320 composed of a combination of the encoder en2 and the decoder de2 having a symmetric structure, the predetermined number of layers located in the center is a dilated gated convolution layer. , so that features of various receptive areas can be extracted to improve the precision of facial expression.

As described above, the UV map completion unit 300 roughly supplements the texture of the hole region of the UV map to distinguish the UV map coarse estimator 310 and the UV map detailed estimator 320 that compensates for the fine texture. By providing this, the 3D face image synthesizing apparatus according to the present embodiment can obtain a high-quality UV map.

Meanwhile, the energy determining unit 330 includes the UV encoder 210 of the face model aligning unit 200 composed of an artificial neural network in the 3D face image synthesizing device, the reflection estimating unit 312 of the UV map completion unit 300, and lighting. It is a configuration for learning the estimator 313 and the fine estimator 322 .

The UV encoder 210 , the reflection estimator 312 , the illumination estimator 313 , and the fine estimator 322 composed of an artificial neural network must learn a pattern estimation method in advance. In the case of the UV encoder 210 , a large amount of training data may be acquired according to the existing 3D face alignment technique of aligning a 3D face model to a 2D face image through mathematical operation. Although the technique of aligning a 3D face to a 2D face image by a mathematical operation method uses a complex operation method, it is not easy to obtain training data, but the number of training data required to train the UV encoder 210 can be obtained. can be

And when it is desired to learn the UV encoder 210 using the learning data, the loss (L _uv ) of the UV map T by the UV encoder 210 may be calculated as in Equation 3 .

Here, T _ref denotes a training UV map obtained by a mathematical operation method for _{the training data, T e} denotes a UV map output from the face model aligning unit 200 _{for the training data, and M ref} denotes a hole region in the training data. means a mask representing And ⊙ denotes an elemental multiplication operator.

Loss is calculated as the difference between the mathematical according to formula 3, UV encoder 210 is a mask (M _ref) learning UV map the remaining area other than the hole area indicated by a (T _ref) and the UV map (T _e) ( Learning may be performed such that L _{uv ) is minimized.}

However, the hole area of the UV map T is not captured in the 2D face image I and thus texture information is not obtained. Therefore, the 2D face image in which all areas of the subject's face are captured for learning the reflection estimator 312, the lighting estimator 313, and the fine estimator 322 of the UV map completion unit 300 is used as learning data. Therefore, it is not easy to obtain training data. That is, it is difficult to obtain a large amount of learning data for learning the UV map completion unit 300 .

Accordingly, in this embodiment, including the energy determination unit 330, the face model aligning unit 200 and the UV map completion unit 300 are configured as a generative adversarial network (GAN) structure based on UV energy. By recognizing, competitive learning is performed between the face model alignment unit 200 and the UV map completion unit 300 .

This is based on the concept that when the face model aligning unit 200 accurately aligns the 3D face model to the 2D face image I, the UV map completion unit 300 completes a higher quality UV map.

To this end, it is necessary to first define the loss of the UV map completion unit 300 . Accordingly, the UV map completion unit 300 calculates the loss by the reflection estimator 312 , the illumination estimator 313 , and the fine estimator 322 as energy, and in the UV map T applied first, the mask M _{The image loss (E uv} (T,M)) in pixels in the effective area excluding the hole area designated by ) can be considered. In the UV map (T), the remaining area except the hole area by the mask (M) is a texture obtained from the 2D face image, so the texture of the 2D face image should be maintained as much as possible, so the UV map image energy (E _uv (T, M) )) can be defined according to Equation (4).

_{And the UV map symmetric energy (E f} (T,M)) indicating whether the reflection estimator 312 obtained the albedo map (A) by normally filling the hole region according to the horizontal symmetry of the human face is expressed in Equation 5 can be defined accordingly.

이진 논리합(binary OR) 연산자이다.where flip() is a left-right flip function,

It is a binary OR operator.

_{Meanwhile, the UV map perception energy E p} (T,M) indicating whether the fine estimator 322 has completed the UV map in detail may be defined according to Equation 5.

where F is the pre-trained Visual Geometry Group (VGG) face identifier of the nth selected layer.

Therefore, the total UV energy of the UV map completion unit 300 may be obtained by Equation 7 by combining the energies calculated in Equations 4 to 6 .

where λ _uv , λ _p and λ _f are energy weights.

When the face model alignment unit 200 and the UV map completion unit 300 are recognized as a GAN structure and competitive learning is performed, the face model alignment unit 200 is assumed as a generator, and the energy determination unit ( 330) by considering the UV map completion unit 300 as a discriminator to obtain the generation loss ( _LG ) and the discrimination loss (L _D ) as shown in Equations 8 and 9 based on Equation 7 can

Here, [·] ₊ = max(0, ·), which is a function that selects the larger value between 0 and ·, and m represents a positive margin value. And the initial generation loss (L _G ') of the face model aligning unit 200 is defined by Equation 10 from _{the loss (L uv} ) of the UV map (T) of Equation 3 and the discrimination loss (L _{D ) of Equation 9} can be

where λ _D is the discriminant loss weight.

The energy determination unit 330 calculates a loss according to Equations 8 to 10, and backpropagates the calculated loss to compete for the face model aligning unit 200 and the UV map completion unit 300 to learn. At this time, the energy determination unit 330 repeatedly calculates and backpropagates the loss until a predetermined number of times or the calculated loss becomes less than or equal to a predetermined reference value to learn the face model alignment unit 200 and the UV map completion unit 300 . can do it

In the 3D face image synthesizing apparatus according to the present embodiment, the face model aligning unit 200 can be viewed as a generator that generates a UV map according to the 3D face model, and the UV map completion unit 300 is configured to convert the 3D face model into a 2D face image. It can be viewed as a discriminator that checks the loss by filling the hole area of the UV map to determine whether the UV map is normally generated by being accurately aligned with the GAN. have.

Here, the energy determination unit 330 may be removed after learning is completed.

The 3D image synthesizing unit 400 maps the UV complete map (T _f ) output from the UV map complete unit ( 300 ) to the 3D face model deformed according _{to the shape parameter ( f s} ) in the face model aligning unit ( 200 ). Thus, a 3D face image is synthesized. Here, the UV completion map (T _f ) includes the image of the entire face and has coordinate values on the UV coordinate system corresponding to the three-dimensional coordinates of the 3D face model. It can be synthesized into a 3D face image. Accordingly, the orientation direction of the synthesized 3D face image without the hole region can be freely adjusted regardless of the orientation direction of the subject in the 2D face image as shown on the right side of FIG. 2 according to the user's request.

In general, the information required for synthesizing a 3D face image is a UV completion map (T _f ), and obtaining an actual synthesized 3D face image from the UV completion map (T _f ) is a requirement in practical applications, so the 3D image The synthesizing unit 400 may be omitted from the 3D face image synthesizing apparatus.

4 is a view showing the comparison of the performance of the face model aligning unit and the UV map completion unit of FIG. 1 by competitive learning.

4, (a) shows an input 2D face image, (b) shows a UV map (T) output from the face model aligner 200, (c) shows an auto-encoder without performing a competitive learning method Shows the result of supplementing the UV map (T) using (auto-encoder), (d) shows that the UV map completion unit 300, which has performed competitive learning according to this embodiment, supplements the UV map (T) Shows the UV completion map (T _f ).

Comparing (c) and (d) in FIG. 4 , in (c), the hole area is filled with a texture irrelevant to the subject's face, whereas in the UV map completion unit 300 on which competitive learning is performed according to the present embodiment. In the case of the output UV completion map (T _f ), it can be seen that the hole area is filled with a natural texture according to the face shape of the subject.

FIG. 5 shows an example of a process in which a 2D face image input to the 3D face image synthesizing apparatus of FIG. 1 is changed while being synthesized into a 3D face image.

In FIG. 5 , (a) is an input 2D face image, (b) is a UV map (T _e ) output from the face model aligning unit 200 , (c) is estimated by the lighting estimator 313 . An illumination map S is shown, and (d) is an albedo map A estimated by the reflection estimation unit 312 . And (e) shows the compensated UV map (T _c ) obtained by the compensated UV map acquisition unit 314 , (f) shows the UV complete map (T _f ) estimated by the UV map detailed estimator 320 . .

(g) and (h) show 3D face images obtained by mapping the illumination map (S) and the UV completion map (T _f ) to the 3D face model, respectively.

As shown in (h), the 3D face image synthesized according to this embodiment can not only synthesize high-quality 3D face images for different 2D face images such as men or women, but also 3D face images for black and white images. It can be seen that face images can be synthesized.

6 shows a 3D face image synthesis method according to an embodiment of the present invention.

Referring to FIGS. 1 to 3 , the 3D face image synthesizing method of FIG. 6 is described. A learning step S10 for learning the artificial neural network included in the 3D face image synthesizing apparatus is performed. In the learning step, first, learning data including a 2D face image for training and a reference UV map (T _ref ) corresponding to the 2D face image for training is acquired ( S11 ). Here, the reference UV map (T _ref ) is a UV map obtained by mapping a 2D face image for training to a 3D face model aligned with respect to the 2D face image through mathematical operation and developing it on two-dimensional UV spatial coordinates.

When the training data is obtained, the pattern of the 2D face image for training is estimated according to the pattern estimation method that is learned by receiving the 2D face image for training, and the matching parameters (f _s , f _c ) are extracted, and the extracted matching parameters (f _s) , f _c ), the shape, position, and orientation of a predetermined 3D face model are adjusted, and a 2D face image for training is mapped to the adjusted 3D face model _{to obtain a training UV map (T e} ) (S12). In this case, by sampling the facial visibility of the 2D face image for training, a learning mask (M _e ) indicating an unscanned hole area in the subject's face may be acquired together.

And according to the pattern estimation method learned using the learning UV map (T _e ) and the learning mask (M _e ), the hole area according to the reflection coefficient is estimated based on the symmetry of the face according to the albedo map (A) and the illumination coefficient By estimating the estimated illumination map (S), a compensation UV map (T _{ec ) is obtained that schematically supplements the training UV map (T e} ), and according to the pattern estimation method that is learned again for the compensation UV map (T _{ec )} By supplementing it to have a precise texture, a learning UV completion map (T _ef ) is obtained (S13).

After learning UV map image energy (E _uv (T _e ,M _e )) of UV completion map (T _ef ), UV map symmetry energy (E _f (T _e ,M _e )) and UV map perceptual energy (E _p ( T _e, M _e)) the total UV energy (e (T _e of the learning UV complete map (T _ef) on the basis of, M _e)) to generate losses (L _G), and learning UV complete map (T _ef obtained by ) with a margin (m) according to the relationship between the total UV energy (E(T _e ,M _e )) and the total UV energy (E(T _ref ,M _ref )) for the reference UV map (T _{ref ).} calculated by backpropagation discriminant loss (L _D) thereby learn the neural network (S14). At this time, the generation loss (L _G ) can be seen as reflecting the loss due to the alignment error of the 3D face model performed by training on the 2D face image for training when the training UV map (T _{e ) is generated, and the discrimination loss} (L _D) can be considered as generated in the process to supplement the study UV complete map (T _ef) the learning UV map (T _e) to determine the alignment error of the learning UV map (T _e) and the determined error. Therefore it may be viewed as performing a competitive learning of the discrimination step that complements the production and study UV map (T _e) of the learning map UV (T _e) is determined.

And it is determined whether the learning is finished (S15). At this time, the learning may _{be set to end after repeating until the generation loss (LG} ) and discrimination loss (L _D ), respectively, or _{the sum of generation loss (LG} ) and discrimination loss (L _D ) is less than or equal to a predetermined reference loss. can Alternatively, it may be set to be completed after being repeated a predetermined number of times.

If it is determined that the learning has been completed by satisfying the set condition, a 2D face image I to be synthesized into a 3D face image is obtained (S20).

And by estimating the pattern of the 2D face image (I) according to the pre-learned pattern estimation method, a matching parameter (f _s , f _c ) is extracted (S21). Also, according to the extracted mapping parameters (f _s , f _c ), the 3D face model is transformed and moved to align the 3D face model to the 2D face image (I), and the 2D face image is mapped to the aligned 3D face model. A UV map T is obtained by developing a 3D face image in a two-dimensional UV coordinate system (S22). In this case, the mask M may also be acquired from the 2D face image I. The mask M may be obtained by analyzing an unscanned hole area in the subject's face in the 2D face image I.

When the UV map (T) and mask (M) corresponding to the 2D face image (I) are obtained, the UV map (T) and the mask (M) are combined in a predetermined manner, and the mask according to the pre-learned pattern estimation method (M) is obtained by estimating the pattern of the combined UV map (T), by extracting the albedo map (A) and the illumination map (S) filled with the hole area designated by the mask (M) (S41). Here, the albedo map (A) uses an artificial neural network composed of an auto-encoder including an encoder and a decoder to estimate the reflection coefficient based on the horizontal symmetry of the human face, thereby expressing the color of the UV map (T) filled with the hole area. it's a map And the lighting map (S) is a map obtained by receiving a feature map output from the corresponding layer of the encoder and the decoder from the auto encoder, and extracting the lighting feature of the UV map based on the pattern difference between the applied feature maps.

When the albedo map (A) and the illumination map (S) are obtained, the compensation UV map (T _c ) in which the hole area is roughly compensated is obtained by elementally multiplying the albedo map (A) and the illumination map (S) ( S42 ).

When the compensation UV map (T _c ) is obtained, the compensation UV map (T _c ) and the mask (M) are combined in a predetermined way, and the compensation UV map ( T _c) estimates the pattern by extracting a fine features of, and obtain, based on the extracted fine features UV map (T _c) UV complete map having a fine texture to complement the subdivision of (T _f) (S50) .

Additionally, in response to a user command, a 3D face image may be synthesized by mapping the _{obtained UV completion map T f to a corresponding 3D face model ( S60 ).}

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: face image acquisition unit 200: face model alignment unit
210: UV encoder 220: UV mapping unit
230: mask acquisition unit 300: UV map completion unit
310: UV map rough estimation unit 311: first mask coupling unit
312: reflection estimator 313: lighting estimator
314: compensation UV map acquisition unit 320: UV map detailed estimation unit
321: second mask coupling unit 322: fine estimation unit
330: energy determination unit 400: 3D image synthesis unit

Claims (18)

A 2D face image to be synthesized into a 3D face image is received, matching parameters are extracted from the pattern of the 2D face image according to a pre-learned pattern estimation method, and a predetermined 3D face model is aligned with the 2D face image to obtain a UV map a face model alignment unit; and
According to the pre-learned pattern estimation method, the hole area generated in the UV map by the unscanned area of the 2D face image is estimated and filled based on the horizontal symmetry of the face to obtain a compensation UV map, and the fineness of the compensation UV map A 3D face image synthesizing apparatus comprising a UV map completion unit that extracts features and obtains a UV complete map by restoring a UV map reflecting fine features. According to claim 1, wherein the face model alignment unit
A pattern estimation method is implemented as a pre-trained artificial neural network to obtain the matching parameters indicating the shape change and position and orientation direction required to align the 3D face model to the 2D face image from the pattern of the 2D face image UV encoder;
According to the matching parameter, the 3D face model is aligned to the 2D face image, the 2D face image is mapped to the 3D face model, and the mapped 3D face model is developed on a predetermined two-dimensional UV space to develop the UV mapping unit to obtain a UV map; and
and a mask acquisition unit configured to detect a hole area generated by an unscanned area of the 2D face image in the UV map and generate a mask corresponding to the hole area. The method of claim 2, wherein the UV map completion unit
The UV map and the mask are combined, and the reflection coefficient based on the horizontal symmetry of the face is estimated from the pattern of the UV map to which the mask is combined according to a pre-learned pattern estimation method to determine the color of the hole area generated in the UV map. a UV map rough estimation unit for obtaining the compensated UV map by filling in and extracting the contrast according to the illumination of the UV map including the hole area from the pattern of the UV map coupled with the mask; and
A 3D face comprising a UV map detailed estimator that combines the compensation UV map and the mask, and obtains the UV complete map by extracting and restoring fine features of the compensation UV map to which the mask is combined according to a pre-learned pattern estimation method image compositing device. The method of claim 3, wherein the UV map completion part
Using the training data included by matching the 2D face image for training and the reference UV map, which is a pre-generated UV map corresponding to the 2D face image for training, the face model aligning unit accurately aligns the 3D face model to the 2D face image Whether or not is obtained as a generation loss from energy calculated in a predetermined manner for a learning UV completion map, which is a UV completion map obtained from the 2D face image for training,
By calculating the difference between the UV completion map and the learning UV completion map in which the margin is considered whether the UV map completion unit normally fills the hole area of the learning UV completion map, a discrimination loss is obtained,
3D face image synthesizing apparatus further comprising an energy determining unit configured to train the face model aligning unit and the UV map completion unit in a competitive learning method so that the generation loss and the discrimination loss are reduced. The method of claim 4, wherein the UV map rough estimation unit
a first mask coupling unit for outputting the UV map and the mask by combining them in a predetermined manner;
It is implemented as an auto-encoder including an encoder and a decoder implemented as an artificial neural network, and according to a pattern estimation method learned in advance, the reflection coefficient based on the horizontal symmetry of the face is estimated from the pattern of the UV map combined with the mask. a reflection estimator for obtaining an albedo map, which is a UV map from which the color of the region is estimated;
The reflection estimator receives a feature map output from a layer corresponding to each of the encoder and decoder, and estimates the illumination coefficient of the UV map according to a pre-learned pattern estimation method based on the feature map applied from each encoder and decoder. a lighting estimator to obtain a lighting map that is a UV map in which the colors of the entire area including the hole area are estimated; and
and a compensation UV map acquisition unit configured to obtain the compensation UV map by elementally multiplying the albedo map and the illumination map. The method of claim 5, wherein the UV map detailed estimation unit
a second mask coupling unit for outputting the compensation UV map and the mask by combining them in a predetermined manner; and
Fine estimation to obtain the UV complete map by extracting and restoring fine features from the pattern of the compensated UV map combined with a mask according to a pattern estimation method learned in advance by being implemented as an auto encoder including an encoder and a decoder implemented as an artificial neural network 3D face image synthesizing device with wealth. The method of claim 6, wherein the energy determination unit
_{In the UV map (T), the UV map image energy (E uv} (T,M)) representing the image loss in pixels in the effective area excluding the hole area specified by the mask (M) is calculated by the equation

(where Tc and Tf represent the compensation UV map and the UV completion map, respectively, and ⊙ is the elemental product operator.)
_{The UV map symmetric energy (E f} (T,M)) indicating whether the albedo map (A) normally fills the hole area according to the horizontal symmetry of the face is calculated

(where flip() is a left-right flip function,

It is calculated according to the binary OR operator),
_{The UV map perceptual energy (E p} (T,M)) representing the granularity of the UV complete map is calculated by the equation

(Where F is the pre-trained Visual Geometry Group (VGG) face identifier of the nth selected layer.)
total UV energy

(Where λ _uv , λ _p and λ _f are energy weights.)
A 3D face image synthesizing device that counts with The method of claim 7, wherein the energy determination unit
The total learning UV energy E (T _{e ) calculated for the training UV map (T e} ) and the training mask (M _e ) that are the UV map and mask obtained by aligning the 3D face model from the 2D face image for training by the face model aligner , M _e ) as the production loss (L _G )

obtained as
The reference UV map (T _ref) and the reference UV map (T _ref) based on a mask (M _ref) a total standard UV energy E (T _ref, M _ref) and the total study UV energy E (T, which is calculated for the corresponding _e , M _e ) and the margin using the above discriminant loss (L _D ) is expressed in Equation

A 3D face image synthesizing device acquired as The method of claim 8, wherein the energy determination unit
The generation loss ( _LG ) and the discrimination loss (L _D ), respectively, or the _{sum of the generation loss (LG} ) and the discrimination loss (L _D ) is repeatedly backpropagated to be less than or equal to a predetermined reference loss, or A 3D face image synthesizing device that performs learning by repeating backpropagation a number of times. A 2D face image to be synthesized into a 3D face image is received, matching parameters are extracted from the pattern of the 2D face image according to a pre-learned pattern estimation method, and a predetermined 3D face model is aligned with the 2D face image to obtain a UV map to do; and
According to the pre-learned pattern estimation method, the hole area generated in the UV map by the unscanned area of the 2D face image is estimated and filled based on the horizontal symmetry of the face to obtain a compensation UV map, and the fineness of the compensation UV map A method for synthesizing a 3D face image, comprising the step of extracting features and restoring a UV map reflecting fine features to obtain a UV complete map. The method of claim 10, wherein the acquiring the UV map comprises:
A pattern estimation method is implemented as a pre-trained artificial neural network to obtain the matching parameters indicating the shape change and position and orientation direction required to align the 3D face model to the 2D face image from the pattern of the 2D face image step;
According to the matching parameter, the 3D face model is aligned to the 2D face image, the 2D face image is mapped to the 3D face model, and the mapped 3D face model is developed on a predetermined two-dimensional UV space to develop the obtaining a UV map; and
and detecting a hole area generated by an unscanned area of the 2D face image in the UV map and generating a mask corresponding to the hole area. 12. The method of claim 11, wherein obtaining the UV complete map comprises:
The UV map and the mask are combined, and the reflection coefficient based on the horizontal symmetry of the face is estimated from the pattern of the UV map to which the mask is combined according to a pre-learned pattern estimation method to determine the color of the hole area generated in the UV map. obtaining the compensated UV map by filling and extracting the light and shade according to the illumination of the UV map including the hole area from the pattern of the UV map to which the mask is combined; and
3D face image synthesis method comprising the step of combining the compensation UV map and the mask, and outputting the UV completion map by extracting and restoring fine features of the compensation UV map to which the mask is combined according to a pre-learned pattern estimation method . The method of claim 12, wherein obtaining the UV complete map comprises:
Using the training data included by matching the 2D face image for training and the reference UV map, which is a pre-generated UV map corresponding to the 2D face image for training, it is determined whether the 3D face model is correctly aligned with the 2D face image for the training 2D obtained as a generation loss from energy calculated in a predefined manner for a training UV completion map, which is a UV completion map obtained from a face image;
By calculating the difference between the UV completion map and the learning UV completion map in which the margin is considered whether or not the hole area of the learning UV completion map is normally filled, a discrimination loss is obtained,
3D face image synthesis method further comprising learning in a competitive learning method so that the generation loss and the discrimination loss are reduced. 14. The method of claim 13, wherein obtaining the compensation UV map comprises:
combining and outputting the UV map and the mask in a predetermined manner;
It is implemented as an auto-encoder including an encoder and a decoder implemented as an artificial neural network, and according to a pattern estimation method learned in advance, the reflection coefficient based on the horizontal symmetry of the face is estimated from the pattern of the UV map combined with the mask. obtaining an albedo map, which is a UV map in which the color of the region is estimated;
Receive a feature map output from each corresponding layer of the encoder and decoder of the auto encoder, and estimate the illumination coefficient of the UV map according to a pre-learned pattern estimation method based on the feature map applied from each encoder and decoder to obtain an illumination map that is a UV map in which the color of the entire area including the hole area is estimated; and
and obtaining the compensated UV map by elementally multiplying the albedo map and the illumination map. The method of claim 14, wherein outputting the UV completion map comprises:
combining and outputting the compensation UV map and the mask in a predetermined manner; and
Obtaining the UV complete map by extracting and restoring fine features from the pattern of the compensated UV map that is implemented as an auto encoder including an encoder and a decoder implemented as an artificial neural network and is combined with a mask according to a pre-learned pattern estimation method 3D face image synthesis method including. 16. The method of claim 15, wherein the learning step
_{In the UV map (T), the UV map image energy (E uv} (T,M)) representing the image loss in pixels in the effective area excluding the hole area specified by the mask (M) is calculated by the equation

(where Tc and Tf represent the compensation UV map and the UV completion map, respectively, and ⊙ is the elemental product operator.)
_{The UV map symmetric energy (E f} (T,M)) indicating whether the albedo map (A) normally fills the hole area according to the horizontal symmetry of the face is calculated

(where flip() is a left-right flip function,

It is calculated according to the binary OR operator.)
_{The UV map perceptual energy (E p} (T,M)) representing the detail of the UV complete map is calculated by the equation

(Where F is the pre-trained Visual Geometry Group (VGG) face identifier of the nth selected layer.)
total UV energy

(Where λ _uv , λ _p and λ _f are energy weights.)
3D face image synthesis method calculated by . The method of claim 16, wherein the learning step
The total learning UV energy E (T _{e ) calculated for the training UV map (T e} ) and the training mask (M _e ) that are the UV map and mask obtained by aligning the 3D face model from the 2D face image for training by the face model aligner , M _e ) into the equation

obtaining as the production loss (L _{G );}
The reference UV map (T _ref) and the reference UV map (T _ref) based on a mask (M _ref) a total standard UV energy E (T _ref, M _ref) and the total study UV energy E (T, which is calculated for the corresponding _{Equation using e} , M _e ) and margin

3D face image synthesis method comprising the step of obtaining the discrimination loss (L _{D ) according to} 18. The method of claim 17, wherein the learning step
The generation loss ( _LG ) and the discrimination loss (L _D ), respectively, or the _{sum of the generation loss (LG} ) and the discrimination loss (L _D ) is repeatedly backpropagated to be less than or equal to a predetermined reference loss, or A 3D face image synthesis method that performs learning by repeating and backpropagating a number of times.

Images