KR20210147647A - Apparatus and method for color synthesis of face images - Google Patents

Abstract

The present invention provides an apparatus and method for color synthesis of face images. The apparatus comprises: an input preprocessing unit configured to receive a UV map obtained in a predetermined manner from a 2D face image and obtain an input UV map by combining a mask in which a hole area and the remaining area of the received UV map are separated and a flip UV map in which the UV map is inverted left and right with the UV map; a UV map completion unit implemented as an artificial neural network to receive the input UV map and configured to generate a complete UV map in which the hole area of the UV map is filled with colors from a pattern of the input UV map obtained using a pre-learned pattern assuming method; and a UV map determining unit that is provided at the time of learning, and configured to receive and divide the complete UV map and the mask into two left and right parts, respectively, assume a real image and a fake image among the two divided completed UV maps using the learned pattern assuming method, and calculate and backpropagate a generation loss for the completed UV map generated by discriminating the real image and the fake image based on the two divided masks. The apparatus performs self-discrimination-based competitive learning using the areas acquired from the 2D image as learning data based on the symmetry of the human face. Therefore, the apparatus and method for color synthesis of face images can complete a UV map by synthesizing colors of an occluded area, not captured from a 2D image, naturally, without needing to acquire a separate 2D face image for learning or a completed UV map.

Description

Apparatus and method for color synthesis of face images

The present invention relates to an apparatus and method for synthesizing a face image color, and to an apparatus and method for synthesizing a face image color for a three-dimensional face model in which self-determination-based learning is performed.

3D face models are used in a variety of applications such as face compositing, 3D animation, and image editing. Building a human-like 3D face is therefore an important issue in computer vision and computer graphics.

3D Morphable Model (3DMM), a statistical model of 3D faces and textures, is the most widely used model for obtaining 3D face images from objects of 2D face images. With the advent of 3D sensors such as stereo cameras, it is possible to collect accurate and large number of 3D face data sets, so 3DMM has a powerful ability to express various natural facial shapes.

If you want to create a 3D face image using a 3D face model such as 3DMM, align the 2D face image to the 3D face model and map it, and then place the 3D face model to which the 2D image is mapped on the specified 2D UV space coordinates An unwrapped UV map is commonly used. However, since the 2D face image is captured in a limited environment, it is very difficult to obtain the overall shape or color of the subject's face expressed in the 2D face image. In some cases, the entire shape and color information of the subject's face may be acquired using a 3D scanning device or a multi-view capture system, but this requires expensive equipment and can be photographed only in a limited space. It becomes a factor that makes it difficult to obtain a face model.

Therefore, recently, research for generating a 3D face model based on a 2D face image obtained by being easily captured in a limited environment such as a photo is being actively conducted. However, as described above, in the 2D face image captured in a limited environment, there is an occluded area that is not captured and thus not expressed in the subject's face, so a technology capable of naturally restoring the occluded area is required. In particular, in order to restore a natural facial texture, a technology capable of accurately synthesizing facial colors in an occluded area is required.

Korean Patent Publication No. 2017-0019779 (published on February 22, 2017)

SUMMARY OF THE INVENTION It is an object of the present invention to provide an apparatus and method for synthesizing a color of a face image capable of naturally synthesizing colors of an occluded region that are not captured in a 2D face image to complete a UV map.

Another object of the present invention is a face image color synthesizing device that can easily self-identify and learn to complete a UV map without a 2D face image or a completed UV map obtained using a 3D scanning device for learning or a multi-view capture system, etc. and to provide a method.

A face image color synthesizing apparatus according to an embodiment of the present invention for achieving the above object receives a UV map obtained in a predetermined manner from a 2D face image, and a mask for dividing the hole area and the remaining area of the applied UV map and an input pre-processing unit for obtaining an input UV map by combining the left and right inverted flip UV map of the UV map with the UV map; a UV map completion unit that is implemented as an artificial neural network to receive an input UV map and generate a complete UV map filled with the color of the hole region of the UV map from the pattern of the applied input UV map according to a pre-learned pattern estimation method; and provided at the time of learning, receiving the completed UV map and the mask, dividing the left and right into two, respectively, estimating a real image and a fake image among the two completed UV maps divided according to the learned pattern estimation method It includes a UV map judging unit that calculates and backpropagates a generation loss for a completed UV map generated by discriminating an actual real image and a fake image based on the two masks.

The input preprocessor analyzes the UV map to classify the hole region and the remaining region, and allocates different bit values to each pixel of the divided hole region and the remaining region to generate the mask composed of binary bit values. acquisition department; a UV map flip unit to obtain the flip UV map by inverting the UV map left and right; and a UV map combiner configured to obtain the input UV map by combining the UV map, the mask, and the flip UV map in a predetermined manner.

The UV map completion unit is composed of a plurality of layers, an encoder that extracts the features of the input UV map step by step according to a pre-learned pattern estimation method to obtain a feature map; and a decoder configured with a number of layers corresponding to the plurality of layers of the encoder, and upsampling the feature map in stages to obtain the complete UV map.

In the decoder, each of the plurality of layers may be connected to a corresponding layer among the plurality of layers of the encoder through a skip connection.

The UV map determination unit receives the completed UV map and divides the left and right into two, a UV map division unit to obtain a left image and a right image; a competitive determination unit implemented as an artificial neural network to estimate a real image corresponding to a portion having a small hole region in the UV map among the left image and the right image and a fake image corresponding to the remaining portion; and a loss back propagation unit for calculating the generation loss based on the estimation result of the competitive determination unit and backpropagating the calculated generation loss.

The UV map determination unit receives the mask and divides it into two left and right to obtain a left mask and a right mask; The method may further include an effective image determining unit for discriminating an actual real image and a fake image of the UV map based on the sum of pixel values of the left mask and the right mask, and notifying the loss back propagation unit.

The loss back propagation unit receives the input UV map (I _{in ) and receives the reconstruction loss (L Grec} ) in the UV map reconstruction process to obtain the complete UV map (I _out ) and real for the left image and the right image _{Backpropagation} may be performed by calculating the generation loss (L _G ) as the sum of the adversarial loss (L Gadv ) based on the image and the estimated decision loss (L _{D ) of the fake image.}

In a face image color synthesis method according to another embodiment of the present invention for achieving the above other object, a UV map obtained in a predetermined manner from a 2D face image is applied, and the hole area and the remaining area of the applied UV map are separated. obtaining an input UV map by combining a mask and a flip UV map in which the UV map is left and right to the UV map; receiving an input UV map using an artificial neural network and generating a complete UV map filled with the color of the hole region of the UV map from the applied input UV map pattern according to a pre-learned pattern estimation method; And in the learning state, the completed UV map and the mask are applied and divided into left and right, respectively, and the real image and the fake image are estimated among the two completed UV maps divided according to the learned pattern estimation method, while the divided two and performing learning by calculating and backpropagating a generation loss for a completed UV map generated by discriminating an actual real image and a fake image based on the mask.

Therefore, the face image color synthesizing apparatus and method according to an embodiment of the present invention performs self-determination-based competitive learning by using a region obtained from a 2D image as learning data based on the symmetry of a human face, thereby separately for learning 2D A UV map can be completed by naturally synthesizing the colors of the occluded area not captured in the 2D face image without acquiring a face image or a completed UV map.

1 is a diagram for schematically explaining a 3D face image synthesis technique.
2 shows a schematic structure of an apparatus for synthesizing a face image color according to an embodiment of the present invention.
3 shows an implementation example using the artificial neural network of the UV map completion unit and the UV map determination unit of FIG. 1 .
4 shows an example of a 3D face image using a UV map completed by the face image color synthesizing apparatus according to the present embodiment.
5 shows a result of comparing a UV map completed in the face image color synthesizing apparatus according to the present embodiment with an existing UV map.
6 is a diagram illustrating a face image color synthesis method according to an embodiment of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it does not exclude other components, unless otherwise stated, meaning that other components may be further included. In addition, terms such as "...unit", "...group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

1 is a diagram for schematically explaining a 3D face image synthesis technique.

Referring to FIG. 1, the 3D face image synthesis technique is described. First, as in (a), a 2D face image is received, and the position, size, and capture direction of the subject included in the applied 2D face image are analyzed. Aligns preset 3D face models in a predefined manner.

And, as shown in (b), the 2D face image is mapped to the aligned 3D face model to be converted into a 3D face image, and the converted 3D face image is unwraped in a two-dimensional UV space to obtain a UV map (I _ref ). acquire At this time, since there is an uncaptured area in the 2D face image, as shown in the upper part of (b), a hole area in which the subject's face texture is not expressed appears in the _{UV map (I ref ).} The hole region may be extracted in the form of a mask in which the hole region and the remaining regions excluding the hole region are separated as shown in the lower image of (b). _{The UV map (I ref} ) obtained as in (b) is not suitable for synthesizing 3D face images due to the included hole area. This is because 3D face image synthesis must be able to acquire natural 3D face images in various positions, sizes, and capture directions, but when there is a hole area, the capture direction is inevitably fixed so that the hole area is not expressed. That is, the 3D face image can be synthesized only in the same direction as the direction of the 2D face image.

Therefore, in the UV map (I _ref ) including the hole area in (b), after completing the UV map by supplementing the texture such as the subject's face color as shown in (c) in the hole area, it must be wrapped in a 3D face model. As shown in (d), various 3D face image synthesis is possible.

Accordingly, in the present invention, by estimating and complementing the color of the hole area by reflecting the left-right symmetry of the human face with respect _{to the UV map (I ref} ) including the hole area of (b) _{, a naturally completed completed UV map (I out} ) make it possible to obtain In addition, based on the left-right symmetry of the human face _{, using the area acquired from the 2D face image of the completed UV map (I out} ) as training data to perform learning based on a competitive learning technique to determine whether the hole area is normally filled. do. That is, by performing self-discrimination-based competitive learning using itself as learning data, it is easily learned without requiring separate learning data.

FIG. 2 shows a schematic structure of an apparatus for synthesizing a face image color according to an embodiment of the present invention, and FIG. 3 shows an implementation example using an artificial neural network of the UV map completion unit and the UV map determination unit of FIG. 1 .

Referring to FIG. 2 , the face image color synthesizing apparatus according to the present embodiment includes a face image acquisition unit 100 , a UV map acquisition unit 200 , an input preprocessor 300 , a UV map completion unit 400 , and a UV map A determination unit 500 may be included.

The face image acquisition unit 100 acquires a 2D face image for generating a 3D face image. That is, the face image acquisition unit 100 is configured to acquire a 2D face image to be converted into a 3D face image, and is implemented as a video image capturing device such as a camera, or a storage device or network in which a video image acquired in advance is stored. It may be implemented as a communication unit that receives a video image from another device through the communication unit.

The UV map acquisition unit 200 obtains a 3D face image by aligning and mapping a predetermined 3D face model with respect to the 2D face image acquired by the face image acquisition unit 100, and uses the obtained 3D face image with a two-dimensional UV A UV map (I _ref ) is obtained by projecting on space. 3D Here, a 3D face model may be obtained based on 3DMM.

The 2D face image obtained by the face image acquisition unit 100 is an image obtained by capturing the subject's face in a specific direction and location using a camera, etc., and the subject's face shape is also different for each individual. Therefore, first, the 3D face model is transformed into a shape corresponding to the face shape of the 2D face image, and the 3D face model is converted It should be able to be sorted accordingly. Accordingly, as shown in Fig. 1 (a), a landmark or feature point is set in the 2D face image, and the 3D face model is aligned based on the set landmark or feature point to match the 2D face image and the 3D face model. can

When the 2D face image and the 3D face model are aligned, the UV map acquisition unit 200 maps the 2D face image to the 3D face model to convert the 2D face image into a 3D face image.

Since the 3D face model has a mesh structure, the UV map acquisition unit 200 calculates the three-dimensional position coordinates (v _I = (x, y, z)) of each vertex of the converted 3D face image into a sphere according to Equation 1 _{By transforming the position coordinates (v U} = (u, v)) of the two-dimensional UV coordinate system by spherical unwrap, a _{UV map including holes (I ref} ) can be obtained as shown in (b) of FIG. 2 . have.

이고, α_u, β_u 및 α_v, β_v 는 이미지의 경계에서 랩핑되지 않은 영역을 찾기 위한 배율 및 변환 상수이다.where r is the radius of the sphere when unfolding the sphere

, and α _u , β _{u ,} and α _v , β _v are magnification and transformation constants for finding the unwrapped region at the boundary of the image.

_{Since a technique for acquiring the UV map (I ref} ) from the 2D face image has been disclosed in various ways in addition to the above-described technique, the UV map acquisition unit 200 may acquire the UV map (I _ref ) using another technique. Since this is a known technique, it will not be described in detail here.

_{The input preprocessor 300 receives the UV map I ref} including the holes acquired by the UV map acquirer 200 , and the mask M and the hole indicating the hole area in the applied UV map I _{ref .} A flip UV map (I _flip ) in which the included UV map (I _ref ) is inverted is obtained, and an input UV map (I _in ) is obtained by combining with the applied UV map (I _{ref ).}

The input preprocessor 300 may include a mask acquisition unit 310 , a UV map flip unit 320 , and a UV map combiner 330 .

_{The mask acquisition unit 310 analyzes the hole region in the UV map (I ref} ) applied from the UV map acquirer 200, and divides the hole region and the region other than the hole region into a binary bit value mask (M) to acquire Mask obtaining unit 310 includes a mask that represents the hole regions by sampling the face visibility (facial visibility) for each pixel of the 2D face image from, but can identify the hole regions in various ways, one example UV map (I _ref) ( M) can be created. Here, the face visibility can be determined by projecting the 3D face model onto the image plane through rasterization. The mask M is a binary mask, and may be created such that, for example, the hole region is filled with 0 and the remaining region is filled with 1.

When the UV map completion unit 400 generates the completed UV map (I _{out ) based on the input UV map (I in} ), the mask acquisition unit 310 maintains the area where the color of the original 2D face image exists as much as possible. This is to allow the color of the hole area to be filled.

Meanwhile, the UV map flip unit 320 _{inverts the UV map (I ref} ) applied from the UV map acquisition unit 200 left and right to obtain a flip UV map (I _{flip ).} In general, a person's face shape or texture has left-right symmetry. Therefore, if there is an image for the left or right half of the subject's face from the 2D face image based on the front, the color harmful to the other half can be easily estimated. Accordingly, the UV map flip unit 320 _{obtains a flip UV map (I flip ) in which the UV map (I ref} ) is inverted left and right, and the UV map completion unit 400 completes the UV map based on the _{input UV map (I in ).} When generating the map (I _out ), it is possible to estimate the hole area specified by the mask (M) very accurately.

The UV map combining unit 330 includes the UV map I _ref applied from the UV map acquiring unit 200 , the mask M acquired from the mask acquiring unit 310 , and the flip acquired from the UV map flip unit 320 . By combining the UV map (I _flip ) in a predetermined manner, the input UV map (I _in ) is obtained, and the obtained input UV map (I _in ) is transmitted to the UV map completion unit ( 400 ).

The UV map combining unit 330, for example, simply concatenates the _{UV map (I ref} ) and the mask (M) and the flip UV map (I _{flip ), or elementally multiplies the UV map (I ref} ) and the mask (M). After the calculation, the input UV map I _{in may be obtained by combining the flip UV map I flip} , and in addition, the input UV map I _in may be obtained by combining in another predetermined manner.

The UV map completion unit 400 receives the input UV map (I _in ) and estimates the color of the hole area of the _{UV map (I ref} ) from the pattern of the applied input UV map (I _{in ) and fills the completed UV map.} (I _out ) is obtained. Here, the UV map completion unit 400 may estimate the color of the hole region of the _{input UV map I in} from the pattern of the input UV map _{I in} according to a pattern estimation method that is implemented as an artificial neural network and learned in advance.

In FIG. 3 , the UV map completion unit 400 implemented as an artificial neural network is expressed as a face UV generation network (G). And, as shown in FIG. 3, the face UV generation network G is composed of an auto-encoder structure including an encoder (en) and a decoder (de) including a plurality of layers of a structure symmetric to each other. can be Each of the plurality of layers of the encoder (en) obtains a feature map by stepwise extracting features _{of the input UV map (I in ).} Here, the number of layers included in the encoder en may be variously set. And each of the plurality of layers of the decoder (de) obtains the complete UV map (I _out ) by upsampling the feature map obtained from the encoder (en) step by step. Here, each of the plurality of layers of the decoder (de) gradually fills the hole region with color by referring to the flip UV map (I _{flip ) and the adjacent region in which the UV map (I ref ) is inverted left and right in the upsampling process,} It is possible to obtain _{a complete UV map (I out} ) in which the color of the region reflects the symmetry of the human face and changes naturally from the adjacent region.

At this time, the layers corresponding to each other among the plurality of layers of the encoder (en) and the decoder (de) are connected through a skip connection as shown in FIG. 3 to form a one-scale image of the _{UV map (I ref )} Information can be preserved so that the complete UV map (I _out ) can be acquired as a high-quality image.

_{The UV map completion unit 400 obtains the reconstruction loss (L Grec} ) in the UV map reconstruction process in which the input UV map (I _in ) is received and the completed UV map (I _out ) is obtained from the UV map acquisition unit 200 . It can be defined as the pixel-unit L1-norm loss of the effective area obtained from the two-dimensional image in which the hole area is excluded by the mask M from the UV map I _{ref , and can be calculated according to Equation 2} .

Where ⊙ denotes element-by-element multiplication, N _Iref is UV map as the number of elements in (I _ref), UV map (I _ref) the height (H), width (W) and a channel (C) represents the product of the size.

The UV map determination unit 500 receives the completed UV map (I _out ) obtained from the UV map completion unit 400 , divides the applied complete UV map (I _out ) in the vertical direction, and estimates the pattern learned in advance According to the method, it is determined whether the divided complete UV map is a real image obtained from a 2D face image or a fake image supplemented by the UV map completion unit 400 . At the same time, the UV map determination unit 500 receives the mask M obtained from the mask acquisition unit 310 _{and divides it in the vertical direction like the completed UV map (I out} ), and uses the divided mask to complete the divided By separating the actual real image and the fake image in the UV map, it is possible to determine whether the real image and the fake image discrimination result for the divided complete UV map are accurate.

The UV map determination unit 500 may include a UV map division unit 510 , a mask division unit 520 , a valid image determination unit 530 , a competitive determination unit 540 , and a loss back propagation unit 550 .

The UV map division unit 510 receives the completed UV map I _{out from the UV map completion unit 400 , divides it into two left} and right sides, and divides it into a left image I left and a right image I _right . The mask dividing unit 520 receives the mask M obtained from the mask acquiring unit 310, divides the left _{and right into two, and divides them into a left mask (M left} ) and a right mask (M _right ) and outputs them.

_{The effective image determining unit 530 analyzes the left mask (M left} ) and the right mask (M _right ) applied from the mask dividing unit 520 , and a real face among the left image (I _left ) and the right image (I _{right )} The image acquisition unit 100 determines an actual image corresponding to the generated UV map based on the acquired 2D face image. As an example, the effective image determining unit 530 may _{calculate the sum of pixel values of each of the left mask M left} and the right mask M _right , and determine an actual real image according to the sum of the calculated pixel values. Here, since it is assumed that the mask acquiring unit 310 creates the mask M by filling the pixel values of the hole region with 0 and the pixel values of the remaining regions with 1, the left mask M _left and the right mask M _right ) It can be seen that the one with the larger sum of pixel values represents the actual image. Accordingly, the effective image determination unit 530 loses which image is the real image among _{the left image (I left} ) and the right image (I _right ) divided by the UV map division unit 510 and transmitted to the competitive determination unit 540 . It may notify the back propagation unit 550 .

As shown in FIG. 3 , the competitive determination unit 540 is implemented as an artificial neural network and competition learning is performed for the pattern estimation method together with the UV map completion unit 400 , and the left image (I _left ) and the right image (I _right ) to estimate which image is the real image.

And the loss back propagation unit 550 compares the result of estimating the real image in the competitive determination unit 540 and the real image notified by the valid image determination unit 530 _{to calculate the generation loss (LG} ), and calculate the calculated generation By backpropagating the loss ( _LG ), the UV map completion unit 400 is trained. At this time, learning is performed together with the competitive determination unit 540 and the UV map completion unit 400 .

_{Here, the determination loss L D} by the competitive determination unit 540 implemented as an artificial neural network may be calculated as in Equation (3).

는 오차 함수(Error function)이고, x_uv 는 좌측 이미지(I_left)와 우측 이미지(I_right) 중 유효 이미지 판별부(530)에서 리얼 이미지를 나타내고, x_z 는 UV 맵 완성부(400)에서 추정 보완된 페이크 이미지이며, D(x_uv)와 D(x_z) 는 각각 리얼 이미지(x_uv)와 페이크 이미지(x_z)에 대해 경쟁적 판정부(540)가 추정한 판정 결과를 나타낸다.here

is an error function, x _uv represents a real image in the effective image determining unit 530 among the left image (I _left ) and the right image (I _{right ), and x z} is the UV map completion unit 400 It is a fake image that has been estimated and supplemented, and D(x _uv ) and D(x _z ) represent determination results estimated by the competitive determination unit 540 with respect to the real image (x _uv ) and the fake image (x _{z ), respectively.}

And the hostile loss according to the competition learning of the UV map completion unit 400 and the competitive determination unit 540 may be calculated by Equation (4).

The generation loss (L _G ) generated to generate the complete UV map (I _out ) from the UV map (I _ref ) is the sum of the reconstruction loss (L _Grec ) of Equation 2 and the hostile loss (L _{Gadv ) of Equation 4} It can be defined as Equation 5.

Here, λ _G is a weight for maintaining a balance between the reconstruction loss (L _Grec ) and the adversarial loss (L _Gadv ), and may be set to, for example, 0.001.

The hostile loss (L _Gadv ) by learning the encoder (en) of the UV map completion unit 400, the UV map completion unit 400 is a fake image (x _z ) in which the hole region is large as well as the real image (x _{uv )} ) to obtain a higher quality UV map.

Table 1 shows that the UV map determination unit 500 _{divides the completed UV map (I out} ) into two _left and right images (I left ) and a right image (I _right ), and divides the mask (M) into two left and right to the left An example of an algorithm for classifying a real image (x _uv ) and a fake image (x _z ) by dividing into a mask (M _left ) and a right mask (M _{right ) is shown.}

The above-described UV map determination unit 500 is configured to configure a UV map completion unit 400 and a generative adversarial network to learn in a competitive learning method together with the UV map completion unit 400, It is provided only during learning, and may be removed after the face image color synthesizing apparatus completes learning.

4 shows an example of a 3D face image using a UV map completed by the face image color synthesizing apparatus according to this embodiment, and FIG. The result compared with the UV map is shown.

4, (a) shows the 2D face image acquired by the 2D image acquisition unit 100, (b) shows the UV map (I _ref ) acquired by the UV map acquisition unit 200, (c) is _{The complete UV map (I out} ) synthesized by the face image color synthesizing apparatus of this embodiment is shown, and (d) to (h) are the 3D face reconstructed by synthesizing the 3D face model using the _{completed UV map (I out )} It represents an image captured from multiple angles.

And in FIG. 5, (a) shows the input UV map (I _ref ), (b) shows the separately acquired ground truth UV map, (c) is learned by reflecting only left-right symmetric reconstruction loss represents the completed UV map. And (d) shows a case in which it is learned using the effective area of the UV map (I _{ref ) rather than the completed UV map (I out} ) obtained in the UV map completion unit 400, (e) is in this embodiment It shows the completed UV map (I _out ) learned according to it.

4 and 5 , the face image color synthesizing apparatus according to this embodiment _{estimates the hole area of the UV map (I ref} ) based on the left-right symmetry of the human face to obtain the completed UV map (I _out ). By generating and dividing the completed UV map (I _out ) into left and right to perform self-discrimination-based competitive learning, it is possible to complete a UV map having a natural facial texture without a 2D image or UV map that is separately acquired for learning.

6 is a diagram illustrating a face image color synthesis method according to an embodiment of the present invention.

Referring to FIGS. 1 to 3 , the face image color synthesis method of FIG. 6 is described. First, a UV map I _ref is obtained ( S10 ). Here, the UV map (I _ref ) is a UV map including a hole area obtained from a 2D face image obtained from a photo or a video, etc. in a predetermined manner, and aligns the 3D face model to the 2D face image, and aligns the 2D face image. It can be obtained by mapping and developing a 3D face model. _{Various methods for obtaining a UV map (I ref} ) including holes from a 2D face image obtained from a photo or a video have been disclosed, and it is assumed here that one of the disclosed techniques is used.

When the UV map (I _ref ) is obtained, the obtained UV map (I _ref ) is inverted left and right _{to obtain a flip UV map (I flip} ), and the UV map (I _ref ) separates the hole area from the other areas to mask (M) is obtained (S20). Here, the mask M is a binary mask filled with a pixel value of 0 or 1 depending on whether it is a hole region.

When the flip UV map (I _flip ) and the mask (M) are obtained, the obtained flip UV map (I _flip ) and the mask (M) are _{combined with the UV map (I ref} ) in a predetermined manner to combine the input UV map (I _{in )} ) is generated (S30).

And according to the pre-learned pattern estimation method, the pattern of the input UV map (I _in ) is estimated, the color of the hole area of the UV map (I _ref ) is estimated and filled, and natural texture change is also observed for areas other than the hole area The completed UV map (I _out ) is estimated by supplementing it so that it occurs (S40). Since the input UV map (I _in ) is created by combining not only the UV map (I _ref ) but also the flip UV map (I _flip ) and the mask (M), the completed UV map (I _out ) is a pattern according to the left-right symmetry of the human face. _{The hole area can be estimated and filled with , and a complete UV map (I out} ) can be obtained by supplementing the texture for areas other than the hole area so that the change from the estimated hole area is made naturally.

On the other hand, when the complete UV map (I _out ) is obtained, it is determined whether the current state is a learning state (S50). If it is determined that it is a learning state, the completed UV map (I _out ) and the mask (M) are divided into two _left and right, respectively, and the left image (I left ) and the right image (I _right ) and the left mask (M _left ) and the right mask (M _right ) is obtained (S60).

And the real image and the fake image are estimated for _{the left image (I left} ) and the right image (I _right ) obtained according to the learned pattern estimation method _{, and together with the left mask (M left} ) and the right mask (M _right ) An actual real image and a fake image are determined based on the pixel value (S70). _{Thereafter, a generation loss (L G} ) is calculated by comparing the estimated real image and the fake image with the determined real image and the fake image, and learning is performed by backpropagating the calculated generation loss (L _{G ) ( S80 ).}

Here, the generation loss (L _{G ) is the reconstruction loss (L Grec} ) generated in the process of estimating the complete UV map (I _out ) from the UV map (I _ref ) and the hostile loss generated in the process of estimating the real image and the fake image It can be calculated as the sum of (L _{Gadv ).}

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is only exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: face image acquisition unit 200: UV map acquisition unit
300: input preprocessor 310: mask acquisition unit
320: UV map flip unit 330: UV map coupling unit
400: UV map completion unit 500: UV map determination unit
510: UV map division 520: mask division
530: valid image determination unit 540: competitive determination unit
550: loss back propagation unit

Claims (17)

A UV map obtained from a 2D face image is applied in a predetermined manner, and a mask that separates the hole area and the remaining area of the applied UV map and a flip UV map in which the UV map is inverted left and right are combined with the UV map to input UV an input preprocessor to obtain a map;
a UV map completion unit that is implemented as an artificial neural network to receive an input UV map and generate a complete UV map filled with the color of the hole region of the UV map from the pattern of the applied input UV map according to a pre-learned pattern estimation method; and
It is provided at the time of learning, receives the completed UV map and the mask, and divides each into two left and right, and estimates a real image and a fake image among the two completed UV maps divided according to the learned pattern estimation method, while the divided 2 A face image color synthesizing device including a UV map determining unit that calculates and backpropagates a generation loss for a completed UV map generated by discriminating an actual real image and a fake image based on the mask. The method of claim 1, wherein the input preprocessor
a mask obtaining unit that analyzes the UV map to classify a hole area and a remaining area, and allocates different bit values to each pixel of the divided hole area and the remaining area to generate the mask composed of binary bit values;
a UV map flip unit to obtain the flip UV map by inverting the UV map left and right; and
and a UV map combiner configured to combine the UV map, the mask, and the flip UV map in a predetermined manner to obtain the input UV map. According to claim 1, wherein the UV map completion unit
an encoder that is composed of a plurality of layers and extracts the features of the input UV map step by step according to a pre-learned pattern estimation method to obtain a feature map; and
and a decoder configured to include a number of layers corresponding to a plurality of layers of the encoder and to obtain the complete UV map by upsampling the feature map in stages. 4. The method of claim 3, wherein the decoder
A face image color synthesizing apparatus in which each of a plurality of layers is connected to a corresponding one of the plurality of layers of the encoder through a skip connection. The method of claim 3, wherein the UV map determination unit
a UV map division unit that receives the completed UV map and divides the left and right into two to obtain a left image and a right image;
a competitive determination unit implemented as an artificial neural network to estimate a real image corresponding to a portion having a small hole region in the UV map among the left image and the right image and a fake image corresponding to the remaining portion; and
and a loss backpropagation unit for calculating the generation loss based on the estimation result of the competitive determination unit and backpropagating the calculated generation loss. The method of claim 5, wherein the UV map determination unit
a mask dividing unit that receives the mask and divides it into two left and right to obtain a left mask and a right mask;
and an effective image determining unit for discriminating an actual real image and a fake image of the UV map based on the sum of the pixel values of the left mask and the right mask, and notifying the loss back propagation unit. The method of claim 6, wherein the loss back propagation unit
_{Reconstruction loss (L Grec} ) in the UV map reconstruction process to obtain the complete UV map (I _out ) by receiving the input UV map (I _in ) and real images and fake images for the left image and the right image A face image color synthesizing apparatus that _{backpropagates} by calculating the generation loss (L _G ) as the sum of the adversarial loss (L Gadv ) based on the estimated decision loss (L _{D ).} The method of claim 7, wherein the loss back propagation unit
Equation for the reconstruction loss (L _{Grec )}

(Where ⊙ denotes element-by-element multiplication, N _Iref is UV map (I a number of elements in _ref), UV map (I _ref) the height (H), width (W) and a channel (C) represents the product of the size. )
Calculated according to the equation, the adversarial loss (L _{Gadv )}

(here

is an error function, x _z is a fake image determined using the left mask and the right mask, and D represents the estimated determination result.)
Calculated according to the formula, the generation loss (L _G )

(Where λ _G is the weight to maintain a balance between the reconstruction loss (L _Grec ) and the adversarial loss (L _{Gadv ).)}
A face image color synthesizing device that calculates according to A UV map obtained from a 2D face image is applied in a predetermined manner, and a mask that separates the hole area and the remaining area of the applied UV map and a flip UV map in which the UV map is inverted left and right are combined with the UV map to input UV obtaining a map;
receiving an input UV map using an artificial neural network and generating a complete UV map filled with the color of the hole region of the UV map from the applied input UV map pattern according to a pre-learned pattern estimation method; and
In the learning state, the completed UV map and the mask are applied and divided into left and right, respectively, and the real image and the fake image are estimated among the two completed UV maps divided according to the learned pattern estimation method, while the divided two masks A method for synthesizing a face image color, comprising: calculating and backpropagating a generation loss for a completed UV map generated by discriminating an actual real image and a fake image based on 10. The method of claim 9, wherein obtaining the input UV map comprises:
generating the mask composed of binary bit values by analyzing the UV map to classify a hole area and a remaining area, and assigning different bit values to each pixel of the divided hole area and the remaining area;
obtaining the flip UV map by inverting the UV map left and right; and
combining the UV map and the mask and the flip UV map in a predetermined manner to obtain the input UV map. 10. The method of claim 9, wherein generating the complete UV map comprises:
an encoder for obtaining a feature map by stepwise extracting features of the input UV map using an encoder that is composed of a plurality of layers and has learned a pattern estimation method; and
and step-by-step upsampling of the feature map using a decoder having a number of layers corresponding to a plurality of layers of the encoder. 12. The method of claim 11, wherein the upsampling comprises:
A face image color synthesis method in which a feature map output from a corresponding layer through a skip connection connected to a corresponding layer among a plurality of layers of the encoder is applied together with a feature map upsampled from a previous layer among a plurality of layers of the decoder. The method of claim 12, wherein the performing of the learning comprises:
obtaining a left image and a right image by receiving the completed UV map and dividing the left and right into two;
estimating a real image corresponding to a portion having a small hole region in the UV map and a fake image corresponding to the remaining portion of the left image and the right image using an artificial neural network; and
and calculating the generation loss based on an estimation result, and backpropagating the calculated generation loss. 14. The method of claim 13, wherein performing the learning comprises:
obtaining a left mask and a right mask by receiving the mask and dividing the left and right into two;
and determining an actual real image and a fake image of the UV map based on the sum of pixel values of each of the left mask and the right mask. 15. The method of claim 14, wherein backpropagating the production loss comprises:
_{Reconstruction loss (L Grec} ) in the UV map reconstruction process to obtain the complete UV map (I _out ) by receiving the input UV map (I _in ) and real images and fake images for the left image and the right image A face image color synthesis method for calculating the generation loss (L _G ) as the sum of the adversarial loss (L _Gadv ) based on the estimated decision loss (L _{D ).} 16. The method of claim 15, wherein backpropagating the production loss comprises:
Equation for the reconstruction loss (L _{Grec )}

(Where ⊙ denotes element-by-element multiplication, N _Iref is UV map (I a number of elements in _ref), UV map (I _ref) the height (H), width (W) and a channel (C) represents the product of the size. )
calculating according to;
The hostile loss (L _Gadv ) is expressed by the equation

(here

is an error function, x _z is a fake image determined using the left mask and the right mask, and D represents the estimated determination result.)
calculating according to; and
The generation loss (L _G ) is expressed by the equation

(Where λ _G is the weight to maintain a balance between the reconstruction loss (L _Grec ) and the adversarial loss (L _{Gadv ).)}
A method for synthesizing face image color, comprising the step of calculating according to 17. A recording medium in which program instructions for performing the face image color synthesis method according to any one of claims 9 to 16 are recorded.

Images