KR19990031324A - Texture mapping method for lip animation - Google Patents

Abstract

The present invention relates to a texture mapping method for lip shape animation in a facial expression change coding system using a human shape model. In the related art, in order to texture map a lips part, . However, in the method of the present invention, the approximated triangles are formed from the normalized texture image lips and the normalized standard shape model lips, and the coordinate values of the candidate feature points on the normalized texture image lips are obtained by the line-proportional relationship of the approximated triangles . Therefore, since the feature points on all the texture image lips can be obtained by a simple method, the calculation time can be shortened.

Description

Texture mapping method for lip animation

The present invention relates to facial expression change coding of a three-dimensional model-based coding system, and more particularly to a lip texture mapping method for lip animation.

Application techniques using a three-dimensional object model, in particular a human shape model, have been applied in the fields of video coding, virtual reality and computer graphics. In the ongoing video coding field, research is continuing to transmit satisfactory image quality with a smaller amount of data. The study on the amount of data to be transmitted is as follows. In MPEG4, since a video signal is transmitted using a very small amount of data, not only a conversion using information theory, for example, a cosine transform coefficient method, but also other computer vision, Techniques used in computer graphics are also applied.

Typically, the shape of a person seen in a videophone or video conference is only the chest, which is above the waist. In particular, the face of the speaker is responsible for the majority of video information transmission. If we can send only the characteristic regularized information about the face type, ignoring the information other than the human face, that is, the background information, the transmission amount can be greatly reduced. In other words, the rules appearing on the face of the common heel, noh, cloak, and rock emotional expressions and the mouth shape of the speech are found and sent to the opponent with one informed code, and the opponent receives the three- By displaying the face shape according to the received code, it is possible to grasp the expression form of the sender with a small amount of data.

More specifically, a three-dimensional model-based coding system is a coding object in which two persons talk in real time in a field like a videophone and transmit not only a sound but also a changing face or body in real time. However, in contrast to general video coding, transmission data does not deal with an image composed of pixels that vary instantaneously. Instead, it extracts a specific motion variable, transmits the data, Dimensional head shape model, body shape model, and the like, and reconstructs the final changed face or body image of the sender.

Of course, in such a system configuration, the amount of data to be transmitted is very limited by the data transmission bandwidth, so that a large amount of data can not be sent, but it can be used when sufficient image quality is required for accurate recognition between both parties. If the bandwidth is big enough to send enough data, the advantage of this system will be less. This is because the regularized code has a lack of a natural expression.

In this 3D model based coding system, mouth shape, especially lip shape, has very important visual information among mutually transmitted face images of the sender. In other words, what the sender is talking about and how he expresses his emotions can be measured in the form of lips.

Lip shape expression animation is a method of synthesizing lip shapes of various shapes from one lips shape. In this method, a standard shape model lips composed of a three-dimensional wire frame is matched to the lips of the sender itself and a texture mapping . The texture mapping must obtain all the coordinate values of the texture image lip and superimpose the corresponding standard shape model lip with this coordinate value.

However, the coordinate values of the standard shape model lips are already known in the wire frame construction, but the coordinate values of the texture image lips are not known except for some feature points. Therefore, there is a problem that it takes a lot of time to obtain all the coordinate values of the remaining minutiae.

Therefore, it is an object of the present invention to provide a lip texture mapping method for lip animation in a three-dimensional model-based coding system.

It is another object of the present invention to provide an improved texture mapping method capable of finding coordinate values of a texture image lip in a simple manner for lip texture mapping in a three-dimensional model-based coding system.

According to another aspect of the present invention, there is provided a texture mapping method for lip shape animation in a facial expression change coding system using a human shape model, comprising: acquiring a texture image of a predetermined size of a sender face; Extracting a lip portion from a standard shape model of the wire frame; Matching the standard shape model lips to overlap the texture image lips; And performing texture mapping by projecting the texture data of the texture image lip onto the matched standard shape model lip.

1 is a schematic block diagram of a texture mapping apparatus for lip animation of a three-dimensional model-based coding system constructed in accordance with a preferred embodiment of the present invention;

2 is a diagram illustrating a wireframe standard shape face model provided for texture mapping,

3 is a diagram illustrating a texture face image provided for texture mapping,

4 is a diagram illustrating a process of superimposing and matching a lip of a standard shape face model and a texture face image,

FIGS. 5A, 5B, 5C, 5D and 5E are diagrams for explaining the process of obtaining coordinates of candidate feature points of a texture image lip for texture mapping according to the present invention,

6 is a diagram illustrating a lip texture mapping result according to the present invention;

≪ Explanation of symbols for main parts >

10: standard shape model providing unit 20: standard shape model lips extracting unit

30: texture image acquisition unit 40: texture mapping unit

These and other objects and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, a schematic block diagram of a lip texture mapping apparatus for lip animation in a facial expression change coding system using a human shape model according to the present invention is shown. The lip image texture mapping apparatus of the present invention includes a standard shape model providing unit 10, a standard shape model lip extracting unit 20, a face texture image obtaining unit 30, and a texture mapping unit 40.

The standard shape model supplier 10 has a three-dimensional hair model, and the three-dimensional hair model 100 includes a plurality of units configured using a silicon graphics interface (SGI) workstation as illustrated in FIG. Dimensional computer graphics format called a wire frame having the form of a network in which polygons of a polygon are connected. The three-dimensional hair model 100 is provided to the standard shape model lip extracting unit 20.

The standard shape model lip extracting unit 20 extracts only the lip part 110 from the three-dimensional head model of the wire frame received from the standard shape model providing unit 10 and extracts the extracted lip part 110 from the texture mapping unit 40).

On the other hand, the face texture image acquisition unit 30 extracts the actual two-dimensional face texture image of the sender by separating the face image of the sender provided from the camera (not shown) into areas having similar pixel values using a sementation method do. The extracted face texture image is obtained by extracting edge portions of a face image using an edge detection method using brightness distribution of an original image in which the brightness of the image is rapidly changed, extracting a contour representing the boundary of the extracted edges, It stores content, which is information about the inside. The texture image acquired by the face texture image acquisition unit 30 is provided to the texture mapping unit 40. [

The texture mapping unit 40 performs texture mapping for projecting the texture data of the lip 210 of the texture image onto the standard shape model lip 110. In order to perform the texture mapping, the texture image lip 210 and the standard shape model lip 110 must be matched to overlap each other. Accordingly, the texture mapping unit 40 forms a window 200 of 512 * 512 size as shown in FIG. 3 for the texture image provided from the mapping face texture image acquisition unit 30, The lip 110 of the standard shape model is synthesized.

4 is a diagram illustrating a result of combining a window 200 of a texture face image and a lip 110 of a standard shape model. The lip 110 of the standard shape model is a lip 210 of the texture face image 200, Are not locally coincident with each other. Accordingly, the texture mapping unit 40 may include a standard shape model 100 as described below to allow the lips 110 of the standard shape model 100 to overlap and match the lips 210 of the texture face image 200, The lip portion 110 of the standard lip model 110 is moved by the offset value of the texture lip portion 210 with respect to the lip portion of the standard lip model 210 to perform the scaling to overlap the edge points of the two lips.

5A and 5B, a standard shape model lip 110 and a texture image lip 210 are shown, respectively. For texture mapping, the minutiae of the texture lip 210 P ₁ ' , P ₂ ' ,. . . , P _n ' ) To the corresponding vertices of the standard shape model lips ( P ₁ , P ₂ ,. . . , P _n ). However, the coordinate values of the respective vertices of the three-dimensional standard shape model are already known when constructing the standard shape model of the entire face, but in the texture coding, the coordinate values of all the feature points are not known, Only the minutiae coordinate values of the minutiae are known. Therefore, for the overall texture mapping, the remaining coordinate values of the texture image should be obtained. Since the vertices of the standard shape model lips 110 have three-dimensional coordinate values (x, y, z), the coordinate values on all z-axes are set to " 0 " in order to be displayed on the two- . Then, the feature points of the two-dimensional texture lips 210 corresponding to the vertices of the standard shape model lips 110 P _n ' ( s _n , t _n ).

Therefore, the images of the standard shape model lip 110 and the texture image lip 210 should be normalized to the lengths of the lips of the standard shape model and the texture image, respectively. More specifically, in the standard shape model lip 110, the center point of the lip portion ( P ₀ ) To the right lip end point ( P _n ) And the length ( D ₁ = P _n - P ₀ ) And the left lip end point P ₁ ) And the length ( D ₂ = P ₀ - P _n ), And the maximum length of the two lengths ( D = max (D ₁ , D ₂ ) ). Next, the standard shape model lip 110 is normalized to the maximum length D as shown in the following equation (1).

In the above Equation 1, x _new and y _new are the normalized coordinate values of the feature points P existing in the horizontal direction of the standard shape model lip 110, x _old and y _old are the coordinate values before the normalization, x ₀ and y ₀ represent the coordinate values of the center point (P ₀ ), respectively.

Similarly, the maximum length D 'is obtained for the texture image lip 210, and the texture image lip 210 is normalized to the maximum length D' as shown in the following equation (2).

In the above Equation 2, s _new and t _new are normalized coordinate values of the feature point P 'existing in the horizontal direction of the texture image lip 210, s _new and t _new are coordinate values before normalization, s ₀ and t ₀ represent coordinate values of the center point (P ₀ '), respectively. FIG. 5C illustrates a normal shape model lips 110 and a texture image lips 210 in a normalized state.

Next, each vertex (P ₁ , P ₂ , ..., P ₂ ) of the standard shape model lip 110 is dimensioned so that the normalized standard shape model lip 110 overlaps the corresponding feature points of the _two- P _n ) to the coordinate values of the corresponding minutiae points P ₁ ', P ₂ ', ..., P _n 'of the two-dimensional texture lip 210. Accordingly, the feature points of the texturing lip _{(210) (P 1 ',} P 2', P 4 ') coordinate value because it already knows the respective vertices of a standard shape model lips 110 corresponding to their corresponding (P ₁ of P ₂ , P ₄ ), but since the coordinate values of the remaining candidate feature points of the texture lip 210, for example, the vertices P ₃ 'and P ₅ ', are unknown, It will not be possible to move the corresponding vertices P ₃ , P ₅ .

According to the present invention, the coordinate values of the remaining vertices (P ₃ ', P ₅ ') of the texture image lip 210 are obtained by using the triangle approximation approach described below.

First, to the vertex (P ₃ ') (a to the vertex a feature point from the apex, for example (P _2'), the texture image lips one end, for example the left end point of the (210) P ₁₎ to obtain the coordinate values of the ' The line extending is the first side 310 and a vertex corresponding to the feature point P ₂ 'of the texture image lip 210 from one end point of the standard shape model lip 110, for example, the left end point P ₁ P ₂ ) as the second side 320 and a third side 330 extending from the apex P _{2 to the characteristic} point P ₂ '(see FIG. 5D) ). In the approximated triangle 300, from the apex P ₃ on the second side 320 of the standard shape model lips 110 to the corresponding apex P ₃ 'on the first side 310 of the texture lips 210 Displacement amount ΔP Is proportional to the amount of displacement reaching the apex P ₂ 'from the apex P ₂ on the third side 330. At this time, the displacement amount from the vertex P ₃ of the standard shape model lip 110 to the corresponding vertex P ₃ 'of the texture image lip 210 ΔP ) Is the line segment ), And the line segment ( ) Is the line segment of the third side ), The following equations (3) and (4) can be derived.

Therefore, the remaining vertices P ₅ 'of the texture image lip 210 are the coordinate values of the feature points P ₄ ', P _n 'and the corresponding vertices P ₄ , P _n of the standard shape model lip 110 It can be obtained by using the above-mentioned Equations (3) and (4). FIG. 5E illustrates a result of matching the standard shape model lip 110 to the texture image lip 210 using the above-described process.

The standard shape model lips 110 matched in FIG. 5E are restored on the window using the inverse transformation equation of Equation (1) as illustrated in Equation (5) below.

Similarly, the texture image lips 210 matched in FIG. 5E are restored to texture coordinate values using the inverse transform equation of Equation 2 as illustrated in Equation 6 below.

As a result, as shown in FIG. 6, a polygon of the texture image lip 210 having a given texture coordinate value on the window is subjected to a natural texture mapping to the polygon of the standard shape model lip 110 corresponding to the polygon.

As described above, in the three-dimensional model-based coding system using the human shape model, the coordinate values of the texture image lips are more easily obtained by using the triangle approximation approach in texture mapping for lip animation, You can extend it to natural texture mapping.

Claims (5)

A texture mapping method for lip shape animation in a facial expression change coding system using a human shape model, Acquiring a texture image of a predetermined size of a sender's face; Extracting a lip portion from a standard shape model of the wire frame; Matching the standard shape model lips to overlap the texture image lips; And projecting texture data of the texture image lip to the matched standard shape model lip to perform texture mapping. 2. The method of claim 1, wherein the matching comprises: Normalizing the standard shape model lip and the texture image lip to a width of a length based on each lip end point; Moving the vertices of the normalized standard shape model lip corresponding to the feature points of the normalized texture image lip to the coordinates of the minutiae points of the normalized texture image lip; Obtaining texture coordinates of candidate feature points between feature points of the normalized texture image lip; Moving the vertices of the normalized standard shape model lips corresponding to the candidate feature points to the coordinates of candidate feature points having the obtained texture coordinates, and matching the vertices of the normalized standard shape model lips to the coordinates of the candidate feature points having the obtained texture coordinates. 3. The method of claim 2, wherein normalizing the standard shape model lips comprises: The center point of the standard shape model lips P ₀ Calculating a P _n) -) a first length _{(D 1 = P n -P 0} ) and the second length (D ₂ = P ₀ of lips and the left side end point (P ₁₎ of the lip and the right end point (P _n) from ; Taking a maximum length D = max (D ₁ , D ₂ ) of the first and second lengths; Obtaining new coordinates in the horizontal direction of the standard shape model lip using the following equation 1, Equation 1 In the above Equation 1, x _new and y _new are the normalized coordinate values of the vertices P existing in the horizontal direction of the standard shape model lip, x _old and y _old are the coordinate values before the normalization, x ₀ Wow y ₀ Each representing a coordinate value of the center point (P ₀ ); Wherein the texture image normalizing step comprises: The first length D ₁ 'and the second length D ₂ ' between the right lip end point P _n 'and the left lip end point P _{1 are obtained} from the center point P ₀ ' of the texture image lip step; Taking the maximum length D '= max (D ₁ ', D ₂ ') of the first and second lengths; Obtaining new coordinates in the horizontal direction of the texture image lip using the following equation (2) Equation 2 In Equation (2), s _new and t _new are the feature points existing in the horizontal direction of the texture image lip P ' ), S _new and t _new are coordinate values before normalization, and s ₀ and t ₀ have respective coordinate values of the center point (P ₀ '). / RTI > 3. The method of claim 2, wherein obtaining the texture coordinates of the candidate feature points comprises: The normalized texture the end point of the image lip (P ₁ ') a to the vertex from the vertex and a line extending to the characteristic point a through the candidate feature point in the first side, from the end point of the normalized standard shape model lips Constructing an approximated triangle having a second side extending to a vertex corresponding to the feature point of the normalized texture image lip and a third side extending from the vertex to the feature point; Wherein when the second side is shifted to be overlapped with the first side and matched, a proportional expression of the amount of displacement of the corresponding side of the second diagonal phase shifted to the candidate specific point of the first diagonal phase and the amount of displacement of the third side is used And obtaining the coordinates of the candidate feature point. 3. The method of claim 2, wherein the texture mapping comprises: Restoring the matched standard shape model lip using Equation (3) Equation 3 Restoring the matched texture image lips into texture coordinate values using Equation (4) Equation 4 Wherein the texture mapping method comprises: