KR100278065B1 - Apparatus for composing a lip shape in a 3 dimension model-based coding system - Google Patents

Abstract

The present invention relates to a modified lip synthesizing apparatus for a facial expression change coding system using a human shape model. The apparatus of the present invention includes a basic lip providing unit for providing a basic lip shape of a sender, x and means for extracting a parameter indicating the degree of change of the actual lip shape points deformed in the y-axis direction, and a lip deformation function expressing the degree of deformation, and applying the parameter value to the function to perform the basic Means for calculating a position moved from the lip shape, and means for synthesizing the modified actual lip shape from the basic lip shape according to the calculated position.

Description

Deformation lip synthesis apparatus of three-dimensional model-based coding system {APPARATUS FOR COMPOSING A LIP SHAPE IN A 3 DIMENSION MODEL-BASED CODING SYSTEM}

The present invention relates to a facial expression change coding system using a human shape model, and more particularly, to a device for synthesizing a new lip shape by finding a deformation position of each edge point on a lip using a specific lip deformation function. .

Application techniques using three-dimensional object models, especially human shape models, have been applied in the fields of video coding, virtual reality and computer graphics. In the field of video coding currently underway, research to transmit a satisfactory image quality with a smaller amount of data continues. Looking at the research projects according to the amount of data to be transmitted, there is MPEG4 which is being actively studied starting from HDTV. In MPEG4, a video signal is transmitted using a very small amount of data, so that not only the transformation using information theory such as cosine transform coefficient method but also other computer vision and computer graphics The techniques used may also be applied.

In general, the shape of a person seen in a videophone or video conference only shows the chest above the waist. In particular, the speaker's face takes up most of the video information transmission. If only non-human face information, ie background information, can be ignored and only regular information on the face shape can be sent, the transmission rate can be greatly reduced. In other words, it finds the rules of the general face, face, expression, and emotions of the face and mouth when speaking, and sends them to the other party through one code. By deforming the face shape according to the received code, the expression shape of the sender may be recognized with a small amount of data.

More specifically, the three-dimensional model-based coding system targets coding in which two people talk in real time, such as a video phone, and transmit not only sound but also a changing face or body in real time. However, unlike general video coding, the transmitted data does not handle an image composed of pixels that change every moment. Instead, it extracts and transmits a certain motion variable, and then has the data on the receiving side, that is, the face of the sender. It is combined with the dimensional head shape model, body shape model, etc. to form the face or body image of the final changer after reconstruction.

Of course, such a system configuration can be used when the amount of data to be transmitted is very limited by the data transmission bandwidth, but not a large amount, but requires sufficient image quality for accurate recognition between the two parties. If the bandwidth is large enough to send enough data, the system's advantages will be lower. This is because regular code lacks a natural look.

In the 3D model-based coding system, the shape of the mouth, especially the shape of the lips, has a very important visual information in the face images of the senders. In other words, it is possible to estimate what the sender is saying and what emotions are expressed in the form of lips.

However, the lip shape expression method used conventionally is a method of synthesizing the lip shapes of different shapes from one lip shape, and simply expresses the lip shape through a complex process such as opening and closing the lips or a muscle model. In more detail, the existing lip shape expression method expresses various lip shapes by extracting various feature points of the lip boundary and moving these feature points by a predetermined amount according to the degree of movement. Therefore, since the synthesized mouth shape is simple and requires considerable calculation time, it is difficult to synthesize a face shape like a voice in real time.

It is therefore an object of the present invention to provide an apparatus for synthesizing deformed lips in a simpler manner in a three-dimensional model based coding system.

According to the present invention for achieving the above object, the modified lip synthesizing apparatus in the facial expression change coding system using a human shape model includes a basic lip providing unit for providing a basic lip shape of the sender itself; Means for extracting a parameter representing a degree of change of edge points of an actual lip shape deformed in the x and y axis directions from the basic lip shape; Means for calculating a position shifted from the basic lip shape by having a lip deformation function representing the degree of deformation and applying the parameter value to the lip deformation function; Means for synthesizing the modified actual lip shape from the basic lip shape according to the calculated movement position.

1 is a block diagram of a modified lip synthesis apparatus of a three-dimensional model based coding system constructed in accordance with a preferred embodiment of the present invention;

2a and 2b show the shape of a primary lip and a modified real lip, respectively;

3A, 3B and 3C are graphs illustrating a lip deformation function for each edge point on the lips moving in the x and y axis directions used in the movement position calculator shown in FIG.

<Description of the code | symbol about the principal part of drawing>

10: basic lips providing unit 20: image processing unit

30: displacement amount extraction unit 40: moving position calculation unit

50: Lip Shape Synthesis

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1, a block diagram of an apparatus for synthesizing a modified lip of a sender in a facial expression change coding system using a human shape model according to the present invention is shown. The modified lip synthesizing apparatus of the present invention includes a basic lip providing unit 10, an image processing unit 20, a displacement extracting unit 30, a moving position calculating unit 40, and a modified lip synthesizing unit 50.

The basic lip providing unit 10 has a basic lip form 100 of the sender as shown in FIG. 2A, which has a feature point of right and left lip edges a and b and an upper lip lower point. (c) and the lower lip upper point (d). Although the upper lip lower point (c) and the lower lip upper point (d) are shown apart in FIG. 2A, it will be understood that the feature points c and d are substantially stuck together as when the lips are closed.

The basic lip form 100 is provided by the displacement extracting unit 30 and the lip shape combining unit 50. In the displacement extracting unit 30, the basic lip 100 is used to extract a parameter representing the degree of movement deformed therefrom, and the basic lip 100 provided to the lip shape synthesizing section 50 is a new lip according to the degree of deformation therefrom. Used to synthesize forms.

The image processor 20 divides the lip image of the sender provided from the camera (not shown) into regions having similar pixel values by using a segmentation method, thereby displaying the actual lip image 200 of the sender as illustrated in FIG. 2B. ). In FIG. 2B, the modified real lip image 200 is shown to have feature points a ′, b ′, c ′, and d ′ modified from the base lip image 100. The extracted real lip image is extracted from the lip image using an edge detection method using the brightness distribution of the original image in which the brightness of the image changes drastically, and the contour representing the boundary of the extracted lip and the area of the region are extracted. The content, which is information about the inside, is provided to the displacement extracting unit 30.

The displacement extracting unit 30 measures the displacement of the actual lip 200 which is deformed according to the change in the shape of the lip generated by the sender's speech and emotion expression from the sender's basic lip 100 provided from the basic lip providing unit 10. The degree of lip movement represented is extracted and provided to the next position movement position calculator 40.

The movement position calculation unit 40 according to the present invention has a lip deformation function expressed by a specific equation as described in detail below, and uses the lip deformation function to determine the displacement amount provided from the displacement extraction unit 30. Calculate the deformed position of the lips

2A and 2B, if the positions of the feature points a, b, c, and d are deformed and moved to the positions of the feature points a ', b', c 'and d' in the basic lips 100, the movement amount The parameters indicating are the degree of the upper lip spreading in the x-axis or the left-right direction (α), the degree of the lip rising in the y-axis or the upper direction (β) and the degree of falling in the lower direction (γ). This movement degree is transmitted from the displacement extracting unit 30 to the movement position calculation unit 40 as displacement information.

In the movement position calculation unit 40, the lip deformation function has a specific function configured as follows. That is, it can be seen that the feature points a 'and b' moved from the feature points a and b move the right and left edge lips by an amount of ± α in the x-axis direction and almost no movement on the y-axis. When this is expressed, it is as following Formula (1).

Applying Equation 1 to the remaining points on the x-axis, the feature points (a, b) have a specific function such as Equation (2) which is moved by a parameter.

In Equation 2, specific functions f (a) and f (b) are values between "0" and "1", and as shown in FIG. 3A, f (-b) = 1, f (0) = 0 , f (a) = 1 is satisfied.

양만큼 올라가는 움직임이 있고, y 축방향으로 아래 입술이

양만큼 내려가는 움직임이 있음을 알 수 있다. 이를 수학식으로 표현하면, 하기 수학식 3과 같다.Similarly, the feature points c 'and d' moved at the positions of the original feature points c and d have little movement in the x-axis direction, and the upper lip in the y-axis direction.

There's a movement going up by the amount, the lower lip in the y axis

You can see that there is a downward movement. If this is expressed as an equation, Equation 3 below.

,

)에 의해 움직이는 하기 수학식 4 와 같은 입술 변형 함수를 갖는다.Applying Equation 3 above to the points on the remaining y-axis, the feature points (c, d) is a parameter (

,

It has a lip deformation function such as the following equation (4) to move by.

In Equation 4, the specific function g (c) is a value between "0" and "1", and as shown in FIG. 3B, g (-c) = 0, g (0) = 1, g (c) = 0 is satisfied. Similarly, in Equation 4, the specific function h (d) is a value between "0" and "1", and as shown in FIG. 3C, h (-d) = 0, h (0) = -1, h (d) = 0 is satisfied.

및

)를 상술한 수학식 3 및 4와 같은 특성을 갖는 입술 변형 함수에 적용하여 기본 입술 제공부(10)로부터 제공되는 기본 입술(100)의 에지점들의 위치로부터 이동되는 실제 입술 영상(200)의 위치를 구한다.The movement position calculator 40 may determine parameters provided from the deformation parameter extractor 30.

And

) Is applied to the lip deformation function having the characteristics as described above in Equation 3 and 4, the actual lip image 200 moved from the position of the edge points of the basic lip 100 provided from the basic lip providing unit 10. Find the location.

,

)만큼 이동되어 도 2b에 도시된 에지점 e'(x, y)로 변위되었다면, 이동 위치 계산부(40)는 이동된 위치 e'(x, y)를 다음과 같은 수학식 5를 이용하여 구할 수 있다.For example, one edge point e (x, y) of the lips shown in FIG.

,

2) and moved to the edge point e '(x, y) shown in FIG. 2B, the moving position calculator 40 converts the moved position e' (x, y) by using Equation 5 as follows. You can get it.

,

) 정보에 따라 입술상의 모든 에지점의 이동된 위치를 계산해낼 수 있다. 이동 위치 계산부(40)에서 계산된 이동 위치의 정보는 다음단의 입술 모양 합성부(50)로 제공된다.Therefore, the movement position calculator 40 uses the lip deformation function to calculate the displacement amount (

,

Based on the information, we can calculate the shifted position of every edge point on the lip. The information of the movement position calculated by the movement position calculator 40 is provided to the next lip composition unit 50.

Therefore, the lip shape combining unit 50 deforms the basic lip provided from the basic lip providing unit 10 according to the information of the moving position obtained by the moving position calculating unit 40 and synthesizes the changed new lip shape.

As described above, in the facial expression change coding system using the human shape model, a parameter representing the amount of displacement on the lip of the sender is calculated and applied to the preset function, thereby reducing the computation time for constructing a new lip shape. It will be possible to apply to a variety of applications when lip shape synthesis.

Claims (2)

In the modified lip synthesis apparatus of the facial expression change coding system using a human shape model, A basic lip providing unit for providing a basic lip shape of a passenger; Means for extracting a parameter value representing a degree of change of each edge point of the actual lip shape deformed in the x and y axis directions from the basic lip shape; Means for calculating a position shifted from the basic lip shape by applying a parameter value to the lip deformation function, the lip deformation function representing a degree of deformation of the lips; And means for synthesizing the modified actual lip shape from the basic lip shape according to the calculated position. The method of claim 1, wherein the moved position is represented by the following equations (1) and (2), [Equation 1]

[Equation 2]

In Equations 1 and 2,

And

Is a parameter representing the degree of movement of each edge point on the primary lip in the x-axis and y-axis directions, respectively, (a, b, c, d) are the right and left edges of the primary lip and the upper lip lower point, respectively. And feature points including the upper and lower lip upper points, wherein (a ', b', c ', d') are the feature points (a, b, c, d) in the x or y axis, respectively.

or

Deformed lip synthesizing apparatus, characterized in that f (a), f (b), g (c), h (d) is the lip deformation function, respectively.

Images