KR100220699B1 - Apparatus for coding lip shape information in a 3 dimension model-based coding system - Google Patents

Abstract

The present invention relates to an apparatus for extracting lip shape information from a lip image and a voice in a three-dimensional model-based coding system, comprising: a sound detector for detecting presence of a sound from the voice and outputting a switching control signal; A speech recognition unit for outputting a deformation parameter value of a lip pattern corresponding to a voice, a first coding unit for coding the lip image by a first coding scheme, and a deformation parameter value of the lip pattern provided from the voice recognition unit; And a second coding section for coding in a coding scheme, and a switching section for selectively outputting information coded by the first and second coding sections in response to a switching control signal from the sound detecting means.

Description

Lip Shape Information Extraction Device of 3D Model-based Coding System

The present invention relates to a facial expression change coding system using a three-dimensional model, in particular a human shape model, and more particularly to an apparatus and method for extracting the shape information of the lips.

Application techniques using three-dimensional object models, especially human shape models, have been applied in the field of video coding and computer graphics. In the field of video coding currently underway, research to transmit a satisfactory image quality with a smaller amount of data continues. Looking at the research projects according to the amount of data to be transmitted, there is MPEG4, which starts from HDTV and is currently being actively studied. In MPEG4, a video signal is transmitted using a very small amount of data. Therefore, not only a transformation using information theory, for example, a cosine transform coefficient method, but also other computer vision, The techniques used in computer graphics are also applied.

In general, the shape of a person seen in a videophone or video conference only shows the chest above the waist. In particular, the speaker's face takes up most of the video information transmission. If only non-human face information, ie background information, can be ignored and only regular information on the face shape can be sent, the transmission rate can be greatly reduced. In other words, it finds the rules of the general face, face, expression, and emotions of the face and mouth when speaking, and sends them to the other party through one code. By deforming the face shape according to the received code, the expression shape of the sender may be recognized with a small amount of data.

More specifically, the three-dimensional model-based coding system targets coding in which two people talk in real time, such as a video phone, and transmit not only sound but also a changing face or body in real time. However, unlike general video coding, the transmitted data does not handle an image composed of pixels that change every moment. Instead, it extracts and transmits a certain motion variable, and then has the data on the receiving side, that is, the face of the sender. It is combined with the dimensional head shape model, body shape model, etc. to form the face or body image of the final changer after reconstruction.

Of course, such a system configuration can be used when the amount of data to be transmitted is very limited by the data transmission bandwidth, but not a large amount, but requires sufficient image quality for accurate recognition between the two parties. If the bandwidth is large enough to send enough data, the advantage of this system will be that low. This is because regular code lacks a natural look.

In the 3D model-based coding system, the shape of the mouth, especially the shape of the lips, has a very important visual information in the face images of the senders. In other words, it is possible to estimate what the sender is saying and what emotions are expressed in the form of lips.

Therefore, there is a demand for a system capable of extracting enough information to reconstruct a satisfactory mouth shape with a small amount of data.

Therefore, an object of the present invention is to provide a method and apparatus for extracting lip shape information in a 3D model based facial expression coding system.

According to the present invention for achieving the above object, there is provided an apparatus for extracting the lip shape information from the lip image and the voice in a three-dimensional model-based coding system, which detects the presence of sound from the voice to provide a switching control signal A sound detection unit for outputting, a voice recognition unit for outputting a deformation parameter value of a lip pattern corresponding to the voice recognized from the voice, a first coding unit for coding the lip image by a first coding method, and the voice recognition unit Selectively outputting information coded by the first and second coding units in response to a switching control signal from the sound detecting means, and a second coding unit for coding the deformation parameter values of the lip patterns provided from the second coding scheme; And a switching unit.

1 is a block diagram of an apparatus for extracting lip shape information of a 3D model based coding system constructed according to a preferred embodiment of the present invention.

FIG. 2 is a diagram illustrating a pattern of a lip shape used in the speech recognition unit illustrated in FIG. 1.

3 is a diagram of a basic lip used to generate a variable according to lip movement

* Explanation of symbols for main parts of the drawings

10: voice processing unit 20: image processing unit

30: sound measuring unit 40: voice recognition unit

50: switching unit 60: first coding unit

70: second coding unit

The above and other objects and various advantages of the present invention will become more apparent from the preferred embodiments of the present invention described below with reference to the accompanying drawings.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1, a block diagram of an apparatus for extracting lip shape information from lip image data and voice data in a 3D model based coding system according to the present invention is shown. The apparatus for extracting lip shape information of the present invention includes a speech processing unit 10, an image processing unit 20, a sound measuring unit 30, a speech recognition device 40, a switching unit 50, a first coding unit 60, and a first processing unit. 2 coding section 70 is provided.

The voice processor 10 quantizes the voice generated through the microphone (not shown), and provides the voice measurer 30 and the voice recognition device 40.

The image processor 20 extracts the lips image by separating the sender's mouth image provided from the camera (not shown) into regions having similar pixel values. The extracted lip images are provided to the first and second coding units 60 and 70, respectively.

The extracted lip image may be classified into a form having no pattern, such as an emotional expression, and a form having a certain pattern, such as a pronounced word. Since the expression of emotions varies from person to person, and mouth shape varies according to subtle emotions, it will be impossible to express mouth shape according to a proper rule. For this, the first coding unit 60 codes using the contour coding technique.

However, it would be possible to provide an appropriately regular rule for the shape of the lips with a certain pattern to represent the shape of the mouth. In this regard, the deformation parameter value of the lip image provided from the speech recognition unit 40 is coded by the second coding unit 70 according to a parameter coding technique.

The first coding unit 60 extracts an edge portion from the lip image by using a conventional edge detection method, and includes a contour indicating the boundary of the extracted lip and information on the inside of the region. Code each separately.

The edge detection is found by using the brightness distribution of the original image in which the brightness of the image changes rapidly. That is, when the original image is input, the level difference of the luminance signal is detected. When the detected level difference is larger than a predetermined threshold value, the input original image is determined as the boundary portion and the boundary data is output.

In addition, in processing the contour of an object, the contour information is very important for analyzing and synthesizing the shape of the object. As a conventional coding technique for representing such contour information, for example, a chain coding method is used. have. In this case, the chain encoding method is a technique of encoding contour information starting from an arbitrary point on the contour line and arranging the boundary lines as a permutation of direction vectors in a predetermined direction according to the connection state between pixels. Detailed description is omitted since it is a well known technique. As described above, the lip information chain-coded by the first coding unit 60 is provided to the switching unit 50.

On the other hand, the sound measuring unit 30 detects the presence or absence of the sound of the transmitter from the input voice data and provides the sound presence signal to the multiplexer 50 as a switching control signal. The detection of the presence of sound is determined by comparing whether the height of the input voice is higher than the preset reference value or is unfamiliar. The sound presence signal detected by this determination is transmitted to the multiplexer 50 by the first coding unit 60 and the second coding unit 70 for the lip shape with the speech sound and the lip shape without the speech sound. It is used as a judgment signal for selectively outputting the processed signal.

As described in detail below, the speech recognition apparatus 40 performs a function of classifying the primary neutral species from the input speech and outputting the lip deformation variable values already configured according to these three combinations.

In any language, speech sounds are divided into consonants and vowels, and mouth shape depends largely on vowels rather than consonants. Since Korean is a letter based on speech, it is very systematic and analytical. If you build a speech and mouth-based system based on Korean language, you will be able to express more than 90% of the visual sounds of speech in all countries. Therefore, the present invention establishes a visual rule for the shape of the mouth based on the Korean language and uses this as information on the shape of the mouth.

Representative mouth shapes have a pattern of visual forms depending on the vowels being neutral or longitudinal. The consonant, which can be a first consonant and a final consonant, is used to determine the instant start and end of the change of the lips, namely the closed state and the open state.

Referring to Figure 2, in accordance with the present invention is shown that the mouth shape according to each neutral formed in eight patterns to set the shape of the lips.

In the speech recognition apparatus 40, a lip pattern corresponding to the neutral extracted from the voice pronounced by the sender is selected from FIG. 2, and the selected lip pattern is selected from the lips of the basic shape as shown in FIG. 3. It is created with a variable value representing the amount of displacement.

As shown in FIG. 3, the deformation of the basic lip form mostly occurs as a change of the lip, and runs a very complex change form. The variables include middle and upper movement variables L3 and L4 of the lips, lip movement variables L7 and L8 of the lips, lip movement variables L1 and L2, and lip movement variables L5 and L6.

In this case, the lip variable value is selected within a predetermined range, for example, a range of -1 to 1, and the median value "0" indicates that there is no deformation of the lip shape and indicates a position value of the basic lip shape. The variable value selected by the speech recognition apparatus 40 is provided to the second coding unit 70 and coded. The lip information coded by the second coding unit 70 is provided to the switching unit 50.

The switching unit 50, which may be configured as a multiplexer, may selectively output coded data output from the first coding unit 60 or the second coding unit 70 according to the switching control provided from the sound measuring unit 30. do. That is, when the sound detection signal is provided from the sound measuring unit 30, the switching unit 50 transmits the lip image information output from the second coding unit 70 to the receiving side, and detects the sound from the sound measuring unit 30. If no signal is provided, the switching unit 50 transmits the lip image information output from the first coding unit 60 to the receiving side.

As described above, when the 3D object model-based coding system transmits information about the lip shape of the sender, the lip shape information generated by applying general coding and the lip information generated by using parameter coding are selectively selected according to the presence of sound. Because of this, the amount of data transmitted can be reduced, and even with this small amount of data, the facial expression of the sender, especially the shape of the sender's lips, can be identified.

Claims (4)

An apparatus for extracting lip shape information from a lip image and a voice in a 3D model based coding system, Sound detection means for detecting presence of sound from the voice and outputting a switching control signal; Speech recognition means for outputting a deformation parameter value of a lip pattern corresponding to the speech recognized from the speech; First coding means for coding the lip image in a first coding scheme; Second coding means for coding a deformation parameter value of the lip pattern provided from the speech recognition means in a second coding scheme; And switching means for selectively outputting information coded by the first and second coding means in response to the switching control signal from the sound detection means. 2. The variable according to claim 1, wherein the speech recognition means discriminates the primary neutral species from the input voice, selects one lip pattern corresponding to the differentiated neutral, and indicates a displacement amount in which the selected lip pattern is deformed from the basic lip pattern. Lip shape information extraction device, characterized in that for outputting. According to claim 2, wherein the deformation parameter of the mouth is the left and right movement variable (L1, L2) indicating the displacement amount according to the left and right movement of the lips, the middle lips up and down respectively indicating the displacement amount according to the vertical movement of the middle and the edge of the lips Lip shape information extraction comprising the motion variables L3 and L4, the lip up and down motion variables L7 and L8, and the lip front and back motion variables L5 and L6 representing the displacement amount according to the front and rear movement of the lips. Device. The method of claim 1, wherein the first coding scheme performed by the first coding unit extracts an edge portion of the lip image, and includes a contour representing a boundary of the extracted lips and information about the inside of the region. An apparatus for extracting lip shape information, characterized by coding content.