JPH0236687A - Face moving picture synthesis system - Google Patents

Abstract

PURPOSE:To synthesize a more natural moving picture of a face by modifying a mouth so as to be matched with audio information sent from a sender side and displaying a mouth internal picture corresponding to the audio information in the case of reproducing a picture. CONSTITUTION:A two-dimensional mouth patch model is modified in response to a mouth code corresponding to the inputted audio information at the communication after the initializing operation and a changeover means 5 is switched in response to the mouth code. A mouth internal picture data inputted via the changeover means 5 is assigned to the two-dimensional mouth internal patch model representing the shape of mouth inside in terms of the set of patches and the result is outputted as a mouth internal picture in response to the mouth code. Then a picture synthesis means 6 synthesizes picture information from a still face picture data storage means 1, a two-dimensional mouth patch model modification means 2 and a two-dimensional mouth internal patch model modification means 4. Thus, a more natural moving picture of a face is synthesized and displayed in response to the audio information sent during the communication.

Description

[Detailed description of the invention] [Table of contents] Overview Industrial application field Prior art (Figure 12) Means for solving the problem to be solved by the invention (Figure 1) Effect (Figure 1) ) Embodiment (Figures 2 to 11) Effects of the invention [Summary] By using a small amount of initialization data transmitted at the time of initialization, it is possible for the receiving side to adjust the facial expression according to the voice information transmitted during communication. Regarding a facial dynamic image synthesis system that synthesizes and displays moving images, when the receiving side deforms the mouth part to match the audio information sent from the transmitting side and plays the image,
The aim is to be able to display images of the inside of the mouth that correspond to audio information, thereby making it possible to synthesize more natural facial moving images. means for storing data; means for transforming a two-dimensional mouth shape patch model according to a mouth shape code corresponding to input audio information; means for applying the internal mouth image data selected according to the mouth shape code to an internal mouth two-dimensional patch model and outputting it as an internal mouth image according to the mouth shape code, and an image synthesizing means. Configure.

[Industrial Application Field] The present invention uses a small amount of initialization data transmitted at the time of initialization to synthesize a moving image of a face on the receiving side according to audio information transmitted during communication. This invention relates to a facial moving image synthesis system for display.

In television (TV) telephones, TV conferences, etc., it is considered appropriate to ultimately adopt a transmission method that uses public telephone lines, and for this reason, it is necessary to compress the obtained image information as much as possible. is requested.

[Conventional technology] Images transmitted on videophones, etc. are normally used.

Although this is a moving image of a person, such moving image information is
As shown in the figure, the conventional method is to transmit audio information independently.

That is, an input image is generated as an analog image signal by a TV camera 61 on the transmitting side, and this image signal is converted into a digital signal by an image encoding device 62, encoded, compressed, and sent to the receiving side. On the receiving side, an image decoding device 63 decodes the received image into an original signal and displays it on a display 64 as an output image.

In addition, the input voice is obtained as voice information by the micro 5 on the transmitting side, and is then compressed by voice-specific encoding in the voice encoding device 66. Then, on the receiving side, it is decoded by the voice decoding device 67, and then it is transmitted to the speaker. 68 as output audio.

However, since the amount of information contained in moving images is large, this conventional video transmission method that has been commonly used cannot use low bit rate communication lines.
There were problems in that the cost was high and that it was far from being applicable to TV telephones using public telephone lines.

Therefore, the sending side sends, for example, still image information of the face in advance, and the receiving side deforms the mouth part etc. from the audio information sent from the sending side to match this audio information. It is also possible to play back images.

[Problems to be Solved by the Invention] However, with such means, when synthesizing a face image with an open mouth, the inside of the mouth (teeth, tongue, etc.)
Since the image of the face is not displayed, there is a problem in that the moving image of the face remains unnatural.

The present invention aims to solve such problems, and by using a small amount of initialization data transmitted at the time of initialization, the receiving side can adjust the facial expression according to the voice information transmitted during communication. In a device that synthesizes and displays moving images, the receiving side transforms the first mouth part from the audio information sent from the transmitting side to match this audio information, and when playing the image, the audio It is an object of the present invention to provide a facial moving image synthesis system that can also display an image of the inside of the mouth corresponding to information and synthesize a more natural facial moving image.

[Means to solve the problem] FIG. 1 is a block diagram of the principle of the present invention.

In FIG. 1, reference numeral 1 denotes still face image data storage means, and this still face image data storage means 1 stores one frame of still face image data sent at the time of initialization.

2 is a mouth shape two-dimensional patch model transforming means, and this mouth shape two-dimensional patch model transforming means 2 converts a mouth shape two-dimensional patch model, which represents the shape of the mouth including the mouth as a set of patches, into a shape corresponding to the input audio information. It transforms depending on the mouth shape code.

Reference numeral 3 denotes an internal mouth image data storage means, and this internal mouth image data storage unit 3 stores internal mouth side image data obtained by cutting out the internal part of the oral cavity from a direct image of the mouth, as many as the number of whole mouth type codes. Therefore, a plurality (M: natural number) of intraoral surface image memories 3-1 to 3-M are provided.

Reference numeral 4 denotes an internal mouth two-dimensional patch model deforming means, and this internal mouth two-dimensional patch model deforming means 4 is selected from the internal mouth image data storage means 3 input via the switching means 5 which is switched according to the mouth shape code. The internal mouth image data is applied to an internal mouth two-dimensional patch model in which the internal shape of the mouth is represented by a set of patches, and is output as an internal mouth image corresponding to the mouth shape code.

Reference numeral 5 denotes a switching means, and this switching means 5 selects an intraoral surface image corresponding to the mouth shape code according to the mouth shape code.
(1=112t . . . , M) is switched to connect the internal mouth two-dimensional patch model deforming means 4.

Reference numeral 6 denotes an image synthesizing means, and this image synthesizing means 6 synthesizes image information from the still face image data storage means 19, the square two-dimensional patch model transforming means 2, and the internal mouth two-dimensional patch model transforming means 4. .

[Function] With the above configuration, first, at the time of initialization, still face image data is sent from the sending side to the still original image data storage means 1, but after that, the mouth shape two-dimensional patch model is matched with the still face image. The still face image data in the still face image data storage means 1 and the mouth shape two-dimensional patch model transforming means 2 are
The eye white part two-dimensional patch model deforming means 4 is mapped with the data of the mouth shape two-dimensional patch model, and the eye white part two-dimensional patch model deforming means 4 is mapped with the data of the mouth shape two-dimensional patch model. Mapping between the mouth interior two-dimensional patch model and the mouth interior side image data stored in the mouth interior image data storage means 3 is performed for all mouth shape codes.

After such initialization operations, communication is performed, and during such communication, the mouth shape two-dimensional patch model is transformed according to the mouth shape code corresponding to the input audio information, and , switching means 5 according to the mouth shape code.
The internal mouth side image data inputted through this switching means 5 is applied to the internal mouth two-dimensional patch model that represents the internal mouth shape as a set of patches, and the internal mouth side image data is applied to the internal mouth internal side image according to the mouth shape code. Output. That is, in accordance with each mouth shape code, the internal mouth image is reproduced from the mapping data of the corresponding internal mouth image.

Then, the image synthesis means 6 synthesizes the image information from the still face image data storage means 12, the square two-dimensional patch model transformation means 2, and the internal mouth two-dimensional patch model transformation means 4.

As a result, a moving image of a face is synthesized and displayed as a received image according to audio information transmitted during one communication.

[Example] Hereinafter, the present invention will be described in detail with reference to the drawings.

FIG. 2 is a block diagram showing an embodiment of the present invention. In this embodiment, a transmitting section 10 and a receiving section 20 are provided, and the transmitting section 10 is equipped with an image processing section 11 that processes an input facial image. , and a voice encoding unit 12 that encodes the voice input into code 2.

The receiving unit 20 also includes a background image memory (still face image data storage means) 19. Audio decoding section 21. Voice recognition section 227
Square model deformation section 36. Control point coordinate memory (table)
23. Eyelid shape model deformation section 24, synthesis section 25. Interpolation point calculation unit 27. Random pulse generator 28. It has a coordinate table control section 29.

Here, the background image memory 19 stores one frame of still image data of a face image (for example, face image data with a closed mouth) sent from the transmitting side at the time of initialization.

The audio decoding unit 21 decodes the audio code encoded by the transmitting unit 10, and the audio recognition unit 22 performs audio recognition on the audio signal output from the audio decoding unit 21.

The mouth shape model modification section 36 transforms the mouth shape according to the mouth shape code corresponding to the speech information inputted through the speech recognition section 22, and as shown in FIG.
2. External model transformation section 362. Mouth internal image data storage section 363. Mouth interior model deformation part 364. Changeover switch section 3
652 composite image memory 366.

The codebook 361 is for sequentially selecting a set of mouth shape parameter values from phoneme codes (consisting of vowels, consonants, etc., which are the basic structural units of speech) output one after another from the speech recognition unit 22. In the codebook 361, as shown in FIG. 4, each phoneme is recorded by a specific speaker! , ■...
・Parameters for the shape of the mouth when 9m is generated ■ (for example, the side of the mouth), II (for example, the thickness of the lips), ...+
A table quantified as n (for example, the vertical width of the mouth) is stored in advance as the personal information.

In addition, the external model deformation unit 362
In accordance with the mouth shape parameter values of IMi sequentially selected in step 1, the mouth shape 2 is expressed as a set of multiple (26 in this example) batches R to R2G, as shown in FIG. 7, including the mouth. By transforming the dimensional patch model using the vertices of each patch R to R2s as control points, the external model image is transformed. For example, the above phonemes i, n,
Fig. 6 (a) schematically shows an example of a self-portrait for m.
)(b),(Q).

Therefore, the codebook 3612 and the external model transformation unit 362 constitute a mouth shape two-dimensional patch model transforming means that transforms the mouth shape two-dimensional patch model according to the mouth shape code corresponding to the input audio information.

In addition, at the time of initialization, the external model transformation unit 362 receives in advance one screen (one frame) of self-portrait data of the specific speaker as personal information via the background image memory 19, and converts this data into the geometric shape of the mouth. The image mapped onto the patch model that serves as the framework of the shape is stored as a mouth shape model image, but even if one screen of the eye image is first sent from the transmitter 10 in this way, the food book 361 must be made in advance.

The internal mouth image data storage unit 363 stores the internal mouth image data obtained by cutting out the internal parts of the oral cavity (teeth parts, tongue parts, etc.) from the self-portrait as many as the number (M) of the whole mouth shape code. A plurality (M) of internal mouth image memories 363-
1 to 363-M. In this case, examples of internal mouth images of mouth shapes 1, 2°, and M are shown in the lower part of FIG. 9.

The mouth internal model deformation unit 364 has a mouth shape code 1 (i=1, 2
．． . . , M), the internal mouth image data storage unit 36 input via the changeover switch unit 365
The internal mouth image data selected from 3 is divided into a plurality of (8 in this example) patches S□~ as shown in FIG.
This is applied to a two-dimensional internal mouth patch model represented by a set of S, and output as an internal mouth image corresponding to the mouth shape code. The situation is schematically shown in FIG. 9.

The changeover switch section 365 is switched in accordance with the mouth shape code i to connect the mouth interior image memory 363-1 and the mouth interior model transformation section 364 corresponding to this mouth shape symbol.

The composite image memory 366 combines image information from the external model transformation section 362 and the internal mouth model transformation section 364.

In this case as well, mapping is performed between the still face image data and the data of the mouth-shaped two-dimensional patch model so that the one-mouth-shaped two-dimensional patch model (see Figure 7) matches the still face image at the time of initialization. , Mapping between the internal mouth 2D patch model and the internal mouth image data is performed for all mouth shape codes so that the internal mouth image matches the internal mouth 2D patch model (see Figure 8) for all mouth shape codes. I'll keep it.

Next, the interpolation point calculation unit 27 shown in FIG. 2 calculates all vertices P1 of the eyelid shape model (see FIG. 10) corresponding to the still image data.
~P, is received at the time of initialization, and the control points p,, p, , at each frame point from the start to the end of blinking are determined.
The coordinates of p4 are calculated by linear interpolation, and the data is sent to the control point coordinate memory 23.

That is, as shown in FIG. 10, this eyelid shape model has eight vertices Pi to P (each point has two-dimensional coordinate values of x and y) and these vertices Pi to P.

This eyelid shape model is composed of six triangular patches T□ to T6, which are formed by connecting the point), and the other five points are immovable (fixed points).
It is said that

Then, in this interpolation point calculation unit 27, in addition to the coordinates of eight vertices P1 to P8, P, , P, .
p2, p indicating the lowest point of P4.

The coordinate values of the three points P and ' are also given, and from the pre-given number of frames per blink N, P2 → P2'
→P z , P x → P, ' → P, , P4 → P4 → P4
It is designed to perform linear interpolation for each interval.

The control point coordinate memory 23 stores the blinking motion of the eyelids based on the eyelid parameters of the shadow model image. Specifically, three control points Pz, P3 . The coordinates of P4 are stored in the storage area of the control point coordinate memory 23 in the form of a table. An example of the structure of this control point coordinate table is shown in FIG.

The random pulse generator 28 generates a blink signal (random pulse signal).

Further, the coordinate table control unit 29 sequentially reads out all vertex data in the coordinate table of the control point coordinate memory 23 from the time when the blink start signal is received from the random pulse generation unit 28, and sends the data to the shadow model transformation unit 24 for each frame. This is what is transferred.

The shadow model transformation unit 24 stores a shadow model image defined by shadow parameters indicating the geometrical shape of the eyelid portion of the face. The shadow model image is then transformed based on the eyelid parameters. Specifically, by the action of the coordinate table control unit 29, the eyelid parameters sequentially sent from the control point coordinate memory 23 are taken in, and the shadow model image is transformed based on the eyelid parameters. Here, the state of deformation of this shadow model image is schematically shown in Fig. 11 (a
) to (c).

The synthesis unit 25 generates an image mouth image generated from the single-mouth model transformation unit 36 (this image is a composite of an extra-mouth shadow image including the periphery and an intra-mouth image including the teeth and gums). The eyelid image generated from the shadow model transformation unit 24 is then synthesized with an image other than the eye and eyelid portions of the still face image stored in the background image memory 19.

Next, the operation of this embodiment will be explained.

The audio input is encoded by the audio encoder 12 and sent to the receiver 20.
This audio code is decoded by the audio decoding section 21 and output as audio.

On the other hand, this voice output is sent to the voice recognition unit 22, and its phoneme codes are sequentially extracted and the codebook 3
Sent to 61. In the codebook 361, a set of parameter values I, II, . . . regarding the corresponding mouth shape are selected from the codebook shown in FIG.
・, select n.

Then, based on the selected set of parameter values, the external model deforming section 362 generates a mouth area image obtained by transforming the external model image stored in advance. As a result, the correspondence between the generated self-image and the phonemes extracted by the voice recognition unit 22 is as shown in FIGS. 6(a), (b), and (c), for example.

Further, the changeover switch section 365 is switched according to the mouth shape code i, and the internal mouth image data inputted through the changeover switch section 365 is applied to the internal mouth two-dimensional patch model, and the mouth according to each mouth shape code i is applied. Output as an internal image. That is, according to each mouth shape code, the internal mouth image is reproduced from the mapping data of the corresponding internal mouth image. The situation is schematically shown in FIG. 9.

Then, in the composite image memory 366, the image information from the external model deforming section 362 and the internal mouth model deforming section 364 are combined to create a self-portrait.

In addition, at the time of initialization, mapping is performed between the still face image data and the data of the mouth shape two-dimensional patch model so that the mouth shape two-dimensional patch model shown in FIG. 7 matches the still face image. , the mapping of these internal mouth 2D patch models and the internal mouth image data is performed to match the entire mouth shape code to the internal mouth 2D patch model (see Figure 9). It is carried out about.

On the other hand, the random pulse generator 28 generates a blink start signal at random time intervals.

When the pulse train signal is outputted from the random pulse generator 28 in this way, the coordinate table controller 29 starts the control point coordinate memory 2 from the time when this blink start signal is received.
All vertex data in the coordinate table No. 3 is read out and transferred to the shadow model transformation unit 24 for each frame. Such transfer ends when the number of frames per unit blink has elapsed since the blink start signal was generated. Then, the shadow model transformation unit 24 generates an eyelid image by transforming the shadow model image stored in advance according to the above vertex data.

The self-portrait (this self-portrait is a composite of the external mouth image and the internal mouth image) and eyelid image transformed and generated in this way are then combined with the mouth of the still face image stored in the background image memory 19. and images other than the eyelids, in the compositing section 25,
The images will be combined and output as a moving image of the entire face.

This makes it possible to further compress the information in the original video, greatly reducing the amount of information.As a result, it is possible to realize an inexpensive image transmission method that uses a low bit rate line, and the eyelids of the reproduced image can be adjusted appropriately. While blinking,
Furthermore, when deforming the mouth to match the input audio information, it is also possible to display an image of the inside of the mouth that corresponds to the audio information, making it possible to synthesize a more natural moving image of the face.

The method used to transform the mouth shape model image in the mouth shape model transformation section 36 and the shadow model image in the shadow model transformation section 24 is as described in IEICE Technical Report E87-2. 87, No. 19, 1987.

[Effects of the Invention] As detailed above, according to the facial moving image synthesis system of the present invention, the information of the original moving image can be further enhanced, f! As a result, the amount of information can be greatly reduced, and as a result, it is possible to realize an inexpensive image transmission method that uses a low bit rate line. When playing back an image by deforming the mouth part to match the image, it is also possible to display an image of the inside of the mouth that corresponds to the audio information, which has the advantage of being able to synthesize a more natural-looking moving image of the face.

[Brief explanation of the drawing]

Fig. 1 is a block diagram of the principle of the present invention, Fig. 2 is a block diagram showing an embodiment of the present invention, Fig. 3 is a block diagram of a mouth shape model deformation section, Fig. 4 is a configuration diagram of a codebook, and Fig. 5 is a block diagram showing an embodiment of the present invention. The figure is a configuration diagram of the control point coordinate table, Figure 6 (a) =
(b) and (c) are diagrams showing mouth images corresponding to phoneme codes. FIG. 7 is a diagram showing a mouth shape two-dimensional patch model. Figure 8 shows a two-dimensional patch model of the inside of the mouth.
The figure is a schematic diagram showing a method for generating an internal mouth image. Figure 10 is a diagram showing the configuration of the shape model of the pupil region, Figures 11 (a), (b), and (c) are diagrams explaining the concept of deformation of the shadow model image, and Figure 12 is the conventional general FIG. 2 is a system diagram showing an image transmission method. In the figure, 1 is a still face image data storage means, 2 is a mouth shape two-dimensional patch model transformation means, and 3 is an internal mouth image data storage means. 3-1 to 3-M are internal mouth image memories. 4 is an internal mouth two-dimensional patch model deformation means, 5 is a switching means, and 6 is an image synthesis means. 10 is a transmitting unit, 11 is an image processing unit, 12 is an audio encoding unit, 19 is a background image memory, 20 is a receiving unit, 21 is an audio decoding unit, 22 is an audio recognition unit, 23 is a control point coordinate memory ( 24 is an eyelid model transformation section, 25 is a synthesis section, 27 is an interpolation point calculation section, 28 is a random pulse generation section, 29 is a coordinate table control section, and 36 is a mouth shape model transformation section. 361 is a codebook. 362 is an external model transformation part. 363 is an internal mouth image data storage unit, 363-1 to 363-M are internal mouth image memories, and 364 is an internal mouth model transformation unit. 365 is a changeover switch section, and 366 is a composite image memory. Codebook Sugiri Yunguchi No. 4 Figure 51 Ken94 Ex) i Wholesale, Zeng, 7 壬 小 連 ＠ 5 目 5 Country f-L Good business ■ = 1 11i (O) (b) If ” λ To Shinmoto Zuzuoro D Dan I family 1 Uncho to skill 6 Lo shape 2 N hole base V- / To 4 people in Aohimochino '1' D Ko // ＼ Roro tongue rock jI 徨Fl raw, sucking
In the figure (internal table P2 dimension), 7 hand mosuru begging 2 and j, ÷ shell area / 1'71'9 shape 7 i'le 7 mushrooms 7 mosquitoes? , Seto Figure 10 - Frame 1 7 Shi-mu N/2 7 Shi-mu N 8 Meeting Wamosuru V! Image η-1-IAtrt/Zhong Egji bright phase Fig. 11

Claims (2)

[Claims] (1) By using a small amount of initialization data transmitted during initialization, depending on the audio information transmitted during communication,
In a facial moving image synthesis system that synthesizes and displays facial moving images on a receiving side, the receiving side includes still face image data storage means (1) for storing one frame of still facial image data sent at the time of initialization. , 19) and mouth shape 2, which represents the shape of the mouth area including the mouth as a collection of patches.
Mouth shape two-dimensional patch model transformation means (2, 361, 362) that transforms a dimensional patch model according to a mouth shape code corresponding to input audio information; and mouth internal image data that is obtained by cutting out an internal part of the oral cavity from a mouth image. internal mouth image data storage means (3, 363) for storing the same number of mouth shape codes as the number of mouth shape codes;
63) is applied to an internal mouth 2-dimensional patch model in which the internal mouth shape is represented by a set of patches, and outputs an internal mouth image according to the mouth shape code. Transformation means (4, 364)
and combining image information from the still face image data storage means (1, 19), the mouth shape two-dimensional patch model transformation means (2, 362), and the mouth internal two-dimensional patch model transformation means (4, 364). 1. A face moving image synthesis system, comprising: an image synthesis means (6, 366). (2) At the time of initialization, the still face image data and the data of the mouth shape two-dimensional patch model are mapped so that the mouth shape two-dimensional patch model matches the still face image, and the mouth internal image is is characterized in that mapping is performed between the intra-mouth shape two-dimensional patch model and the intra-mouth image data for all mouth shape codes so that the internal mouth shape two-dimensional patch model matches the entire mouth shape code with the intra-mouth shape two-dimensional patch model. The face moving image synthesis system according to claim 1.