JPH0652290A - Method and device for synthesizing mouth-shaped picture - Google Patents

Abstract

PURPOSE:To correct the mouth shape of the characters appeared on stage to correspond to the voice in each language when the voices spoken by plural languages are imparted for one video signal. CONSTITUTION:The three-dimensional movement of the character's head in a screen is detected by a three-dimensional movement detection section 1. On the other hand, a mouth shape parameter genearation section 2 generates the mouth shape parameter at every frame based on the phoneme string consisting of the voice to be impart and the time duration of each phoneme. With the use of these results, a coordinate correction section 3 corrects the direction and position of the three-dimensional shape model of the head of the characters and the shape around the mouth. The brightness and color information are given to the mouth of the obtained shape model and the part around the mouth by a texture mapping section 4. Further, a synthesis section 5 replaces the result with the mouth in the original picture and the picture of the part around the mouth. Thus, even when the voices by different languages are imparted, the natural face movement picture having the mouth shape corresponding to the voice can be outputted.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is for expressing a mouth shape adapted to voice in different languages for a human face image in a video image of a TV telephone / TV conference or a movie / broadcasting TV. The present invention relates to a method and an apparatus.

[0002]

2. Description of the Related Art In a TV telephone or a video conference, it is possible that a speaker utters a content in a certain language A (eg, English) and translates it into another language B (eg, Japanese) and transmits it. At this time, the change in the mouth shape of the speaker due to the utterance is
When the receiver receives the image of the speaker and the speech translated into the language B, the mouth shape changes and the speech does not match, resulting in an unnatural impression. Will be given.

As another example, when a voice is translated and broadcast in a movie or a TV broadcast program, a change in the mouth shape of the characters on the screen and a voice after translation occur inconsistency. In addition, a plurality of audio channels in different languages may be given to one video signal to perform TV broadcasting. In this case as well, a change in the mouth shape of the characters on the screen may occur when the program is produced. It corresponds to the language at the time of utterance, and a mismatch will occur with voices in other languages.

As described above, when a plurality of voices based on different languages are added to one video signal, there arises a problem that the mouth shape of the character changes and the voices do not match. As a conventional technique for reducing this inconsistency, a patent (Procede de television by Guinet, Yves) is available.
pour programs multilingues: European Patent Off
ice EP 0 179 701 A1: 1986.4.30.).

FIG. 9 shows the method of the Guinet, Yves patent. In FIG. 9, 100 is an original image, 101 is an original voice, 102 is an output image in which the shape of the mouth part is corrected, 1
Reference numeral 03 indicates a translated voice, and reference numeral 104 indicates a combination operation. Figure 9
It shows a general concept of outputting an image representing a mouth shape change corresponding to a translated voice by using data obtained from the original image, the original voice, and the translated voice. The following way of thinking is stated as a concrete realization method.

First, spatial analysis is performed on the original image 100 to obtain data on the center position (x1, y1) of the vocalization area, the relative size, and the posture. From the original voice 101,
Get information on whether the speaker in the original image is speaking or silent. From the translated voice 103, information indicating whether the speaker of the translated voice is speaking or silent, and information that describes the vocalization activity of the speaker, that is, the change in the mouth shape, is obtained. Here, regarding the change of the mouth shape, the mouth is closed or opened by the detection device physically connected to the speaker.
It is said that it is easy to get information that it is closing or opening. Based on the above data, first, for the vocalization area of the original image, by averaging the brightness values,
Make a blurred image so that you don't know the shape of the lips. Next, the color component value of the utterance area is corrected based on the information regarding the change in the mouth shape of the speaker of the translated voice, and the result is combined with the combination operation 10
An output image 102 is obtained by superimposing it around (x1, y1) of the image in which the vocalization area is blurred by 4. It is assumed that the correction of the color component value corresponds to the synthesis of the mouth shape image corresponding to the translated voice.

[0007]

The above-mentioned patent as a prior art is a patent that describes the concept of generating an image having a mouth shape corresponding to translated speech, and will be described in detail in the description part of the embodiment. Little is shown how. That is, first, it is described that the position, relative size, and posture of the vocalization area are detected from the original image by spatial analysis, but no specific method is used to detect them. No specific method is described for detecting the change in the mouth shape of the speaker of the translated speech. Further, regarding the generation of the mouth shape corresponding to the translated speech, the method for blurring the vocalization area in the original image is described, but the method for correcting the color component to generate the mouth shape is not described. . Therefore, it is unclear whether an image having a mouth shape corresponding to the translated speech can be actually generated or, if it can be generated, an image of good quality can be obtained.

The present invention has been made to solve these problems found in the above-mentioned prior art patents. That is, first, the head of a person in the screen usually has a three-dimensional movement such as facing sideways or nodding, but including this point, the vocalization area in the screen, that is, the mouth. An object of the present invention is to provide a method and an apparatus capable of correcting a mouth shape corresponding to a voice of the language in consideration of a position and an orientation of a part. Further, regarding generation of a mouth shape image corresponding to a voice of the language, it is possible to provide a method and an apparatus capable of explicitly handling the shape, brightness, and color information of the mouth portion and obtaining a high-quality output image. With the goal.

[0009]

The first feature of the present invention is to:
An original image is a digital moving image signal obtained by digitizing a video signal obtained from a television camera or a video tape recorder, and the translated voice obtained by translating the original voice given to the video signal into a different language is decomposed into a phoneme string. The information of the type and duration of each phoneme at time is input as to the voice, and the three-dimensional movement of the human head is detected from the three-dimensional shape model expressing the human head shape in the image and the original image. on the other hand,
A mouth shape parameter for each frame corresponding to the translated speech is generated from the phoneme and the duration information, so that the shape of the mouth and the periphery of the three-dimensional shape model of the head becomes the shape specified by the mouth shape parameter. The coordinates are modified, and further, the coordinates are modified so that three-dimensional motion is expressed according to the three-dimensional motion parameters of the head for the entire head including the mouth part, and then the three-dimensional shape after the coordinate modification. The texture information of the person's mouth and the area around the mouth that has been obtained in advance is added to the mouth and the area around the mouth of the model, and the image of the mouth and the area around the mouth newly obtained as a result of It is to replace the images of the mouth and the area around the mouth and synthesize a natural image having a mouth shape change corresponding to the translated speech.

A second feature of the present invention is that a first input terminal for inputting a digital moving image signal obtained by digitizing a video signal obtained from a television camera, a video tape recorder or the like as an original image, and a video signal. Second and third input terminals for inputting information on the type and duration of each phoneme when the translated voice obtained by translating a given original voice into different languages is decomposed into a phoneme sequence, and the person in the image A three-dimensional motion detecting unit that detects a three-dimensional motion of a human head from a three-dimensional shape model expressing the head shape of the human head and an original image input from the first input terminal; A mouth shape parameter generation unit that generates a mouth shape parameter for each frame corresponding to a translated speech from a phoneme type and duration information input from the third input terminal, and a mouth of a three-dimensional shape model of the head. And around the mouth Is corrected so that the shape becomes the shape specified by the mouth shape parameter obtained from the mouth shape parameter generation unit, and the head obtained from the three-dimensional motion detection unit for the entire head including the mouth portion. The coordinate correction unit of the shape model that corrects the coordinates so as to express the three-dimensional movement according to the three-dimensional movement parameter of the above, and the mouth of the three-dimensional shape model after the coordinate correction obtained from the coordinate correction unit of the shape model. A texture mapping unit for giving the texture information of the person's mouth and the surroundings of the mouth to the mouth and the surroundings of the person, and the first input terminal by the image of the mouth and the surroundings of the mouth obtained from the texture mapping unit. And a synthesizing unit that replaces the image of the person's mouth and the area around the mouth in the original image obtained from FIG.

[0011]

1 is a block diagram for explaining an embodiment of the present invention. In FIG. 1, 1 is a three-dimensional motion detection unit, 2 is a mouth shape parameter generation unit, 3 is a shape model coordinate correction unit, 4 is a texture mapping unit, 5 is a synthesis unit,
90 is a first input terminal for inputting an original image, 91,
Reference numeral 92 designates second and third input terminals for inputting the type and duration of each phoneme constituting the translated speech to be added to the original image, and 99 outputs the image after the mouth shape is corrected. Is an output terminal for. Here, the original image is T
Consider a digital moving image signal after A / D conversion (sampling and quantization) of an analog moving image signal obtained from a V camera or a VTR. Therefore, it is assumed that the processing in each part of the present invention is basically performed by digital processing.

Before explaining the operation of each part of FIG.
The relationship between the original image and the output image in the present invention will be described with reference to FIG. FIG. 2 is an explanatory diagram for explaining the operation of the present invention. In FIG. 2 (1), when the face is almost in front, FIG. This is the case when facing. In FIG. 2 (1), a1 and a2 are the original image and the output image, Ei1 and E02 are the mouth and the peripheral portion of the mouth in the original image and the output image, respectively. In FIG. 2 (2), b1 and b2 are the original image and the output image respectively, and Ei2 and E02 are the mouth and the mouth peripheral part in the original image and the output image, respectively. According to the present invention, as shown in FIG. 2, the shape of the mouth and the vicinity of the mouth are converted into a shape corresponding to the translated voice, and the output image is synthesized, taking into consideration the three-dimensional movement of the entire head. It

Before explaining the handling of image information,
First, the handling of voice information will be described. In FIG. 1, it is premised that, regarding the translated voice that is newly added to the image signal, the type and duration information is given to each phoneme when it is decomposed into a phoneme sequence that is a unit of the voice. At this time, it is possible to use a real voice by a dubber as a newly added translated voice, or to automatically synthesize a translated voice from an original voice. Figure 3
FIG. 7 is a block diagram for explaining the operation in the former case. In FIG. 3, 6 is a voice recognition unit (A), and 93 is an input terminal for inputting a real voice. Regarding speech recognition itself, existing speech recognition technology can be used, and since it is not a requirement directly aimed at by the present invention, detailed description thereof will be omitted. The voice recognition unit (A) 6 analyzes the input real voice using a method such as frequency analysis, cepstrum analysis, and linear prediction analysis to identify each phoneme and calculate the duration. For an overview of these, see, for example, Yoshiaki Shirai: “Pattern Understanding” (Ohmsha) pp. 179-255. Note that when a face moving image of the speaker is obtained at the same time for the newly added real voice, the result of measuring the shape of the mouth portion in this face moving image is used to correct the mouth shape in the output image. May be. As a method for this, for example, literature, Masahide Kaneko, Atsushi Koike, Yoshinori Hatori: “Real-time motion system for model-based coding of facial moving images”, 1990 Image Coding Symposium Material 3-6,
pp. There are 59 to 62. In this document, first, noise reduction processing is performed on an original image by a median filter, and then binarization is performed using an appropriate threshold Th. A label is given to each element in the binarized image, connected components are extracted, and the minimum circumscribed rectangle and fillet coordinates are detected for each connected component. Using these results, the small region is considered to be a noise component and removed, and the region of the mouth portion is selected with reference to the detection results up to the immediately preceding time point. As a result, it is possible to determine the degree of opening of at least the mouth portion in the horizontal and vertical directions. Next, FIG. 4 is a block diagram for explaining the operation in the latter case, that is, in the case of automatically synthesizing translated speech from original speech.
In FIG. 4, 7 is a voice recognition unit (B), 8 is a translation processing unit, 9 is a voice information generation unit, and 94 is an input terminal for inputting an original voice. The voice recognition unit (B) 7 has substantially the same function as the voice recognition unit (A) 6 in FIG. 3, but outputs the recognition result for the original voice as character information (sentence). The translation processing unit 8 translates a sentence in a designated language based on the character information (sentence) output from the voice recognition unit (B) 7. As for the automatic translation technology by machine, existing technology can be used, and since it is not a target requirement of the present invention, detailed description thereof will be omitted. The speech information generation unit 9 utilizes the rule synthesis technology of speech for the translated sentence to automatically generate the information on the type and duration of the phoneme. As for this part, the existing technology can be used and it is not the target requirement of the present invention, and thus the detailed description is omitted.

Hereinafter, each part of FIG. 1 will be described, but before that, the features of the method of handling a human face portion according to the present invention will be described. That is, in the present invention, a three-dimensional shape model expressing the three-dimensional shape of the human head is used in order to detect the position and orientation of the mouth portion in the original image and to generate a high-quality mouth shape image. This shape model is represented by, for example, a set of small triangular patches, and the shape change can be expressed by changing the vertex coordinate position of each triangle. When there are a plurality of persons on the screen, a three-dimensional shape model is prepared for each person, and the method described below is applied to each person.

Referring to FIG. 1, the operation of the three-dimensional motion detecting section 1 will be described first. The three-dimensional motion detector has 2
The object is to obtain the movement information in the three-dimensional space, that is, the change in the orientation and position of the face, from the three-dimensional input moving image for the face portion of the character on the screen. A method for this is described in Jun Koike, Masahide Kaneko, and Yoshinori Hatori, “3 heads in model-based coding of facial motion images.
Estimation of dimensional movement "IEICE Transactions BI, vo
l.J74-BI, no.10, pp.789-798 (October 1991)
There is. FIG. 5 is a block diagram showing an example of a specific configuration of the three-dimensional motion detection unit 1 based on the method proposed in this document. In FIG. 5, 11 is a frame memory, 12 is a block division unit, 13 is a motion vector detection unit, 14 is a three-dimensional motion estimation unit, and 15 is a depth coordinate calculation unit.

Here, the principle of the method for estimating the three-dimensional movement of the object (here, the human head portion) from the two-dimensional input moving image will be described.

Assuming that the head including the face portion is a rigid body and the movement between frames is minute, the point P (x, y, z) in the previous frame is a rotational movement component as shown in equation 1. And the translation component are combined to move to the point P ′ (x ′, y ′, z ′) in the current frame.

[0018]

[Equation 1] In Equation 1, t represents transposition, R represents rotational movement, and T represents translational movement. Further, if the projection from the three-dimensional space to the two-dimensional image is regarded as parallel projection, Tz = 0 can be set.

When the coordinates x, y, z of the point P and the coordinates x ', y'of the point P'in the previous frame and the current frame are obtained, these are substituted into the formula shown in Formula 1, and Ωx, Ωy, Ωz. , Tx
, Ty and z ′ may be set up to calculate the solution. However, it is difficult to solve the equation because the z coordinate information cannot be obtained directly from the input image. In order to solve this problem, motion information in a two-dimensional plane is obtained from an input moving image, and three-dimensional structure information (particularly, z-coordinate information) obtained from this result and the shape model is used.
Estimate motion parameters in three dimensions.

Specifically, the three-dimensional movement of the human head is detected by the configuration shown in FIG. The frame memory 11 is a memory for storing the input image at the immediately preceding time point, that is, the previous frame image. The block division unit 12 obtains a three-dimensional shape model corresponding to the position / orientation of the human head in the previous frame image from the coordinate correction unit 3 of the shape model, which will be described later, and uses this three-dimensional shape model as a frame. Memory 1
The area of the human face portion in the previous frame image is determined by projecting it onto the previous frame image stored in No. 1, and this area is further divided into small blocks of N × N pixels (N is, for example, 13). Divide into groups. Motion vector detection unit 13
Then, for each block obtained by the block division unit 12,
Regarding the image in the current frame obtained from the input terminal 90, a place where the distribution of pixel values is most similar is obtained, and a two-dimensional motion vector is calculated from the difference between this place and the position of the block in the previous frame image. Ask. As a result, the x, y coordinate values (x, y) of the point P and the x, y coordinate values (x ', y') of the point P ′ in the above description for the corresponding block are obtained.
Is obtained.

On the other hand, in the depth coordinate calculation unit 15, for each block obtained by the block division unit 12, the position (x, y) in the screen is calculated for each block, and the three-dimensional shape model in the previous frame is calculated. The z-coordinates of the small blocks are calculated by finding the vertices within the small block area centered at (x, y) and finding the average value of the depth coordinates (z) of those vertices. In the three-dimensional motion estimation unit 14, the value of (x, y), (x ′, y ′) obtained from the motion vector obtained from the motion vector detection unit 13 for each small block and the depth coordinate calculation unit 15 are obtained. Using the depth coordinate z, the three-dimensional motion parameters Ωx, Ωy, Ωz, T
Obtain a system of linear equations for x, Ty, and z '. By solving this equation by the method of least squares, a three-dimensional motion parameter is obtained and passed to the coordinate correction unit 3 of the shape model.

Next, the operation of the coordinate correction unit 3 of the shape model will be described with reference to FIG. In FIG. 6, 31 is a coordinate changing unit (A), and 32 is a coordinate changing unit (B). In the coordinate correction unit 3, based on the mouth shape parameter given from the mouth shape parameter generation unit 2 and the three-dimensional movement parameter of the head given from the three-dimensional movement detection unit 1, the mouth in the three-dimensional shape model of the head And the purpose is to modify the shape of the periphery. First, the coordinate changing unit (A)
At 31, the position of the vertex coordinates forming the mouth portion in the three-dimensional shape model of the head is changed based on the mouth shape parameter provided from the mouth shape parameter generation unit 2 to express the desired shape. As the mouth shape parameters, parameters expressing the degree of opening of the mouth in the vertical direction, width, lip rounding, and lip thickness are used, and the three-dimensional shape of the mouth portion is formed by these parameters. It should be noted that not only the mouth portion but also the vertex coordinates forming the mouth peripheral portion including the jaw are changed in accordance with the movement of the mouth portion, for example, the jaw is lowered downward depending on the opening degree of the mouth. .

The information of the three-dimensional shape model in which the vertex coordinate values of the mouth and the area around the mouth are changed in this manner is passed to the coordinate changing unit (B) 32. In the coordinate changing unit (B) 32, 3
Each of the three-dimensional shape models is configured based on the parameters of the rotational movement components Ωx, Ωy, Ωz and the parallel movement components Tx, Ty expressing the three-dimensional movement of the head, which are obtained from the dimensional movement detection unit 1. An operation according to the formula shown in Formula 1 is applied to the vertex. As a result, a shape model whose position / direction matches the head portion of the person in the image of the current frame can be obtained. This result is sent to the texture mapping unit 4, and
It is also sent to the three-dimensional motion detection unit 1 to perform three-dimensional motion detection on the image of the next frame.

In the texture mapping unit 4, with respect to the three-dimensional shape model of the head provided from the coordinate correction unit 3 of the shape model, the group of triangles forming the mouth and the area around the mouth is
Brightness and color information corresponding to the mouth of the person and the area around the mouth, that is, texture information is added to generate a natural mouth shape image. FIG. 7 is a block diagram for explaining an example of the operation of the texture mapping unit 4. In FIG. 7, 41 is a mapping coordinate calculation unit, 42 is a texture providing unit, 43 is a memory (three-dimensional shape model), and 44 is a memory (image). First, before performing the mouth shape correction operation, at least one image of the face portion of the character is prepared in advance in the memory 44 as a pre-processing, and the head corresponding to the shape of the face portion in this image is prepared. A three-dimensional shape model of the part is obtained and stored in the memory 43. At this time, the brightness / color information to be given to each triangle, that is, the texture information is determined by the correspondence relationship between each triangle in the shape model and the face portion in the image. As a result, in the lip part of the shape model,
The brightness / color of the lips in the image is assigned to the chin of the shape model, and the brightness / color of the chin in the image is assigned to the chin. Next, when actually correcting the mouth shape, in the mapping coordinate calculation unit 41, for the triangle group of the mouth and the peripheral portion of the mouth of the three-dimensional shape model of the head given from the coordinate correction unit 3 of the shape model, For each triangle, the corresponding triangle in the shape model stored in the memory 43 is obtained, and the triangle in the shape model given from the coordinate correction unit 3 of the shape model is stored in the memory 44. The coordinate position of which part of the texture is given is calculated. This coordinate position is sent to the texture providing unit 42, the texture information of the corresponding position is read from the image stored in the memory 44, and is added to the shape model. At this time, the shape model is a three-dimensional model, but by projecting this on a two-dimensional plane,
As a three-dimensional image, a corrected image of the mouth and the area around the mouth is obtained.

The synthesizing unit 5 rewrites the mouth and the mouth peripheral portion of the original image obtained from the input terminal 90 with the mouth shape-corrected image obtained from the texture mapping unit 4. Since the range of rewriting is clearly identified as the mouth and the part around the mouth expressed by the three-dimensional shape model of the head, rewriting is performed only for the mouth and the part around the mouth. The obtained image after the mouth shape correction is output to the terminal 99.

Next, an example of the operation of the mouth shape parameter generator 2 will be described with reference to the block diagram of FIG. In FIG. 8, 21 is a phoneme / mouth shape correspondence table, 22 is a mouth shape parameter setting unit, and 23 is a parameter setting unit for each frame. First, the phoneme / mouth shape correspondence table 21 stores phonemes in each language and mouth shape feature information corresponding to each phoneme in advance. Here, the mouth shape characteristic information is information representing the opening of the mouth, the degree of rounding, the position of the chin, and the like. Based on the type of phoneme input from the terminal 91, the mouth shape characteristic information corresponding to the phoneme is read from the table and sent to the mouth shape parameter setting unit 22.
The mouth shape parameter setting unit 22 sets the parameters necessary for operating the vertex coordinates of the mouth and the periphery of the mouth of the three-dimensional shape model based on the mouth shape feature information of each phoneme. As the parameters, for example, parameters relating to the width of the mouth, the degree of opening, the degree of rounding of the lips, the thickness of the lips, and the position of the chin are used. One set of mouth shape parameters obtained here is given for each phoneme. On the other hand, as the original image, for example, in the case of an NTSC TV signal, 3
In order to obtain a natural output image that is composed of 0 frame images and the mouth shape is corrected, it is necessary to determine the mouth shape parameter in each frame every 1/30 seconds. The operation for this purpose is performed by the parameter setting unit 23 in units of frames. Information on the duration of each phoneme given from the terminal 92 and the mouth shape parameter setting unit 22
1 from the mouth shape parameter of the corresponding phoneme given by
The mouth shape parameter value at each time point is determined every 30 seconds. At this time, in order to obtain a smooth mouth shape change between adjacent phonemes, in the frame near the time when the phoneme changes, the mouth shape parameter is smoothed with the mouth shape parameter values in the preceding and following frames. Give.

[0027]

As described above, according to the present invention, the position and orientation of the mouth portion are obtained after detecting the three-dimensional movement of the human face portion in the original image, and the result and the result are added to the original image. The shape of the three-dimensional shape model of the person's head is corrected using the mouth shape parameter obtained from the phoneme type and the duration information about the voice, and the mouth of the character and the character's mouth stored in advance on this shape model are corrected. The mouth shape image is synthesized by adding texture information around the mouth, and using this mouth shape image,
Rewrite the image of the person's mouth and the area around the mouth in the original image.
As a result, a specific method and apparatus for generating a natural moving image having a mouth shape corresponding to voices of different languages are provided, and it is possible to match the mouth shape change and the voice and obtain a high-quality output image. It will be possible.

The present invention relates to the field of image communication such as TV telephones and TV conferences in which sounds in a plurality of languages are added to one video signal, a movie / broadcasting TV, a video tape, and a CD-ROM. It can be used for appropriate correspondence between the change in the mouth shape of the characters and the sound in the storage type video media such as, and the effect is extremely large.

[Brief description of drawings]

FIG. 1 is a block diagram corresponding to an embodiment of the invention.

FIG. 2 is an explanatory diagram for explaining a relationship between an original image and an output image according to the present invention.

FIG. 3 is a block diagram showing an example of an operation for obtaining input information related to voice in the present invention from translated real voice.

FIG. 4 is a block diagram showing an example of an operation for obtaining input information about a voice from original voice before translation according to the present invention.

FIG. 5 is a block diagram showing an example of the operation of the three-dimensional motion detection unit 1 in the present invention.

FIG. 6 is a block diagram showing an example of the operation of the coordinate correction unit 3 of the shape model according to the present invention.

FIG. 7 is a block diagram showing an example of the operation of the texture mapping unit 4 in the present invention.

FIG. 8 is a block diagram showing an example of the operation of the mouth shape parameter generation unit 2 in the present invention.

FIG. 9 is a block diagram showing an example of a mouth shape image synthesizing method according to a conventional technique.

[Explanation of symbols]

 1 three-dimensional motion detection unit 2 mouth shape parameter generation unit 3 coordinate correction unit of shape model 4 texture mapping unit 5 synthesis unit 6 speech recognition unit (A) 7 speech recognition unit (B) 8 translation processing unit 9 speech information generation unit 100 original image 101 original voice 102 output image 103 translated voice 104 combination operation

Claims (2)

[Claims] 1. A digital moving image signal obtained by digitizing a video signal obtained from a television camera, a video tape recorder or the like is used as an original image, and a translated voice obtained by translating the original voice given to the video signal into a different language. The information about the type and duration of each phoneme when decomposed into a phoneme sequence is used as an input regarding the voice, and a three-dimensional shape of the human head is calculated from the three-dimensional shape model expressing the human head shape in the image and the original image. The motion is detected, and on the other hand, the mouth shape parameter is generated for each frame corresponding to the translated speech from the phoneme and the duration information.
The coordinates of the mouth and the peripheral part of the three-dimensional shape model of the head are corrected so that the shape becomes the shape specified by the mouth shape parameter, and the three-dimensional movement parameter of the head for the entire head including the mouth part The coordinates are corrected so as to express a three-dimensional movement according to the following, and then, the texture information of the person's mouth and the surrounding area of the mouth obtained in advance is added to the mouth and the surrounding area of the three-dimensional shape model after the coordinate correction. Furthermore, the image of the mouth and the area around the mouth newly obtained as a result replaces the image of the person's mouth and the area around the mouth in the original image, and a natural image having a mouth shape change corresponding to the translated speech is created. A mouth shape image synthesizing method characterized by synthesizing. 2. A first input terminal for inputting a digital moving image signal obtained by digitizing a video signal obtained from a television camera, a video tape recorder or the like as an original image, and an original sound added to the video signal. Represents the head shape of a person in the image, and second and third input terminals for inputting information of the type and duration of each phoneme when the translated speech translated into different languages is decomposed into a phoneme sequence. A three-dimensional motion detection unit that detects a three-dimensional motion of a human head from a three-dimensional shape model and an original image input from the first input terminal, and inputs from the second and third input terminals The mouth shape parameter generation unit that generates the mouth shape parameter in frame units corresponding to the translated speech from the information of the phoneme type and the duration to be generated, and the mouth and the mouth peripheral portion of the three-dimensional shape model of the head are described above. Mouth shape The coordinates are modified so that the shape is designated by the mouth shape parameter obtained from the parameter generation unit, and the whole head including the mouth portion has a three-dimensional shape obtained from the three-dimensional motion detection unit. The coordinate correction unit of the shape model that corrects the coordinates so as to express a three-dimensional movement according to the movement parameter, and the mouth and the peripheral portion of the mouth of the three-dimensional shape model after the coordinate correction obtained from the coordinate correction unit of the shape model. , A texture mapping unit that gives texture information of a person's mouth and the surrounding area of the person obtained in advance, and an original image obtained from the first input terminal by the image of the mouth and the surrounding area of the mouth obtained from the texture mapping unit A mouth shape image synthesizing apparatus comprising: a synthesizing unit that replaces an image of a mouth of a person inside and a portion around the mouth.