JPH09274666A - Portrait synthesizer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a human interface with no sense of incompatibility by inputting voice data, generating a mouth form change or nodding action corresponding to these data synchronously with voices, and displaying an image just like a portrait speaks. SOLUTION: Data from a voice input part 1 are temporarily stored in a buffer 2 and the contents of the buffer are read out, converted to voice signals and outputted from a voice output part 3. The mouth form corresponding to the voice data read out of the buffer is decided by a mouth form deciding part 4, the odding action is decided from the silent term of voice data by an odding action deciding part 5 and based on these decided results, the moth form and an odding action image are synthesized by an image synthesizing part 6 and displayed on an image display part 7.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing device by digital processing, and more particularly to the device for synthesizing a human image expressing a mouth shape and nodding motion associated with utterance.

[0002]

2. Description of the Related Art The following can be shown as a conventional example in this type of technology. JP-A-2-2342
No. 85 bulletin inputs a sentence expressed as a character string,
The present invention relates to an image synthesizing method for generating a face moving image having a corresponding mouth shape change. Here, the character string is divided into phoneme strings, and a speech synthesis method capable of outputting a speech feature and duration for each phoneme is used to determine a mouth shape feature corresponding to each phoneme based on the speech feature. Then, the value of the mouth shape parameter for expressing a specific mouth shape is determined according to the mouth shape feature. Then, with respect to the value of the mouth shape parameter for each phoneme, the value of the mouth shape parameter given for each frame of the moving image is controlled based on the duration for each phoneme, and the mouth shape suitable for voice output is controlled. It is disclosed to display a moving face image that changes.

The method of estimating the corresponding mouth shape change by inputting speech is as follows: Shigeo Morishima, Kiyoharu Aizawa, Hiroshi Harashima: "Study on Automatic Synthesis of Facial Expressions Based on Speech Information" 4th NICOGRAPH Paper Contest Paper Shu, pp.139
~ 146, Japan Computer Graphics Association (1988 1
January). Here, a method of calculating the logarithmic average power of input speech information to control the mouth opening degree, and a method of calculating a linear prediction coefficient corresponding to the vocal tract formant feature to estimate the mouth shape There are two proposals.

[0004]

In the method of inputting a sentence (character string) and determining the mouth shape change corresponding to this in the prior art, the output voice data is previously written as a sentence (character string). This is to convert the prepared data into voice data, and the input must be a character string, and the mouth shape is determined when the voice data is directly input or for voice data without character string information. I can't.
In the method of Morishima et al., The mouth shape can be determined, but the control of the movement of the face is not disclosed. The present invention has been made in view of the above-mentioned problems of the conventional technology, and it is possible to synthesize a facial expression at the time of utterance according to input data other than a character string, and further, it is possible to accurately associate with a voice output. It is an object to be solved to provide an image synthesizing device capable of expressing a nod motion of a mouth image and a face image.

[0005]

According to a first aspect of the present invention, input is made from voice data input means, voice output means, image generation means for generating a face moving image, image display means, and the input means. In an image synthesizing device for a person or the like having processing / controlling means for processing and synthesizing a voice signal and an image from the image generating means and outputting and displaying by the voice output means and the image display means The processing / control means estimates the mouth shape corresponding to the voice from the voice data read from the buffer and the nod motion associated with the utterance, and the image generation means according to the estimation result. The image from is mixed and the reading position on the buffer where the audio data is sequentially input is selected. The state is estimated before the voice is output, and the nodding motion is estimated from the silent period of the voice data, and these results enable the facial expression change accompanying the utterance of the human image to be displayed. .

According to a second aspect of the present invention, in the first aspect of the invention, the nodding motion is estimated by using the temporal change of the logarithmic power of the voice signal, and an effective embodying means is provided. .

According to a third aspect of the present invention, in the first aspect of the present invention, the input position of the voice data to the buffer and the read position of the voice data used for the estimation of the nod motion from the buffer are made to coincide with each other, and the position of the real time is read. It is possible to operate.

[0008]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of an embodiment of a person image synthesizing apparatus of the present invention. In FIG.
The voice input unit 1 inputs voice with a microphone and performs AD conversion to create voice data, or reads data from a recording medium in which voice data is stored in advance. The buffer 2 temporarily stores the voice data input from the voice input unit 1. The storage format is FIFO (Firs
t In First Out: The first input data is output first), the audio data is sequentially input from the audio input unit 1, and the audio data is sequentially output to the audio output unit 3. The voice output unit 3 converts the voice data into a voice signal by DA conversion and outputs a voice.

FIG. 2 is a conceptual diagram for explaining the operation of an example of the buffer used in the present invention, and FIG. 3 is a conceptual diagram for explaining the operation of another example of the buffer used in the present invention. is there. As shown in FIG. 2, the buffer 2 is a FIFO that shifts data one by one each time data is input from “Ps” and outputs data one by one from the rightmost “Pe”. Alternatively, as shown in FIG. 3, a ring-shaped buffer may be used in which each pointer advances one by one each time data is input. The mouth shape determining unit 4 reads the data from a certain position "P2" on the buffer 2 and extracts the feature amount of the audio data for each frame. As a method of determining the mouth shape from the feature amount of the voice, the method disclosed in Japanese Patent Laid-Open No. 5-135755 is used. That is, the mouth shape is determined according to the low-frequency and high-frequency components of the voice. In the example, one frame is 10 (ms), and the number of samples in one frame is N. When the position “P2” on the buffer is set to P2 = Pe−β × N, the mouth shape can be determined before β frames from the time when the output of the voice is actually output. The value of β is a value determined from the processing time required to determine the mouth shape, and is adjusted and determined so that the mouth shape corresponding to the actual voice output is displayed in synchronization. In this example, β = 5.

The nod motion determination section 5 reads the data from a predetermined position "P1" on the buffer and extracts the feature amount of the audio data for each frame. Nodding motions are often seen at sentence breaks or when uttering a word to be emphasized. In the following, an example of detecting a sentence break and performing a nod operation will be described. An example of using logarithmic power as the feature amount is shown in FIG. The solid line is the time change of the logarithmic power when I say "I am Sakamoto of Sharp". The vertical axis represents logarithmic power and the horizontal axis represents time. The threshold value is set to a value with which it can be determined whether or not there is a voice input. After the voice input is started (the time point "Ts" when the threshold value is exceeded), the time point "T" when the logarithmic power of the voice is less than the threshold value and continuously continued for Fs frames
It is determined that the voice input is completed by "e + Fs". Then, the nod operation is started when it is determined that the voice is completed. In the example, one frame is set to 10 (ms).
The value of s is set to be longer than the silent part of the sentence (the time length of the closure before the plosive sound), and Fs = 30 in the example. In the example of FIG. 4, when the utterance content is continuously pronounced, that is, when T1′-T1 <10 × Fs T2′-T2 <10 × Fs, the interval [T1, T1 ′] or the interval [T2, T2 ′ is obtained. ]
Then, when nodding motion is not generated, but when uttering by dividing at "I" or "Sharp", that is, when T1'-T1> 10xFs T2'-T2> 10xFs, the interval [T1, T1 '] and interval [T2, T2']
Then, a nod motion is generated. If the number of samples in one frame is N, and the position “P1” on the buffer is set to P1 = Pe− (Fs + α) × N, the nod motion is α frames before the point “Te” at which voice output ends. The nod motion can be determined. The value of α is determined by separating sentences so that the nod action is performed immediately before the utterance ends. In this example, α = 20. Since the output from the voice input unit 1 is input to the voice output unit 3 through the buffer 2, when voice is input in real time, a gap occurs between the voice input and the voice output. In order to reduce this shift as much as possible, it is preferable that "Ps" and "P1" match. The image synthesis unit 6 synthesizes the mouth shape determined by the mouth shape determination unit 4 and the nod motion determined by the nod motion determination unit 5 to generate a face image. Specifically, as for the nod motion, a plurality of animations of the nod motion as shown in FIG. 5 are prepared, and the nod motion is expressed by continuously reproducing the animation.

[0011]

According to the present invention, it is as if a human image were talking to voice data sent in real time or accumulated voice data via communication.
The facial expression can be displayed as an image in synchronization with the voice.

[Brief description of drawings]

FIG. 1 is a block diagram of an embodiment of a person image synthesizing apparatus of the present invention.

FIG. 2 is a conceptual diagram for explaining an operation of an example of a buffer used in the present invention.

FIG. 3 is a conceptual diagram for explaining the operation of another example of the buffer used in the present invention.

FIG. 4 is a diagram for explaining an example of a case where logarithmic power is used as a feature amount of nodding motion.

FIG. 5 is a diagram showing an example of a screen for creating a person image for performing a nod action.

[Explanation of symbols]

1 ... voice input unit, 2 ... buffer, 3 ... voice output unit, 4 ...
Mouth shape determination unit, 5 ... nod motion determination unit, 6 ... image composition unit, 7 ... image display unit.

Claims (3)

[Claims] 1. A voice data input means, a voice output means, an image generation means for generating a face moving image, an image display means, a voice signal input from the input means, and an image from the image generation means. In an image synthesizing device for a person or the like having a processing / controlling means for processing / synthesizing and outputting and displaying by the voice output means and the image display means, a buffer for temporarily holding voice data from the input means is provided, The control means estimates the mouth shape corresponding to the voice from the voice data read from the buffer and the nod motion associated with the utterance, and synthesizes the image from the image generation means according to the estimation result. An image synthesizing device for a person or the like. 2. The image synthesizing apparatus for a person or the like according to claim 1, wherein the nodding motion is estimated by using a temporal change of logarithmic power of the audio signal. 3. The image of the person or the like according to claim 1, wherein the position of inputting the voice data to the buffer and the position of reading the voice data used for estimating the nod motion from the buffer are made coincident with each other. Synthesizer.