JP3830200B2 - Human image synthesizer - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image synthesizing apparatus using digital processing, and more particularly, to a person image synthesizing apparatus that synthesizes a person image that expresses a mouth shape and a nodding action accompanying utterance.
[0002]
[Prior art]
The following can be shown as a conventional example of this type of technology.
Japanese Laid-Open Patent Publication No. 2-234285 relates to an image composition method for inputting a text expressed as a character string and generating a face moving image having a mouth shape change corresponding to the text.
Here, a speech synthesis method capable of dividing the character string into phoneme strings and outputting a speech feature and duration for each phoneme is used to determine a mouth shape feature corresponding to each phoneme based on the speech feature. Further, the value of the mouth shape parameter for expressing a specific mouth shape is determined according to the mouth shape feature.
Then, with respect to the value of the mouth shape parameter for each phoneme, the mouth shape parameter value given for each frame of the moving image is controlled based on the duration for each phoneme, and the mouth shape suitable for audio output It is disclosed to display a face moving image exhibiting a change.
[0003]
In addition, Shigeo Morishima, Kiyoharu Aizawa, Hiroshi Harashima: “Research on automatic facial expression synthesis based on speech information” 4th NICOGRAPH paper contest papers, pp .139-146, shown in Japan Computer Graphics Association (November 1988). Here, for the input speech information, a method for calculating the logarithmic average power and controlling the degree of mouth opening, and a method for estimating the mouth shape by calculating a linear prediction coefficient corresponding to the formant feature of the vocal tract The following two are proposed.
[0004]
[Problems to be solved by the invention]
In the method of inputting a sentence (character string) in the prior art and determining the mouth shape change corresponding to this, the voice data to be output is converted to voice data prepared in advance as a sentence (character string). Therefore, the input needs to be a character string, and the mouth shape cannot be determined when voice data is directly input or for voice data without character string information.
Further, the Morishima et al. Method can determine the mouth shape, but does not disclose control of facial movement or the like.
The present invention has been made in view of the above-described problems of the prior art. Facial expressions at the time of utterance can be synthesized in accordance with input data other than character strings, and further, correspondence with audio output is made accurately. It is an object of the present invention to provide an image composition device capable of expressing a nodding motion between a mouth shape and a face image.
[0005]
[Means for Solving the Problems]
According to the first aspect of the present invention, in the human image synthesizing apparatus that inputs voice data, generates and outputs a face moving image having a mouth shape change and a face motion corresponding to the voice data, the voice to which the voice data is input An input means; a buffer for temporarily storing voice data input by the voice input means; a mouth shape determination means for determining a mouth shape corresponding to the voice data read from the buffer and determining a mouth shape image corresponding to the mouth shape; Determining the silent portion of the audio data read from the buffer, and determining the mouth shape determination in synchronization with the nodding operation determining means for determining that the silent portion has a nodding operation, and the silent portion determined by the nodding operation determining means. Image synthesizing means for synthesizing the mouth shape image determined by the means and the nodding face moving image to generate a face moving image, and the face moving image generated by the image synthesizing means So as to have a image display means for force, by selecting the read position in the buffer input audio data are sequentially determines a mouth shapes on the basis of the audio data read, also nod from silence audio data The position of the motion is determined , and the expression change accompanying the utterance of the person image can be displayed based on these results.
[0006]
According to a second aspect of the present invention, in the first aspect of the invention, the nodding operation determination means determines the silent portion by using a time change of the logarithmic power of the audio data , and provides effective concrete means. To do.
[0007]
According to a third aspect of the present invention, in the first aspect of the present invention, the input position of the audio data to the buffer and the read position of the audio data used for the nodding operation determination means from the buffer are made to coincide with each other. Operation is possible.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram of an embodiment of a human image composition apparatus according to the present invention.
In FIG. 1, a voice input unit 1 inputs voice with a microphone and performs AD conversion to create voice data, or reads data from a recording medium in which voice data is stored in advance.
The buffer 2 temporarily stores the audio data input from the audio input unit 1. The storage format is FIFO (First In First Out: the first input data is output first). The audio data is sequentially input from the audio input unit 1, and the audio data is sequentially output to the audio output unit 3. The audio output unit 3 converts audio data into an audio signal by DA conversion and outputs audio.
[0009]
FIG. 2 is a conceptual diagram for explaining the operation of an example of the buffer used in the present invention. FIG. 3 is a conceptual diagram for explaining the operation of another example of the buffer used in the present invention.
As shown in FIG. 2, the buffer 2 is a FIFO in which data is shifted to the right every time one piece of data is input from “Ps”, and data is output one by one from the rightmost “Pe”. Alternatively, as shown in FIG. 3, a ring-shaped buffer may be used in which each pointer advances by one each time data is input.
The mouth shape determination unit 4 reads data from a predetermined position “P2” on the buffer 2 and extracts a feature amount of audio data for each frame.
As a method for determining the mouth shape from the feature of the voice, the like method disclosed in Japanese Application flat No. 5-135755 (JP-A-6-348811). That is, the mouth shape is determined according to the low frequency and high frequency components of the voice.
In the example, one frame is 10 (ms), and the number of samples in one frame is N.
Position “P2” on the buffer
P2 = Pe−β × N
In this case, the mouth shape can be determined before the β frame from the time when the voice output is actually output.
The value of β is a value determined from the processing time required to determine the mouth shape, and is adjusted and determined so that the mouth shape corresponding to the actual voice being output is displayed in synchronization. In this example, β = 5.
[0010]
The nodding operation determination unit 5 reads data from a predetermined position “P1” on the buffer, and extracts the feature amount of the audio data for each frame.
Nod action is often seen when sentence breaks or when speaking the word you want to emphasize.
In the following, an example of detecting a sentence break and performing a nodding operation will be described.
An example in the case of using logarithmic power as a feature quantity is shown in FIG.
The solid line is the time change of the logarithmic power when I say "I am Sakamoto of Sharp". The vertical axis is logarithmic power, and the horizontal axis is time.
The threshold value is set to a value that can be used to determine whether or not there is voice input.
The voice input is completed at the time “Te + 10 × Fs” when the logarithmic power of the voice continues below the threshold and continues for Fs frames or more after the voice input is started (when the threshold is exceeded “Ts”). Is determined. Then, when it is determined that the voice is finished, the nodding operation is started.
In the example, one frame is 10 (ms).
The value of Fs is set longer than the silent part in the sentence (the time length of the closure before the plosive), and in the example, Fs = 30.
In the example of FIG. 4, when the utterance content is continuously pronounced,
T1′−T1 <10 × Fs
T2′−T2 <10 × Fs
In the case of no, nodding motion is not generated in the sections [T1, T1 ′] and [T2, T2 ′], but when the voice is divided at “I” or “Sharp”, that is,
T1′−T1> 10 × Fs
T2′−T2> 10 × Fs
In the case of No. 1, a nodding operation is generated in the section [T1, T1 ′] and the section [T2, T2 ′].
If the number of samples in one frame is N, the position “P1” on the buffer is
P1 = Pe− (Fs + α) × N
In the case of the nod operation, the nod operation can be determined no more than α frames before the time “Te” when the output of the voice ends.
The value of α is determined by adjusting so that a nodding operation is performed immediately before the end of utterance at a sentence break.
In this example, α = 20.
Since the output from the audio input unit 1 is input to the audio output unit 3 through the buffer 2, there is a difference between the audio input and the audio output when inputting audio in real time. In order to minimize this deviation, “Ps” and “P1” are preferably matched.
Image synthesizing unit 6 in synchronization with the the nodding operation determined by the operation determining unit 5 before Symbol nods to generate a combined face moving image and the face moving image nods and determined mouth shape of the image in the mouth shape determination section 4 ing.
More specifically, the nod action expresses the nod action by preparing a plurality of nod action animations as shown in FIG. 5 and playing them continuously.
[0011]
【The invention's effect】
According to the present invention, a voice image sent in real time via communication or accumulated voice data is displayed as an image in synchronism with the voice as if a human image is being spoken. be able to.
[Brief description of the drawings]
FIG. 1 is a block diagram of an embodiment of a human image composition device of the present invention.
FIG. 2 is a conceptual diagram for explaining the operation of an example of a buffer used in the present invention.
FIG. 3 is a conceptual diagram for explaining the operation of another example of a buffer used in the present invention.
FIG. 4 is a diagram for explaining an example in the case where logarithmic power is used as a feature amount of a nodding operation.
FIG. 5 is a diagram showing an example of a person image creation screen for performing a nodding operation.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Audio | voice input part, 2 ... Buffer, 3 ... Audio | voice output part, 4 ... Mouth shape determination part, 5 ... Nodding operation | movement determination part, 6 ... Image composition part, 7 ... Image display part.

Claims (3)

In a human image synthesizer that inputs voice data and generates and outputs a face moving image having a mouth shape change or a face motion corresponding to the voice data, voice input means for inputting the voice data, and the voice input A buffer for temporarily storing the audio data input by the means, a mouth shape determining means for determining a mouth shape corresponding to the sound data read from the buffer, and determining a mouth shape image corresponding to the mouth shape; A nodding motion determination means for determining a silent portion of the data and determining that the silent portion has a nodding motion; and a mouth shape image determined by the mouth shape determination means in synchronization with the silent portion determined by the nodding motion determination means Image synthesizing means for synthesizing a nodding face moving image to generate a face moving image, and an image display means for outputting the face moving image generated by the image synthesizing means Thieves Monoga image synthesizing apparatus characterized by having a. The nodding motion determination means, using said time variation of the logarithmic power of the audio data, human Monoga image synthesizing apparatus according to claim 1, characterized in that so to determine the silence. An input position of the audio data to the buffer, the nodding motion determination means human Monoga image synthesizing apparatus according to claim 1, characterized in that so as to match the read position from the buffer of the audio data for use in .

Images