JP3299797B2 - Composite image display system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION The present invention can transmit a message to a partner by a synthesized moving image and a synthesized voice of a face in which a sender is talking like a TV phone, just by transmitting text (text) data. BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a composite image display system applicable to AV (audio / video) e-mail and the like, and more particularly to a composite image display system that can convey to a display side an impression of voice quality and facial expression intended by a text creation side. It is.

[0002]

2. Description of the Related Art A technique for freely generating and producing a synthesized speech corresponding to arbitrary sentence (text) information is called ruled speech synthesis, and a ruled speech synthesis apparatus for realizing this is already manufactured. I have. This rule speech synthesis technology has been applied in various fields to improve the interface between humans and machines. In recent years, similar to speech synthesis, technology has been developed to generate a moving image of a person including mouth movements when speaking from arbitrary sentence information by analyzing the sentence information. Is combined with the above-described speech synthesis technology, a more natural interface can be realized.

For example, when such a technology of synthesizing a voice and a face moving image is applied to an e-mail, a data file such as a face image of the mail sender is prepared in advance on the receiving side, so that a conventional method is used on the screen of the receiving side. In contrast to an e-mail whose text is simply displayed on the screen, an expressive message such as a moving image of the face where the mail sender is speaking appears and is read out with a synthetic voice can be transmitted to the recipient.

[0004] Fig. 4 shows an example of the configuration of a voice / moving image output device that synthesizes a voice and a face moving image based on such a sentence and outputs it. In FIG. 4, reference numeral 1 denotes a sentence decomposition unit to which sentence (text) information is input. The sentence decomposition unit 1 analyzes the input sentence information, generates pronunciation control data for voice output, and generates a rule speech synthesis unit. 2 and output to the voice / mouth shape converter 3. For example, when a sentence “I'm here” is input as sentence information, it is written as “T, A, D, A, I, M,
A ”is decomposed into phoneme data including a vowel and a consonant and output.

The rule speech synthesizer 2 is a device for generating and outputting a synthesized speech for reading out a sentence based on phoneme data of an arbitrary sentence.

[0006] The voice / mouth shape conversion unit 3 is a device for converting phoneme data of an arbitrary sentence into a mouth shape code sequence for representing a series of mouth movements when the sentence is pronounced. Examples of the mouth-shaped code are A (a vowel), I
(A of vowel), U (V of vowel), E (E of vowel), O
There are seven types of vowels (e), S (consonant), and C (closed mouth). Mouth-shaped images are prepared in advance for each of these mouth-shape codes. For example, when the above-mentioned “Ima-san” sentence is input as sentence information, based on the phoneme data “TADAIMA” of the sentence,
"T" → mouth symbol S, "A" → mouth symbol A, "D" →
The mouth shape code S, “I” → the mouth shape code I, “M” → the mouth shape code C, “A” → the mouth shape code A, respectively, and outputs them to the image display control unit 5 as a mouth shape code sequence.

[0007] In the image memory 6, synthetic image data is filed. As the combined image data, one file is obtained by combining the above-shoulder image for one frame of the speaker and the data of the seven types of mouth area images corresponding to the seven types of mouth shape codes synthesized based on the image. And

[0008] The pronunciation time calculation unit 4 calculates the time until each syllable is to be pronounced when synthesizing speech using exactly the same algorithm as the rule speech synthesis unit 2 based on the pronunciation control data from the sentence decomposition unit 1. calculate. That is, when the input sentence is synthesized by the rule speech synthesis unit 2 and output as a sound, the timing of the break of each syllable constituting the sentence is estimated from the beginning of the sentence. The result is output to the image display controller 5.

An image display control unit 5 selects a mouth shape image corresponding to the mouth shape code of the corresponding syllable from the image memory 6 when the sound timing of each syllable arrives based on the timing signal from the sound emission time calculation unit 4. The image display is controlled so that the image is output. That is, synchronization control is performed so that the movement of the mouth of the speaker displayed on the screen matches the sound pronounced by the regular sound synthesis unit 2, that is, the synthesized sound and the face moving image can be synchronized. It is.

The parameter input unit 7 includes a voice quality of the voice synthesized by the rule voice synthesis unit 2, a display location of the face moving image on the screen,
This is a part for inputting various parameters such as a display magnification using a keyboard or the like. Parameters relating to synthesized speech are passed to the regular speech synthesis unit 2, and parameters relating to face moving images are passed to the image display control unit 5 and the image memory 6. It is.

The operation of the apparatus having the above-described configuration will be described. When the sentence information is input, the sentence information is analyzed by the sentence decomposing unit 1 and the phoneme data is collectively passed to the regular speech synthesis unit 2 and is output as a synthesized speech. In parallel with this sounding operation, the phoneme data is converted into a series of mouth-shaped codes by the speech / mouth-shaped conversion unit 3. The pronunciation time calculation unit 4 estimates the time of each syllable break from the phoneme data, and the time data is passed to the image display control unit 5. The image display control unit 5 adjusts the timing of the mouth-shaped code to the sounding timing of each syllable, and corresponds to the mouth-shaped code obtained by the voice / mouth-shaped conversion unit 3 from among the images of each mouth-shaped code developed on the image memory 6. The facial moving image data is transferred to the VRAM, and the facial moving image of the speaker is displayed on the screen of the display device via the VRAM. As a result, the sentence information is transmitted to the receiver as a message composed of a synthesized voice that actually produces the speech and a face moving image of the speaker whose mouth movement timing is included in the synthesized voice.

The apparatus shown in FIG. 4 is realized as a small and economical system by using a conventional small speech synthesis unit for the regular speech synthesis unit 2 and using a personal computer or the like for the other parts. can do.

[0013]

When such a voice / face moving image output apparatus is realized on a personal computer, a composite image is prepared in advance and the images are input as described above in order to reduce the processing amount. It is common practice to switch and display according to the sentence sentence. In generating a synthesized voice and a face moving image in these devices, parameters such as voice quality, a display position of an image on a screen, and a display magnification are set in advance as initial values in a display system (parameter input unit). 7) is used.

As described above, in the conventional apparatus, the voice quality of the synthesized voice and the generation mode of the face moving image are set in advance on the receiving side, and the person and voice quality of the face image registered in advance are, for example, a message. If this is not the case, it will give an unnatural feeling to the viewer.

Also, as typified by the use of this apparatus for e-mail, etc., a person who created text information and a composite image and a person who actually displays the text information as a voice and a moving image and sees it. Is different, the voice information and the image size are not always displayed and displayed on the receiving side with the voice quality and image size desired by the person who created the sentence information and the composite image. May give the recipient a completely different impression.

In other words, in the conventional apparatus, the impression given to the person on the display side from the voice quality and the appearance of the face in transmitting the message by voice and moving image is determined by the parameters set in advance on the display side. It was not possible to accurately convey the impression expression intended by the person on the side of creation to the person on the display side.

The present invention has been made in view of such a problem, and an object of the present invention is to display a synthesized voice or a face synthesized image based on text information on a display side, and to create a text or the like on the display side. It is intended to enable display as intended by the person.

[0018]

FIG. 1 is an explanatory view of the principle according to the present invention. In the present invention, as one form,
In a synthetic image display system for generating a synthetic voice corresponding to a corresponding voice and a synthetic moving image of a face of a person whose mouth moves in accordance with the synthetic voice from arbitrary text information, a synthetic motion image data of a face is generated on a text information generating side. Means for generating synthesized moving image data, and parameter adding means for adding various parameters for determining the generation mode of the synthesized voice and the synthesized moving image on the display side to the synthesized moving image data generated by the synthesized moving image data generating means And sentence information, and synthetic moving image data created by the synthetic image data creating means,
A transmission unit for transmitting, to the display side, various parameters added by the parameter addition unit to the composite moving image data as one transmission data.

The above-mentioned various parameters may include at least a parameter relating to the sound quality of the synthesized voice and a parameter relating to a display magnification and a display position of the synthesized moving image.

In the present invention, as another form,
Receiving the transmission data transmitted from the transmission means, transmitting data input means for separating synthesized moving image data, text information, and various parameters included in the transmission data from each other, and generating synthesized speech based on the text information; Voice synthesizing means for generating and outputting, an image memory for filing the synthesized moving image data separated by the transmission data input means, and a mouth shape code representing a series of mouth shape movements when the sentence information is uttered as the sentence information Conversion means for converting into a sequence of the speech, and pronunciation time calculation means for calculating the pronunciation time of each syllable of the synthesized speech output from the speech synthesis means based on the sentence information and estimating the timing of each speech break, A display for controlling the display image to be switched to a mouth-shaped image corresponding to the mouth-shaped code from the converting means at the timing of the break of each syllable estimated by the sounding time calculating means; And control means, a parameter input means for sending to the internal circuit corresponding to various parameters that have been separated by said transmission data input means, the composite image display system having a are provided.

[0021]

According to the combined image display method of the present invention, when a face image required for display is combined on the transmitting side, the display magnification, display position, Embed voice quality and other parameters of synthesized speech. On the display side, a value embedded in the image data is used as an initial value of the system. As a result, the synthesized speech and the synthesized image can be generated on the display system side as intended by the person who synthesized the face image.

In a combined image display system according to another aspect of the present invention, the transmission data received by the transmission data input means is separated into combined moving image data, text information, and various parameters, and the text information is analyzed by the text decomposition means. To generate sound control data, generate and output synthesized speech based on the sound control data by the sound synthesizer, file the received synthesized moving image data in the image memory, and convert the sound control data into a mouth code by the converter. The pronunciation time of each syllable to be pronounced by the speech synthesis means is calculated by the speech synthesis means based on the pronunciation control data by the pronunciation time calculation means, and the timing of each syllable break is estimated. The mouth shape image of the syllable is controlled to be read from the image memory in accordance with the timing signal of the syllable, and the received various parameters are input into the parameter input means. Human production aspects sentence creation side sends the corresponding internal circuit as synthesized speech composite image made to be intended in.

[0023]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 2 shows a voice / face moving image output apparatus using a composite image display system according to an embodiment of the present invention. FIG.
In the above, the sentence decomposition section 1, the ruled speech synthesis section 2, the voice / mouth shape conversion section 3, the pronunciation time calculation section 4, the image display control section 5, and the image memory 6 are the same as those described in the aforementioned conventional example.

A difference from the conventional apparatus is that, in addition to the original text information, the transmission data sent from the transmitting side has a face image desired by the transmitting side to be synthesized and displayed on the receiving side. And the synthesized image data such as the mouth shape image and the display magnification, display position, voice quality, and other parameters on the screen desired by the person are embedded in the synthesized image data.

FIG. 3 shows the concept of processing for embedding these parameters in transmission data on the transmission side. Perform mapping on the face model based on the original image of the face image desired to be displayed on the receiving side, create synthetic image data using the parameters of each mouth shape, and display magnification, display position, voice quality, etc. Embed parameters related to the impression given to the recipient. Create text information separately from this,
Both are sent to the receiving side as transmission data. In this case, once the composite image data in which the parameters are embedded is sent, the text information only needs to be sent repeatedly.

On the receiving side, the transmission data is input to a transmission data input unit 8, where it is separated into text information, composite image data, and various parameters. The text information is transmitted to the text decomposition unit 1, and the composite image data is stored in an image memory. 6, the various parameters are sent to the parameter input unit 7, respectively.

When the parameter input unit 7 receives the various parameters, the parameter input unit 7 examines the various parameters. The parameters such as the voice quality related to the speech synthesis are input to the rule speech synthesis unit 2 and the parameters related to the image such as the display magnification and the display position of the image. It is sent to the image display control unit 5 and the image memory 6, respectively.

With this configuration, on the receiving side,
As the facial image to be displayed and the parameters to be embedded as the initial values of the system, the facial image desired by the transmitting side, the display magnification, the display position, the voice quality, and other parameters can be used. Therefore, the message can be displayed on the display system on the receiving side with the voice and image as intended by the person on the transmitting side.

In carrying out the present invention, various modifications are possible. For example, in the above embodiment, the mouth shape is 7
Although the case of using different types of images has been described, the present invention is not limited to this. Of course, in order to synthesize a more natural movement of the mouth, the types of images of the mouth shape may be further increased. In the above-described embodiment, the movement of the mouth area is taken as the movement part of the face image to be synthesized on the display side. However, the movement is not limited to this. For example, in addition to the movement of the mouth, the movement of the eyes according to the sentence, etc. Is changed, a more expressive AV message can be sent to the receiving side.

In the above-described embodiment, the case where the present invention is applied to AV electronic mail has been described. However, the present invention is not limited to this, and can be applied to a voice / face moving image output device alone. Yes, or if, for example, speech recognition technology allows real-time recognition of phonemes of uttered voices, a simulated videophone that can display the facial expression of the speaker's face to the recipient side as a moving image simply by making a normal telephone call It is also possible to apply to services such as.

[0031]

As described above, according to the present invention, when a synthesized speech or a face synthesized image is displayed on the receiving side based on the sentence information, the display is performed as intended by the person on the sending side of the sentence. Becomes possible.

[Brief description of the drawings]

FIG. 1 is an explanatory view of the principle according to the present invention.

FIG. 2 is a diagram showing an audio / moving image output device using a composite image display system as one embodiment of the present invention.

FIG. 3 is a diagram illustrating a processing concept on a sending side according to the embodiment system.

FIG. 4 is a diagram showing a conventional audio / video output device.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Sentence decomposition part 2 Rule speech synthesis part 3 Voice / mouth shape conversion part 4 Onset time calculation part 5 Image display control part 6 Image memory 7 Parameter input part 8 Transmission data input part

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-2-234285 (JP, A) JP-A-2-196585 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06F 3/00 G06T 13/00 G06T 15/70 G10L 13/00 G10L 21/06 H04N 7/14-7/15

Claims (3)

(57) [Claims] 1. A synthetic image display system for generating a synthetic voice corresponding to an arbitrary text information and a synthetic moving image of a face of a person whose mouth moves in accordance with the synthetic voice. create a dynamic image data
Means for generating synthesized moving image data, and various parameters for determining the generation mode of the synthesized voice and the synthesized moving image on the display side.
Parameter to be added to the synthetic moving image data created by means
Data adding means, text information, and the composite image data
The composite moving image data and the parameter
The various parameters added by the
Composite image display system, characterized in that a transmission unit for transmitting to the display side as the transmission data. 2. The method according to claim 2, wherein the various parameters are at least
And parameters related to the audio quality of the serial synthesized speech, the composite image display system of <br/> claim 1, further comprising a parameter related to the display magnification and the display position of the synthesized moving image. 3. A composite image display system according to claim 1 or 2.
Transmission data transmitted from the transmission means in the system.
Receiving the data, a transmission data input means for separating synthesized moving image data included in the transmission data, text information, various parameters to one another, and speech synthesis means for generating and outputting a synthesized speech based on the text information, An image memory for filing the combined moving image data separated by the transmission data input means, and a converting means for converting the sentence information into a series of mouth-shaped codes representing a series of mouth-shaped movements when the sentence information is uttered; A pronunciation time calculation means for calculating a pronunciation time of each syllable of the synthesized speech output from the speech synthesis means based on the sentence information and estimating a timing of a break of each voice; Display control means for controlling switching of a display image to a mouth shape image corresponding to a mouth shape code from the conversion means at the timing of a syllable break; Composite image display system having a, a parameter input means for sending to the internal circuit corresponding to various parameters.