JP2003037826A - Substitute image display and tv phone apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a TV phone apparatus that can send a substitute image having natural expression suitable for the contents of voice. SOLUTION: When a transmission target is set to be a substitute image, a voice recognition section 10 carries out specific voice recognition processing to the voice of a speaker that is collected by a microphone 2a in a handset 2, and specifies pronunciation contents (six kinds of Japanese vowels and a Japanese snapping sound). A substitute image generation section 14 selects the substitute image having a mouth shape corresponding to the pronunciation contents specified by the voice recognition section 10 out of a plurality of kinds of substitute images stored into a substitute image memory 12. The selected substitute image is sent to a listener with the voice of the speaker.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a proxy image display device for generating and displaying a proxy image of a speaker in accordance with the content of the voice of the speaker, and the voice of the speaker and the image of the speaker (or the speaker). (Representative image of) is transmitted and received to and from each other, the present invention relates to a videophone device for making a call.

[0002]

2. Description of the Related Art Recently, a so-called videophone device has been put into practical use, which is capable of transmitting and receiving images of each other when making a voice call. . By using such a videophone device, for example, a grandfather or grandmother can talk while looking at the faces of grandchildren who live in distant land, or lovers can talk while looking at each other's faces, Communication can be enriched compared to conventional telephones that exchange only voice.

Further, in a general videophone apparatus, basically, it is possible to select two types, that is, an image obtained by photographing a speaker is sent to the other party as it is, or no image is sent. There is. Also, privacy protection,
Alternatively, from the viewpoint of improving entertainment, a method has also been proposed in which various character images are sent to the other party as a substitute image representing the image of the speaker.

[0004]

By the way, in the conventional videophone apparatus, when a proxy image using various character images or the like is transmitted, the expression of the proxy image to be transmitted is fixed, or It was just a matter of opening and closing the mouth and lacked expressiveness. Therefore, for the other party who is talking while looking at this proxy image, the content of the voice spoken here and the facial expression of the proxy image (particularly mouth movement)
There was a problem that was not suitable and felt unnatural.

The same problem can be said for various devices that display a predetermined proxy image in accordance with the guidance voice. For example, in a vehicle-mounted navigation device, a proxy image using a character is displayed together with a predetermined guidance voice when guiding a route when performing route guidance or when performing various other operations, and it is as if this character gave a guidance voice. Some of them appear to be uttered, but the displayed proxy image has a uniform expression and lacks expressiveness, and it does not match the content of the guidance voice and feels unnatural. There is.

The present invention has been made in view of the above circumstances, and an object thereof is to provide a proxy image display device capable of displaying a proxy image having a natural expression suitable for the contents of voice. Especially. Another object of the present invention is to
It is an object of the present invention to provide a videophone device that can send a substitute image with a natural expression that matches the contents of voice.

[0007]

In order to solve the above-mentioned problems, the proxy image display device of the present invention uses the proxy image generation means to match the contents of the voice of the speaker output from the voice output means. The proxy image of the speaker whose mouth shape changes is generated, and the generated image is displayed by the display means.
Since the proxy image in which the mouth shape is changed according to the content of the voice of the speaker is displayed, it is possible to display the proxy image having a natural expression suitable for the content of the voice.

It is desirable that the above-mentioned substitute image generating means extracts a vowel of a voice output from the voice output means as a target sound and generates a substitute image corresponding to the target sound. Generally speaking, the shape of the mouth when a person speaks is
Almost all of them are decided corresponding to the vowels, as is almost the case in each case. Therefore, by extracting the vowel sound of the voice as the target sound, it is possible to generate a mouth-shaped proxy image that matches the content of the voice with high accuracy and with a small processing load.

Further, it is desirable that the substitute image generation means extracts a vowel sound as a target sound in addition to the vowel sound. In general, when uttering a vowel (sound equivalent to "n" in Japanese), the mouth is closed unlike the case of a vowel. Therefore, by extracting this vowel as the target sound, it is possible to match it with the content of the voice. The mouth-shaped surrogate image can be generated more accurately.

Further, a plurality of types of still substitute images corresponding to the above-described sound of interest are prepared in advance, and the substitute image generation means selects the voice to be output from the plurality of types of still substitute images. It is desirable to generate a proxy image by selecting the corresponding one. Since the proxy image can be generated by a simple process of selecting any one of a plurality of types of still proxy images corresponding to the extracted target sound, the processing load can be reduced.

Further, a plurality of types of mouth shape images corresponding to the above-mentioned sound of interest and a common image other than these mouth shape images are prepared in advance, and the proxy image generating means is configured to select among the plurality of types of mouth shape images. From, select the one that corresponds to the voice that is the output target and synthesize it with the common image,
A proxy image may be generated. In this case, compared to the case of preparing multiple types of still proxy images,
Since the data amount of the prepared image can be reduced, the storage capacity of the memory for storing the data for generating the proxy image can be reduced.

Further, the proxy image generating means generates a proxy image by inserting an interpolated image using the previous and next selected images between these selected images when switching the selected images corresponding to the sound of interest. It is desirable to do. Thereby, the change of the image from the previous selection image to the subsequent selection image becomes smooth, and a more natural expression can be expressed.

Further, it is desirable that the substitute image generation means generates a substitute image corresponding to the sound repellency when the output of the voice by the voice output means is stopped. As a result, when the voice of the speaker is interrupted, the proxy image corresponding to the sound repellency, that is, the proxy image with the mouth closed is displayed, so that the mouth open state is displayed even though the speaker is not speaking. It is possible to prevent the proxy image from being displayed and the display content to be unnatural.

Also, sound collecting means for collecting the voice of the speaker,
A voice recognition means for determining the content of the voice collected by the sound collection means is further provided, and the above-mentioned voice output means performs voice output based on the voice collected by the sound collection means, and the proxy image. It is desirable that the generation unit generate the proxy image based on the content of the voice discriminated by the voice recognition unit. This makes it possible to generate a proxy image corresponding to the content of the speaker's voice in real time by responding in real time.

Further, in the video telephone device of the present invention, the voice of the speaker is collected by the sound collecting means and the image of the speaker is photographed by the photographing means, and the speaker collected by the sound collecting means. The voice recognition means determines the content of the voice, and the proxy image generation means generates a proxy image of the speaker whose mouth shape changes in accordance with the content of the voice determined by the voice recognition means. Then, together with the sound collected by the sound collecting means, the image of the speaker photographed by the photographing means, or the proxy image generated by the proxy image generating means is transmitted to the communication partner by the transmitting means,
The voice and the image sent from the other party are received by the receiving unit, the voice received by the receiving unit is output by the voice output unit, and the image received by the receiving unit is displayed by the display unit. Since the proxy image in which the mouth shape is changed according to the content of the voice of the speaker is generated, the proxy image with a natural expression suitable for the content of the voice can be sent to the call partner.

The above-mentioned voice recognition means determines the vowel of the voice collected by the sound collection means, and the proxy image generation means sets the vowel determined by the voice recognition means as the sound of interest, It is desirable to generate a proxy image corresponding to the sound. As described above, since the shape of the mouth when a human speaks is almost determined corresponding to the vowel, by distinguishing the vowel of the voice and setting it as the focused sound, the processing accuracy is low with a small processing load, It is possible to generate a mouth-shaped surrogate image that matches the content of the voice. Further, the voice recognition unit determines the sound repellency in addition to the vowel of the voice, and the substitute image generation unit sets the sound of interest including the sound repellency. It is desirable to do. As described above, when the utterance is uttered, the mouth is in a closed state. Therefore, by extracting the utterance as the sound of interest, it is possible to more accurately generate the mouth-shaped proxy image that matches the content of the voice. it can.

Further, a plurality of types of still substitute images corresponding to the above-mentioned sound of interest are prepared in advance, and the substitute image generation means selects the voice determined by the voice recognition means from the plurality of types of still substitute images. It is desirable to generate the proxy image by selecting the one corresponding to the contents of. Since a proxy image can be generated by a simple process of selecting any one of a plurality of types of still proxy images, the processing load can be reduced.

Further, a plurality of types of mouth shape images corresponding to the above-mentioned sound of interest and a common image other than these mouth shape images are prepared in advance, and the proxy image generating means sets the plurality of types of mouth shape images. From the above, the proxy image may be generated by selecting the one corresponding to the content of the voice discriminated by the voice recognition means and synthesizing it with the common image. In this case, the amount of image data to be prepared can be reduced as compared with the case where multiple types of still proxy images are prepared, so that the memory for storing the data for generating the proxy image is stored. The storage capacity can be reduced.

Further, the proxy image generating means generates a proxy image by inserting an interpolated image using the selected images before and after the selected image corresponding to the sound of interest, between these selected images. It is desirable to do. Thereby, the change of the image from the previous selection image to the subsequent selection image becomes smooth, and a more natural expression can be expressed.

Further, it is desirable that the substitute image generating means generate a substitute image corresponding to the sound repellency when the sound collection by the sound collecting means is interrupted. Accordingly, when the voice of the speaker is interrupted, it is possible to prevent the display image from being unnatural because the proxy image with the mouth open even though the speaker is not speaking is displayed.

Further, a transmission image setting unit is further provided, and when the transmission image setting unit receives an incoming call, the proxy image first generated by the proxy image generation unit is transmitted from the transmission unit to the other party. An image of the speaker photographed by the photographing means is transmitted in response to the speaker switching instruction, and at the time of transmission, either the substitute image generated by the substitute image generating means or the image of the speaker photographed by the photographing means is transmitted. It is desirable to perform control such that the message is transmitted according to the speaker's selection instruction.

In many cases, the speaker at the time of receiving an incoming call does not want to immediately send his / her own image to the other party, such as when he / she is not dressed and has no hair immediately after getting up. Therefore, it is possible to protect the privacy of the speaker by first transmitting the proxy image when receiving a call and then transmitting the image of the speaker when the speaker switching instruction is issued. Further, at the time of making a call, the speaker can freely set the image to be sent according to his or her will.

Further, an emotion determining means for determining the emotion of the speaker is further provided, and the proxy image generating means generates a proxy image in which the emotion of the speaker determined by the emotion determining means is reflected in the facial expression. Is desirable. By reflecting the emotion of the speaker on the facial expression of the proxy image, the facial expression of the proxy image can be enriched and communication can be facilitated.

Further, it is preferable that the emotion determining means determines the emotion of the speaker using at least one of the voice of the speaker collected by the sound collecting means and the image of the speaker captured by the capturing means. If at least one of the voice of the speaker and the image of the speaker is used, the emotion of the speaker can be determined by various methods. Specifically, if the voice of the speaker is used, the emotion can be determined based on the intonation and speed of the voice, the content of words, and the like. Also, if the image of the speaker is used, for example,
The position and size of each element such as eyes, eyebrows, mouth included in the face,
Alternatively, the emotion can be determined based on the number of blinks.

Further, it is desirable to further include a delay means for delaying the voice transmission time by the time required for the generation of the substitute image by the substitute image generation means.
As a result, it is possible to prevent the voice output timing and the substitute image display timing from deviating from each other and causing the other party to feel uncomfortable.

Further, image input means for capturing an arbitrary character image used as a proxy image is further provided, and the proxy image generation means changes the mouth shape of the character image captured by the image input means to generate the proxy image. Is preferably generated. Since an arbitrary character image can be captured and a proxy image can be generated, the degree of freedom in selecting a proxy image can be increased. Therefore, an image created by a speaker using a digital camera or a scanner, the Internet, or a CD-ROM
Various character images, such as images obtained via a medium such as the above, can be taken in and a proxy image can be generated based on them.

Further, a speaker sex changing means for changing the gender of the speaker by changing the sound quality of the sound collected by the sound collecting means is further provided, and the gender is changed by the speaker sex changing means. The voice after the completion may be transmitted from the transmitting means. This gives, for example,
When a nuisance call is received by a female user (speaker), the nuisance call can be repelled by switching the voice of the talker as if it were a male voice.

[0028]

BEST MODE FOR CARRYING OUT THE INVENTION [First Embodiment] A videophone device according to an embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a diagram showing the configuration of the videophone device according to the first embodiment. The videophone device 10 shown in FIG.
0 includes a camera 1, a handset 2, a display unit 3, an operation unit 4, a control unit 5, a communication processing unit 6, and a memory IF (interface) unit 7.

The camera 1 is for photographing a user (speaker) who makes a call using the videophone device 100. The handset 2 includes a microphone 2a and a speaker 2b, collects a voice uttered by a speaker by the microphone 2a, converts the voice into a voice signal and outputs the voice signal to the control unit 5, and a voice output from the control unit 5. The voice of the other party is output from the speaker 2b based on the signal.

The display unit 3 displays the contents of the operation (for example, telephone number, etc.) and the operating state of the videophone device 100. Further, the display unit 3 displays the image of the other party when the image is sent together with the voice from the other party. The operation unit 4 is for inputting the telephone number of the other party of the call and for giving operation instructions such as various settings, and is provided with various operation keys.

The control unit 5 controls the entire operation of the video telephone 100 in order to transmit and receive voice and images and make a call. Specifically, the control unit 5 controls the speaker's voice and the speaker's image captured by the camera 1 (hereinafter,
A process of transmitting a "self-portrait") or an image (hereinafter, referred to as "surrogate image") as a substitute for this self-portrait to the other party, and receiving the voice or image sent from the other party to display the handset 2 or display. Control such as outputting to the unit 3 is performed. Generally, the control unit 5 includes a CPU, RO
The function is realized by executing a predetermined operation program using hardware such as M and RAM. The details of each component provided in the control unit 5 will be described later.

The communication processing unit 6 carries out a predetermined communication process necessary for making a call with the other party. Specifically, the communication processing unit 6 transmits the image (the substitute image or the self-portrait) and the voice output from the control unit 5 to the other party,
A process of receiving an image and a sound sent from the other party and outputting the image and sound to the control unit 5 is performed.

The memory IF section 7 is an interface section when the control section 5 reads the data stored in the memory card 50. The memory card 50 is a data storage medium composed of a semiconductor memory. In the present embodiment, desired image data is stored in the memory card 50, inserted into the memory IF unit 7, and the control unit 5 is caused to read the image data, whereby the content of the substitute image can be arbitrarily set. You can do it. Details of the proxy image will be described later.

Next, the detailed structure of the control unit 5 will be described. The control unit 5 includes a voice recognition unit 10 and a substitute image memory 1
2, the proxy image generation unit 14, the audio delay processing unit 16, and the transmission image setting unit 18 are provided. The voice recognition unit 10 performs a predetermined voice recognition process on the voice of the speaker collected by the microphone 2a in the handset 2 to determine the pronunciation content. Specifically, the voice recognition unit 10 of the present embodiment.
Are the five Japanese vowels "A,
Six types of pronunciation contents, which are "a, u, d, o" and a vowel "n", are extracted as the sound of interest. For example, in any case where the voice uttered by the speaker is "a", "ka", "sa", ..., "wa", the voice recognition unit 10 determines that the pronunciation content is the vowel "a". Specify.

The substitute image memory 12 stores six types of substitute images (still substitute images) having mouth shapes corresponding to the above-mentioned six types of sound generation contents. Figure 2
6 is a diagram showing an example of six types of proxy images stored in a proxy image memory 12. FIG. As shown in FIG. 2, in the proxy image memory 12, the above-mentioned six types of pronunciation contents “a, a, c,
6) with mouth shapes corresponding to "D, O, N"
Stores types of proxy images. Specifically, FIG.
The proxy image 150a shown in (a) is “A”, the proxy image 150b shown in FIG. 2B is “A”, the proxy image 150c shown in FIG. 2C is “U”, and the proxy image 150c shown in FIG. Proxy image 15
0d is “e”, proxy image 150e shown in FIG. 2 (e) is “o”, proxy image 150f shown in FIG. 2 (f) is “n”,
It corresponds to each.

In the example shown in FIG. 2, a proxy image using a predetermined character image is shown, but in addition to this, the speaker himself / herself can make a face corresponding to each of the six kinds of pronunciation contents. Images captured by the camera 1, images created using a digital camera, a scanner, or the like, or any character image obtained through various media such as the Internet (for example, a male character, a female character, or an animal model). Characters, images of characters appearing in manga, etc.) may be used as proxy images. In addition, some character images or images created by photographing each person's face by each of a plurality of speakers (for example, family members) may be selectively used.

The substitute image generation unit 14 includes the substitute image memory 1
2 from the substitute images stored in the voice recognition unit 10
A proxy image is generated by selecting one corresponding to the pronunciation content specified by. For example, the speaker utters "Kondoha (this time) ..." and the pronunciation is "Oh, n,
., ", The proxy image generation unit 14 determines that the proxy image 150e, the proxy image 150f, the proxy image 150e, and the proxy image 150a shown in FIG.
The proxy images stored in the proxy image memory 12 are selected in this order. The proxy image selected by the proxy image generation unit 14 is transmitted by the communication processing unit 6 to the videophone device (not shown) of the other party.

The voice delay processing section 16 includes the microphone 2a.
The process of delaying the voice of the speaker collected by the predetermined time is performed. Specifically, in the voice delay processing unit 16 of the present embodiment, the pronunciation recognition unit 10 identifies the pronunciation content of the speaker's voice, and the proxy image generation unit 14 creates a proxy image according to the identified pronunciation content. The time it takes to pick
That is, the sound is delayed by the time required to complete the generation of the proxy image. In the videophone device 100 of this embodiment, the voice of the speaker delayed by the predetermined time is transmitted to the other party of the call together with the substitute image.
It is possible to prevent the content of the voice and the facial expression of the substitute image from being deviated from each other and causing a feeling of strangeness to the communication partner.

When transmitting the self-portrait of the speaker,
It is assumed that the delay processing is not performed by the audio delay processing unit 16 or that the predetermined time to be delayed is set to a short value different from the case of the proxy image. Transmission image setting unit 18
Sets whether to send a self-portrait or a proxy image to the other party. Specifically, the transmission image setting unit 18
Is set so that when a call arrives, the proxy image output from the proxy image generation unit 14 is first transmitted to the other party of the call, and then the speaker uses the operation unit 4 to give a predetermined switching instruction. In this case, the self-portrait of the speaker photographed by the camera 1 is set to be transmitted.

Also, the transmission image setting unit 18 takes a photograph by the camera 1 and the proxy image output from the proxy image generation unit 14 in response to a selection instruction made by the user using the operation unit 4 when making a call. One of the selected self-portraits is set as the transmission target. It should be noted that when the user does not give a selection instruction, an image (for example, a surrogate image) preset in advance is automatically set as a transmission target.

As described above, the speaker at the time of receiving an incoming call often does not want to immediately send his / her image to the other party, such as when he / she is not dressed or having hair properly immediately after getting up. There is an advantage that the privacy of the speaker can be protected by transmitting the proxy image and transmitting the image of the speaker when the speaker switching instruction is issued. Further, at the time of making a call, the image to be transmitted is set according to the selection instruction, so that the speaker can freely set the image to be transmitted according to his or her will.

The microphone 2a described above serves as a sound collecting means,
The voice recognition unit 10 serves as a voice recognition means, and substitute image memory 1
2. The proxy image generation unit 14 is a proxy image generation unit, the camera 1 is a photographing unit, the communication processing unit 6 is a transmission unit and a reception unit, the speaker 2b is a voice output unit, the display unit 3 is a display unit, and a voice. The delay processing unit 16 corresponds to the delay unit, the transmission image setting unit 18 corresponds to the transmission image setting unit, and the memory IF unit 7 and the memory card 50 correspond to the image input unit, respectively.

The videophone device 100 of this embodiment has such a configuration, and its operation will be described below. FIG. 3 is a flowchart showing the overall operation procedure of the videophone device 100 of this embodiment. The control unit 5 uses the operation unit 4 to make a call operation (operation for making a call to the other party)
It is determined whether or not has been performed (step 100).

When a predetermined calling operation is performed, step 1
An affirmative decision is made in 00, and the transmission image setting unit 18 in the control unit 5 sets the type of image to be transmitted (self image or proxy image) (step 101). Specifically, the operation unit 4 is provided with an operation key for switching the type of the image to be transmitted, and before (or immediately after) the transmission operation, the speaker uses the operation key to transmit the image to be transmitted. The type of image is selected, and the transmission image setting unit 18 sets the type of image according to the selection instruction. If this selection instruction is not issued, the default image type (eg, proxy image) is automatically selected.

When the type of the image to be transmitted is set, the control section 5 transmits the voice of the speaker to the other party of the call or receives the voice or the image of the other party of the call, and the handset 2 and the display section 3 receive it. Perform predetermined call processing such as outputting (step 1
02). Further, in parallel with the call processing shown in step 102, the control unit 5 performs processing (image transmission processing) of transmitting the self-portrait of the speaker or the substitute image (step 103). The detailed contents of the image transmission process will be described later.

After that, the control section 5 determines whether or not the call is terminated by the other party or the device itself (step 104). While the call is not finished,
A negative determination is made in step 104, and the control unit 5 returns to step 102 described above and continues the predetermined call processing and image transmission processing. Also, when the call ends,
A positive determination is made in step 104, and in this case, the control unit 5 ends the series of processes.

If the predetermined calling operation has not been performed, a negative determination is made in step 100 described above,
In this case, the control unit 5 determines whether or not a call from the other party has arrived (step 105). If no call is received, a negative determination is made in step 105, and in this case, the process returns to step 100 described above, and the subsequent processing is repeated.

When a call is received, an affirmative judgment is made in step 105, and in this case, the transmission image setting unit 18 in the control unit 5 automatically determines the type of image to be transmitted as a proxy image. (Step 106). afterwards,
The control unit 5 moves to the above-mentioned step 102 and performs the subsequent processing.

Next, detailed contents of the above-mentioned image transmission processing will be described. FIG. 4 is a flowchart showing a detailed procedure of the image transmission processing shown in step 103 of FIG. The transmission image setting unit 18 in the control unit 5 determines whether or not the type of image to be transmitted is set to a proxy image (step 200).

If the transmission target is set to the substitute image, an affirmative decision is made in step 200, and the voice recognition unit 10 in the control unit 5 speaks based on the voice signal output from the microphone 2a. A predetermined voice recognition process is performed on the voice uttered by the person to specify the pronunciation content (step 201). As described above, the voice recognition unit 10 according to the present embodiment corresponds to the voice of the speaker by using the five vowels “a, a,
Six types of pronunciation contents, which are "u, d, o" and sound repellency "n", are specified.

The substitute image generation unit 14 constantly outputs the sound content specified by the voice recognition unit 10 (for example, for each sound).
A process of acquiring and selecting a proxy image corresponding to the pronunciation content from the proxy images stored in the proxy image memory 12 is performed (step 202). It should be noted that the proxy image generation unit 14 of the present embodiment, when the result of identifying the pronunciation content is not output from the voice recognition unit 10, in other words, when the sound collection of the voice by the microphone 2a is interrupted, Figure 2
Processing for automatically selecting the substitute image 150f corresponding to the case where the pronunciation content shown in (f) is the sound utterance "n" is performed. Thus, when the voice of the speaker is interrupted, the proxy image 150f, which is the face with the mouth closed, can be transmitted to the other party of the call, so that the mouth is spoken even though the speaker is not speaking. Since the proxy image in the open state is not transmitted to the call partner, it is possible to prevent the call partner looking at the proxy image from feeling uncomfortable.

When the substitute image corresponding to the pronunciation content is selected by the substitute image generation unit 14, the control unit 5 performs a process of transmitting the substitute image to the other party (step 20).
3). Further, in parallel with the processing shown in steps 201 to 203 described above, the transmission image setting unit 18 determines whether or not an operation instruction to switch the type of the image to be transmitted to the self image has been performed (step 204). While the operation instruction to switch the transmission target to the self-portrait is not issued, a negative determination is made in step 204. In this case, the process returns to step 201 described above, and the subsequent processing is repeated.
That is, a substitute image according to the pronunciation content of the speaker's voice is selected at any time and transmitted to the other party.

When an operation instruction to switch the transmission target to the self-portrait is given and an affirmative judgment is made in step 204, or the transmission target is set to the self-portrait from the beginning, a negative judgment is made in step 200 described above. When it is performed, the transmission image setting unit 18 switches the transmission target to the self-portrait.

Next, the control unit 5 activates the operation of the camera 1, captures the image (self-portrait) of the speaker by the camera 1 (step 205), and transmits the captured self-portrait to the other party. Perform (step 206). For example, several to several tens of self-portraits are captured and transmitted every second.

Further, in parallel with the processing shown in steps 205 and 206 described above, the transmission image setting unit 18 determines whether or not an operation instruction to switch the type of the image to be transmitted to the substitute image is issued. The determination is made (step 207). While the operation instruction to switch the transmission target to the substitute image is not issued, a negative determination is made in step 207. In this case, the process returns to step 205 described above, and the subsequent processing is repeated. That is, the self-portrait of the speaker is photographed at a predetermined cycle and transmitted to the other party.

When an operation instruction to switch the transmission target to the proxy image is given, an affirmative decision is made in step 207, and the transmission image setting unit 18 stops the transmission of the self-portrait and sets the transmission target to the proxy image. Switch to. After that, the process returns to step 201 described above, and the subsequent processes are repeated.

As described above, the videophone device 100 of the first embodiment discriminates between the five vowels "a-o" and the sound repellant "n" as the contents of the speaker's voice, and matches them with the determined contents of the voice. Since the proxy image of the speaker with the changed mouth shape is generated, the proxy image with a natural expression that matches the content of the speaker's voice can be sent to the other party. In particular, since the content of the voice is discriminated by focusing on the fact that the shape of the mouth when a human speaks is almost determined corresponding to the vowel and the sound repellency, it is possible to accurately match the content of the voice with a small processing load. A mouth-shaped proxy image can be generated.

6 corresponding to the target sound (vowel + vowel sound)
Types of proxy images (still proxy images) are prepared in advance and stored in the proxy image memory 12, and one of these six types of proxy images corresponding to the content of the voice discriminated by the voice recognition unit 10 is selected. Since the proxy image is generated by selection, the processing is simple, and there is an advantage that the processing load when generating the proxy image can be reduced.

The videophone device to which the present invention is applied is not limited to the above-described embodiment, and various modifications can be made within the scope of the present invention. For example, in the above-described embodiment, as shown in FIG. 2, six types of substitute images corresponding to six types of pronunciation contents including the vowels “a, i, u, e, o” and “n” are combined. However, six types of mouth shape-only images (mouth shape images) according to the pronunciation content and a base face image (common image) are prepared, and one of the mouth shape images is prepared. The common image may be combined to generate the proxy image.

FIG. 5 is a diagram showing the processing contents of a modified example in which the mouth shape image and the common image are combined to generate a substitute image.
As shown in FIG. 5, six mouth shape images 152a to 152f corresponding to six kinds of pronunciation contents are prepared, and a common image 154 serving as a base is prepared in advance, and six mouth shape images are prepared according to the pronunciation contents. By selecting any one of 152a to 152f and synthesizing it with the common image 154, it is possible to generate and transmit a proxy image having a facial expression according to the pronunciation content of the voice. In this case, compared to the case of preparing a plurality of types of proxy images as shown in FIG.
Since the amount of image data to be prepared can be reduced, there is an advantage that the storage capacity of the substitute image memory 12 can be reduced.

In the above-described embodiment, the pronunciation content is specified based on the voice of the speaker who is talking and the corresponding proxy image is transmitted. When a so-called answering machine function for reproducing a response message is used, a proxy image corresponding to the pronunciation content of the response message may be transmitted.

In this case, the speaker first sends a predetermined response message (for example, "I am out of the office. Please talk about your name and business requirements") in advance.
It is input using a and this voice is stored in the internal memory (not shown) of the control unit 5 or the memory card 50. Then, when an incoming call is received, a predetermined number of times (for example, 5
When the call is made more than once, the call is automatically received and step 1
The processing after 06 is performed, and a response message prepared in advance at that time is transmitted to the communication partner, the pronunciation content of this response message is specified by the voice recognition unit 10, and corresponding proxy images are sequentially selected to the communication partner. Just send it.

Further, the emotion of the speaker may be determined, and a proxy image in which the emotion of the speaker is reflected in the facial expression may be generated and transmitted. FIG. 6 is a diagram showing the configuration of a modified videophone device 100a that generates a substitute image in which the emotion of the speaker is reflected in the facial expression. The videophone device 100a shown in FIG.
Has basically the same configuration as the above-described videophone device 100 shown in FIG. 1, but the configuration inside the control unit 5a is different. The following mainly describes the differences between the two.

The control unit 5a includes a voice recognition unit 10, a substitute image memory 12a, a substitute image generation unit 14a, and a voice delay unit 1.
A transmission image setting unit 18 and an emotion determination unit 20 are provided. The emotion determination unit 20 uses a variety of well-known methods based on the image of the speaker captured by the camera 1 and the voice of the speaker collected by the microphone 2a, to determine the emotion of the speaker (“joy, anger, sadness, Easy ”). For example, the emotion determination unit 20 extracts a feature amount such as intonation and speed of the voice based on the voice of the speaker, and also based on the image of the speaker, the eyes, eyebrows, mouth, etc. included in the face. The feature amount such as the position and size of each element, or the number of blinks is extracted, and the emotion of the speaker is determined by integrating these feature amounts. The emotion determination unit 20 corresponds to the emotion determination means.

The emotion may be determined based on only one of the voice and the image of the speaker. Also,
When emotion recognition based on voice is performed, the processing content may partially overlap with the processing by the voice recognition unit 10. Therefore, in such a case, the processing may be shared.

The substitute image memory 12a stores a substitute image having a facial expression corresponding to the type of emotion and having a mouth shape corresponding to each of the six types of pronunciation contents. For example, in the case where there are four types of emotions determined by the emotion determination unit 20, namely “joy, anger, sadness, and comfort”, at least
Twenty-four types of proxy images in which the types of emotions and the types of pronunciation contents are combined are stored in the proxy image memory 12a.

The substitute image generation unit 14a includes a voice recognition unit 10a.
A process of selecting one proxy image corresponding to the pronunciation content specified by and the emotion type determined by the emotion determination unit 20 from the proxy images stored in the proxy image memory 12a is performed. For example, when the emotion is "angry" and the pronunciation content is "a", one proxy image having an angry expression and a mouth shape of "a" is selected.

Next, the operation of the videophone device 100a having the above configuration will be described. Videophone 1
The operation procedure of 00a is basically the same as that shown in FIGS. 3 and 4 described above, and a part of the flow chart shown in FIG. 4 is changed. The specific changes are (1) before the step 202 shown in FIG. 4 (or before step 201), the emotion determination processing by the emotion determination unit 20 described above is added, and (2) step 202 Is to reflect the emotion determination result by the emotion determination unit 20 when the proxy image is selected.

As described above, by reflecting the emotion of the speaker on the facial expression of the substitute image, the facial expression of the substitute image can be further enriched and the communication can be facilitated. Note that, as in the case of replacing the mouth shape image described above, when generating a proxy image that reflects an emotion, the eyes of the proxy image may be partially replaced according to the emotion.

Further, a means for switching the gender of the speaker by changing the sound quality of the voice of the speaker collected by the microphone 2a is provided, and the voice after the gender is switched is transmitted. Good. FIG. 7: is a figure which shows the structure of the video telephone apparatus 100b of the modification which transmits the sound which switched the speaker's sex. The videophone device 100b shown in FIG. 7 has basically the same configuration as the above-described videophone device 100 shown in FIG.
The internal structure is different. The following mainly describes the differences between the two.

The control unit 5b includes a voice recognition unit 10, a substitute image memory 12, a substitute image generation unit 14, a voice delay unit 16, a transmission image setting unit 18, and a speaker gender changing unit 22.
The speaker gender changing unit 22 performs a process of switching the gender of the speaker by performing various processes such as frequency conversion on the voice of the speaker collected by the microphone 2a to change the sound quality. The speaker sex changing unit 22 corresponds to a speaker sex changing unit.

Next, the operation of the videophone device 100b having the above configuration will be described. Videophone 1
The operation procedure of 00b is basically the same as that shown in FIGS. 3 and 4 described above, and when the speaker gives a predetermined gender switching instruction using the operation unit 4, the speaker It suffices to validate the processing by the gender changing unit 22 and transmit the voice after the gender is changed to the other party.

As a result, for example, when a nuisance call is received by a female speaker, the voice of the speaker can be switched and transmitted as if it were a male person, so that the nuisance call can be repelled. You can The speaker gender changing unit 22
When the processing by is effective, it is also preferable to change the content of the substitute image in conjunction with this. For example, when the speaker is a woman and a female character image is used, the character image may be switched to a male character image.

Further, when the proxy image is switched according to the pronunciation content, the proxy image generation unit 14 generates an interpolation image using the proxy images (selected images) before and after, and inserts it between the proxy images before and after. You may do it. FIG. 8 is a diagram schematically showing the processing content in a modified example in which an interpolated image is generated using the proxy images before and after it and inserted. As a concrete example, the proxy image 15 in which the previous proxy image corresponds to the vowel “A”
0a, the processing content when the subsequent image is the proxy image 150f corresponding to the sound repellency "n" is shown.

As shown in FIG. 8, from the substitute image 150a corresponding to the vowel "A" to the substitute image 1 corresponding to the vowel "N".
In the case of switching to 50f, the substitute image generation unit 14 generates an interpolation image 160 having an intermediate mouth shape between a vowel "a" and a vowel "n" by using these substitute images 150a and 150f, and the substitute image is generated. 150a and proxy image 150
It may be inserted between f. As a result, the mouth shape of the proxy image appears to smoothly change from “A” to “N”, and a more natural expression can be expressed. Note that in the example shown in FIG. 8, one interpolation image is inserted between the front and rear proxy images, but by inserting more interpolation images, the change in the mouth shape can be expressed more smoothly. You can

Further, in the above-described embodiment, the proxy image in which the facial expression (mainly the shape of the mouth) is changed according to the pronunciation content is generated. A change such as opening and closing may be added. By adding changes such as hand gestures and opening / closing of eyes, the expressive power of the proxy image can be further enhanced.

In the above-described embodiment (similarly to the second embodiment described later), the pronunciation content is specified on the assumption that the speaker uses Japanese. However, a language other than Japanese is used. Also in the case of using, the present invention can be applied in the same manner as in the above-described embodiment by extracting vowels and the like in each language as the sound of interest.

Second Embodiment By the way, in the above-described first embodiment, one embodiment of the videophone device to which the present invention is applied has been described. However, the present invention is not limited to the videophone device. The present invention can be applied to various devices that display a predetermined character image (for example, a character image that imitates a person) corresponding to voice output.

An embodiment in which the present invention is applied to a vehicle-mounted navigation device will be described below. In the second embodiment, the navigation device itself having the function of performing voice output corresponds to the "speaker" in a pseudo manner, and the character image displayed corresponding to the voice output corresponds to the "proxy image". It will be.

FIG. 9 is a diagram showing the configuration of the navigation device of the second embodiment. The navigation device 200 shown in FIG. 9 includes a navigation controller 60 and a DV.
The D drive 61, the operation unit 62, the vehicle position detection unit 63, the display device 64, and the speaker 65 are included.

The navigation controller 60 displays a map around the position of the vehicle, searches for a travel route connecting the designated starting point and destination, and guides the route along this travel route. It controls the overall operation of the. This navigation controller 6
0 is realized by executing a predetermined operation program using a CPU, a ROM, a RAM and the like. Details of the internal configuration of the navigation controller 60 will be described later.

The DVD drive 61 is loaded with one or a plurality of DVDs, and reads out map data required for map display or route search from any of the DVDs under the control of the navigation controller 60. The loaded disc is not limited to the DVD and may be another disc type storage medium.

The operation section 62 is provided with various operation keys such as up / down / left / right cursor keys and a numeric keypad, and outputs a signal according to the operation content to the navigation controller 60. The vehicle position detection unit 63 includes a GPS receiver, an azimuth sensor, a distance sensor, and the like, detects the own vehicle position (latitude, longitude), and outputs the detection result to the navigation controller 60.

The display device 64 is composed of, for example, a liquid crystal display panel having a screen size of about 8 inches, and displays a map image or the like based on a video signal output from the navigation controller 60.
The speaker 65 outputs various guidance voices such as a guidance voice in the traveling direction at the intersection based on the voice signal output from the navigation controller 60.

Next, details of the internal structure of the navigation controller 60 will be described. The navigation controller 60 shown in FIG. 9 includes a guidance sentence creating unit 70, a voice synthesizing unit 72, a pronunciation content identifying unit 74, a character image memory 76, and a character image generating unit 78.

The guide text creating unit 70 creates a guide text for performing various voice outputs (for example, route guidance at an intersection). The voice synthesis unit 72 generates a voice signal for performing voice output corresponding to the guide sentence generated by the guide sentence generation unit 70, and outputs the voice signal to the speaker 65.

The pronunciation content specifying unit 74 is provided by the guidance sentence creating unit 70.
The guidance sentence created by is acquired, and the pronunciation content of the guidance voice corresponding to this guidance sentence is specified. Specifically, the pronunciation content identifying unit 74 acquires the character data of the guidance sentence, and based on the character data, the five vowels “A, I, U, E, O” of Japanese from the guidance sentence, and Each of the sound repellency "n" is extracted as a sound of interest. For example, "At the next intersection ..."
The pronunciation contents corresponding to the guidance sentence such as "U, I, O,
Oh, u, a, e, n, o ... "

The character image memory 76 is similar to the proxy image memory 12 in the first embodiment described above, and the character image memory
Six types of character images (still proxy images) having mouth shapes corresponding to the respective types of pronunciation contents are stored. Here, a predetermined character image simulating a person is D
It is assumed that it is read from the DVD by the VD drive 61 and stored in the character image memory 76.

The character image generating unit 78 generates a character image by selecting from the character images stored in the character image memory 76 the one corresponding to the pronunciation content specified by the pronunciation content specifying unit 74. Perform processing. The character image generated by the character image generating unit 78 is displayed on the screen of the display device 64 together with other images (intersection guide map, etc.).

The above-described speaker 65 and the guide text creating section 7
0, the voice synthesizing unit 72 corresponds to the voice output unit, the pronunciation content specifying unit 74, the character image memory 76, the character image generating unit 78 corresponds to the substitute image generating unit, and the display device 64 corresponds to the displaying unit.

The navigation device 20 of the second embodiment
0 has such a configuration, and its operation will be described next. FIG. 10 is a flowchart showing an operation procedure of the navigation device when the character images are switched and displayed in correspondence with various audio outputs. As an example, a description will be given on the assumption that the character images are switched and displayed according to the pronunciation content of the guidance voice in the intersection guidance at the time of route guidance.

The pronunciation content identifying unit 74 determines whether or not the guidance sentence to be voice output is created by the guidance sentence creating unit 70 (step 300). If the guidance sentence is not created, a negative determination is made, and in this case, the determination of step 300 is repeated.

If the guidance sentence is created, step 3
An affirmative determination is made at 00, and the pronunciation content identification unit 74 acquires the guidance sentence from the guidance sentence creation unit 70 and identifies the pronunciation content based on this guidance sentence (step 301). When the pronunciation content is specified, the character image generation unit 78 performs a process of selecting a character image corresponding to the pronunciation content from the character images stored in the character image memory 76 (step 302).

The character image generator 78 of this embodiment selects the character image corresponding to the last character of the guidance sentence when performing the process of selecting the character image corresponding to one guidance sentence. When the output of the sound is stopped (that is, when the output of the voice is stopped), the process of automatically selecting the character image corresponding to the case where the pronunciation content is "n" is performed. With this, it is possible to prevent the character image with the mouth open from being continuously displayed when one guidance sentence ends, and to prevent the user from feeling uncomfortable in the display.

When the character image corresponding to the guidance sentence is selected, the navigation controller 60 switches the character image at any time in accordance with the progress of the voice output of the guidance sentence and displays it on the display device 64 (step 30).
3). Specifically, the navigation controller 60
Acquires the pronunciation timing of the guidance voice for each character from the voice synthesis unit 72, and switches the character image in synchronization with this pronunciation timing.

FIG. 11 is a diagram showing a route guide map at an intersection and a display example of a character image displayed in an overlapping manner. As shown in FIG. 11, a map image around the vehicle position is displayed on the left side of the screen, and an enlarged view of an intersection where the route is changed is displayed on the right side of the screen. In the enlarged cross-section, the direction of travel of the vehicle is indicated by an arrow, and with such a display, "You will turn left about 300 m ahead.
The landmark is the △△ bank. A guidance sentence such as "is output by voice. Further, a predetermined character image 210 is displayed on the lower right side of the display area of the enlarged intersection image, and the mouth shape of the character image 210 changes in accordance with the pronunciation content of the above-mentioned guidance sentence.

As described above, the navigation device 200 of the second embodiment, based on the guidance sentence that is the target of voice output, has five vowels "A, I, U, E, O" as the focused sounds.
The sound image “n” is extracted and the character image with the mouth shape changed according to the content of the extracted target sound is displayed. Therefore, it is possible to display a proxy image with a natural expression that matches the content of the voice. it can. In particular, since we extract the contents of the voice by paying attention to the fact that the shape of the mouth when a human speaks is determined almost corresponding to the vowel and the sound repellency, we can accurately match the voice contents with a small processing load. A mouth-shaped surrogate image can be displayed.

Further, six kinds of character images corresponding to the sound of interest are prepared in advance and stored in the character image memory 76, and the voice specified by the pronunciation content specifying unit 72 is selected from these six kinds of character images. Since the character image image is generated by selecting the one corresponding to the content of, there is also an advantage that the processing is simple and the processing load when generating the character image can be reduced.

The navigation device to which the present invention is applied is not limited to the above-mentioned embodiment, and various modifications can be made. For example, similar to the modification of the first embodiment described above, six types of mouth shape images corresponding to the pronunciation content and a common image serving as a base are prepared, and any one of the mouth shape image and the common image is prepared. The character image may be generated by synthesizing. Moreover, you may insert the interpolation image which interpolates between the character images before and behind.

Further, in the above-mentioned embodiment, the case where the character images are switched and displayed according to the pronunciation contents of the guidance voice in the intersection guidance at the time of route guidance has been described as an example. You may make it perform the same process corresponding to the audio | voice output in operation.

The scope of application of the present invention is not limited to the above-mentioned videophone device and navigation device, but can be widely applied to various other devices.
For example, if applied to a device such as a cash dispenser of a bank, it can be used as a pronunciation content for various guidance texts (for example, "Please enter the code number") that are output by voice according to the user's operation. At the same time, the mouth shape of the character image simulating the clerk displayed on the operation screen may be changed and displayed.

Further, in a television program or the like, a predetermined character is displayed separately from the original speaker (for example, voice actor or talent) for the purpose of improving entertainment, and this character is the speaker. Although it may be displayed as described above, the present invention can be applied to a character image display device that performs such processing.

FIG. 12 is a diagram showing the structure of the character image display device in the above-described modification. The character image display device 300 shown in FIG.
The operation unit 82, the control unit 84, the display device 86, and the speaker 88 are included. Further, the control unit 84
A voice recognition unit 90, a voice delay processing unit 92, a character image memory 94, and a character image generation unit 96 are provided.

In the above-described character image display device 300, the microphone 80 serves as a sound collecting unit, the voice recognizing unit 90 serves as a voice recognizing unit, the speaker 88 serves as a voice output unit, and the character image memory 94 and the character image generating unit 96 are provided. The display device 86 corresponds to the proxy image generation means and the display means. Since the operation contents of each of the above-described components are the same as those included in the above-described videophone device 100 or navigation device 200, detailed description thereof will be omitted.

Next, the operation of the character image display device 300 will be described. When the voice of the speaker is collected by the microphone 80, the voice recognition unit 90 performs a predetermined voice recognition process on this voice to determine the pronunciation content.
The character image generation unit 96 generates character images by sequentially selecting character images having mouth shapes corresponding to the pronunciation contents determined by the voice recognition unit 90. Further, the voice delay processing unit 92 delays the voice collected by the microphone 80 by a predetermined time. Then, the control unit 84 outputs the character image generated by the character image generation unit 96 to the display device 86, and outputs the sound delayed by the sound delay processing unit 92 to the speaker 88.

As a result, it becomes possible to realize a character image display device capable of generating and displaying a character image in which the mouth shape changes in response to the voice of the speaker in real time, and a character display full of realism can be realized. It can be carried out. When used for producing a television program or the like, the video signal of the character image output to the display device 86 and the audio signal output to the speaker 88 are parallel to another recording device (not shown). The character image and voice may be recorded.

[0108]

As described above, according to the surrogate image display device of the present invention, the surrogate image in which the mouth shape is changed according to the content of the voice of the speaker is displayed. A surrogate image with a natural expression can be displayed.

According to the video telephone device of the present invention,
Since the proxy image in which the mouth shape is changed according to the content of the voice of the speaker is generated, the proxy image with a natural expression suitable for the content of the voice can be sent to the communication partner.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a videophone device according to a first embodiment.

FIG. 2 is a diagram showing an example of six types of proxy images stored in a proxy image memory.

FIG. 3 is a flowchart showing an overall operation procedure of the videophone device.

FIG. 4 is a flowchart showing a detailed procedure of the image transmission processing shown in step 103.

FIG. 5 is a diagram illustrating processing contents of a modified example in which a mouth shape image and a common image are combined to generate a proxy image.

FIG. 6 is a diagram showing a configuration of a modified videophone device that generates a proxy image in which a speaker's emotions are reflected in facial expressions.

FIG. 7 is a diagram showing a configuration of a videophone device of a modified example that transmits a voice in which a gender of a speaker is switched.

[Fig. 8] Fig. 8 is a diagram schematically showing the processing content in a modified example of generating and inserting an interpolated image using front and rear proxy images.

FIG. 9 is a diagram showing a configuration of a navigation device according to a second embodiment.

FIG. 10 is a flowchart showing an operation procedure of the navigation device when a character image is switched and displayed corresponding to various audio outputs.

FIG. 11 is a diagram showing a route guidance map at an intersection and a display example of a character image displayed in an overlapping manner.

FIG. 12 is a diagram showing a configuration of a character image display device.

[Explanation of symbols]

1 camera 2 handsets 2a microphone 2b speaker 3 Display 4 operation part 5, 5a, 5b control unit 6 Communication processing unit 7 Memory IF (interface) section 10 Speech recognition unit 12, 12a Proxy image memory 14, 14a Proxy image generation unit 16 Audio delay processing section 18 Transmission image setting section 20 emotion determination part 22 Speaker Gender Change Department 50 memory cards 100, 100a, 100b Videophone device

─────────────────────────────────────────────────── ─── Continued Front Page (51) Int.Cl. ⁷ Identification Code FI Theme Coat (Reference) G10L 3/00 561C

Claims (21)

[Claims] 1. A voice output unit that outputs a voice of a speaker, and a proxy image generation unit that generates a proxy image of a speaker whose mouth shape changes according to the content of the voice output from the voice output unit. A proxy image display device, comprising: a display unit configured to display the image generated by the proxy image generation unit. 2. The substitute image generation means according to claim 1, wherein the vowel of the voice output from the voice output means is extracted as a target sound, and the substitute image corresponding to the target sound is generated. Surrogate image display device. 3. The proxy image display device according to claim 2, wherein the proxy image generation unit extracts a vowel sound as the target sound in addition to the vowel of the voice. 4. The method according to claim 2, wherein a plurality of types of still substitute images corresponding to the sound of interest are prepared in advance, and the substitute image generation unit outputs the plurality of types of still substitute images. A proxy image display device, wherein the proxy image is generated by selecting a target sound corresponding to the sound. 5. The method according to claim 2, wherein a plurality of types of mouth shape images corresponding to the sound of interest and common images other than these mouth shape images are prepared in advance. The proxy image display device is characterized in that the proxy image is generated by selecting one of the mouth shape images corresponding to the sound to be output and synthesizing the selected image with the common image. . 6. The substitute image generation means according to claim 2, wherein when the selected images corresponding to the sound of interest are switched, the interpolated images using the selected images before and after the selected images are selected. A proxy image display device, characterized in that the proxy image is generated by inserting it between images. 7. The substitute image generation means according to claim 1, wherein the substitute image generation means generates the substitute image corresponding to sound repellency when the output of the sound by the sound output means is stopped. Surrogate image display device. 8. The sound collecting unit according to claim 1, further comprising a sound collecting unit that collects the voice of the speaker, and a voice recognizing unit that determines the content of the voice collected by the sound collecting unit. The voice output means outputs a voice based on the voice collected by the voice collecting means, and the proxy image generating means displays the substitute image based on the content of the voice determined by the voice recognition means. A surrogate image display device characterized by generating. 9. A sound collecting unit that collects a voice of a speaker, a voice recognizing unit that determines the content of the voice collected by the sound collecting unit, and a voice content that is determined by the voice recognizing unit. A proxy image generation unit that generates a proxy image of the speaker whose mouth shape changes together, a shooting unit that shoots the image of the speaker, and a sound that is picked up by the sound collection unit, and is shot by the shooting unit. An image of the speaker, or a transmitting means for transmitting the proxy image generated by the proxy image generating means to the other party, a receiving means for receiving a voice and an image transmitted from the other party, A videophone device comprising: a voice output unit for outputting a voice received by the receiving unit; and a display unit for displaying an image received by the receiving unit. 10. The vowel sound determined by the voice recognition means according to claim 9, wherein the voice recognition means determines a vowel sound of the voice collected by the sound collection means. Is set as a target sound, and the substitute image corresponding to the target sound is generated. 11. The voice recognition means according to claim 10, wherein the voice recognition means determines a sound repellency in addition to the vowel of the voice, and the proxy image generation means sets the target sound including the sound repellency. Characteristic videophone device. 12. The substitute proxy image according to claim 10, wherein a plurality of types of still substitute images corresponding to the sound of interest are prepared in advance, and the substitute image generation unit selects the A videophone device characterized in that the substitute image is generated by selecting one corresponding to the content of the voice discriminated by the voice recognition means. 13. The method according to claim 10, wherein a plurality of types of mouth shape images corresponding to the sound of interest and common images other than these mouth shape images are prepared in advance. A television characterized in that the proxy image is generated by selecting, from among various types of mouth-shaped images, one corresponding to the content of the voice discriminated by the voice recognition means and combining it with the common image. Telephone device. 14. The substitute image generation means according to claim 10, wherein when the selected images corresponding to the sound of interest are switched, the interpolated images using the selected images before and after the selected images are selected. A videophone device, wherein the substitute image is generated by inserting the image between the images. 15. The substitute image generation means according to claim 9, wherein the substitute image generation means generates the substitute image corresponding to sound repellency when the sound collection by the sound collection means is interrupted. Videophone device. 16. The speaker according to any one of claims 9 to 15, when an incoming call is received, the proxy image generated by the proxy image generating means is first transmitted from the transmitting means to the communication partner, and then the speaker is switched. In response to the instruction, the image of the speaker photographed by the photographing means is transmitted, and at the time of transmission, one of the substitute image generated by the substitute image generating means and the image of the speaker photographed by the photographing means is displayed. A videophone device further comprising transmission image setting means for controlling transmission according to a speaker's selection instruction. 17. The emotion determination unit according to claim 9, further comprising an emotion determination unit that determines the emotion of the speaker, wherein the proxy image generation unit expresses the emotion of the speaker determined by the emotion determination unit. A videophone device characterized in that the substitute image reflected in the above is generated. 18. The emotion determination unit according to claim 17, wherein the emotion determination unit uses at least one of the voice of the speaker collected by the sound collection unit and the image of the speaker captured by the image capturing unit. A videophone device characterized by judging emotions. 19. The delay unit according to claim 9, further comprising a delay unit that delays a voice transmission time by a time required until the generation of the substitute image by the substitute image generation unit is completed. Videophone device. 20. The image processing device according to claim 9, further comprising an image input unit that captures an arbitrary character image used as the proxy image, wherein the proxy image generation unit captures the image by the image input unit. A videophone device, wherein the substitute image is generated by changing the mouth shape of the character image. 21. The speaker sex changing unit according to claim 9, further comprising a speaker gender changing unit that switches the gender of the speaker by changing the sound quality of the voice collected by the sound collecting unit. A videophone device, wherein the voice after the gender is switched by the person gender changing means is transmitted from the transmitting means.