JP2008085421A - Video telephone, calling method, program, voice quality conversion-image editing service providing system, and server - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To transmit an image concerning a target speaker interlockingly to the voice of the target speaker, when a user makes a call in another person's voice while utilizing voice quality transformation, i.e. performs impersonation. <P>SOLUTION: This video telephone set is provided with a telephone data storage section 148 for storing an image 192 about the target speaker and a voice quality transforming filter 194; a speaker selecting section 154 for selecting a target speaker; data selecting section 156 for selecting a target speaker-related image and a voice quality transforming filter from the telephone data storage section; an imaging section 160; an image editing section 162 for editing the image in the imaging section on the basis of the target speaker-related image; a voice input section 164; a voice quality transforming section 166 for transforming the voice quality of a speaker into the voice quality of the target speaker using the voice quality transforming filter; and a telephone transmitting section 168 for transmitting the image edited by the image editing section and the voice quality transformed by the voice quality transforming section to a calling party. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video phone, a program, a calling method, a voice quality conversion / image editing service providing system, and a server that change an image together with voice quality conversion in a video phone.

  Currently, a mobile mobile terminal such as a mobile phone is equipped with a camera that captures the face image of the user and a display that displays the face image of the other party. So-called videophone calls have been implemented.

  As an application of such a videophone, a character (Avatar) acting as a substitute for the user is selected and an image of the avatar is transmitted instead of the image of the user during a call, or an avatar is arbitrarily selected on the other party side. A technique of selecting and displaying the avatar instead of the image of the sender is also disclosed (for example, Patent Document 1).

In addition, by using such a portable mobile terminal, it is possible to replace the voice uttered by itself with another voice and transmit it to the other party. However, when only making a call without transmitting its own image, that is, when the other party cannot identify himself / herself, an inappropriate act of impersonating another person with a voice whose voice has been converted may occur. Therefore, a technique for restricting voice quality conversion in a voice only mode in which an image of a speaker is not transmitted and not inducing such an inappropriate action has been studied (for example, Patent Document 2).
Japanese Patent Laid-Open No. 2003-248841 JP 2002-314638 A

  However, in the above-described technology, there is no relationship between the selected avatar and the voice after voice quality conversion, and the avatar is merely used as a pseudo display in the virtual communication space, and the entertainment of the call is reduced. It was only one of the choices to increase.

  Also, in the voice quality conversion that converts the voice quality of the speaker (self) in the videophone to the voice quality of the target speaker (other person who is not the other party), the voice of the speaker is converted to the voice of the target speaker, and the voice of another person is converted. Communicating to the other party, but the voice alone lacks realism, and unless the speaker explicitly tells who the target speaker is, to know who the other party is speaking Was difficult.

  The present invention has been made in view of the above-mentioned problems of conventional portable mobile terminals, and an object of the present invention is to achieve a so-called imitation of making a call with another person's voice using voice quality conversion. By sending images related to the target speaker in conjunction with the voice of the speaker, the call partner can intuitively determine who the target speaker is by hearing and vision. Another object of the present invention is to provide an improved video phone, a calling method, a program, a voice quality conversion / image editing service providing system, and a server.

  In order to solve the above-described problem, a videophone according to claim 1 of the present invention includes an imaging unit that inputs an image of a speaker and a voice input unit that inputs the voice of the speaker, and the speaker TV phone that converts the voice quality of the voice of the speaker into the voice quality of the target speaker, and stores in advance an image related to the target speaker and a voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the target speaker A data storage unit, a speaker selecting unit for selecting a target speaker, a target speaker related image that is an image related to the target speaker selected by the speaker selecting unit, and a voice quality conversion filter corresponding to the target speaker; A data selection unit that selects the phone data storage unit, an image editing unit that edits an image input from the imaging unit based on the target speaker-related image, and a voice quality of the voice of the speaker input from the voice input unit Use the selected voice quality conversion filter A voice conversion unit which converts the voice quality of the target speaker Te, image and voice conversion unit image editing unit edits is characterized in that it comprises a telephone transmission unit for transmitting to the communication partner and voice converted. Here, the target speaker may be a person or an anime character. The image related to the target speaker may be an image of the target speaker itself or an image that can associate the target speaker.

  In a call between videophones, an image of the speaker is transmitted simultaneously with the voice of the speaker. Using the videophone, the voice of a speaker is automatically converted to the voice of a target speaker such as an actor, and the target speaker's voice is sent to the other party in place of his own voice. "I do. With such a configuration, not only the target speaker's voice but also an image edited by the image editing unit based on an image related to the target speaker, for example, a face photograph of the target speaker, in such “imitation”. Since it is transmitted to the other party's video phone, the other party can intuitively grasp who the speaker is imitating by hearing and vision.

  The invention described in claim 2 is the video phone according to claim 1, further comprising a mimic switch, wherein the image editing unit and the voice quality conversion unit function while the mimic switch is valid. .

  With the configuration of the mimic switch, the speaker can determine the timing for imitating based on his / her will. Therefore, the speaker can perform the imitation only for the utterance he / she wants to imitate, and can convey the portion with emphasis to the other party.

  According to a third aspect of the present invention, the videophone according to the first or second aspect further includes a speaker state information generating unit that generates speaker state information that is information relating to the state of the speaker. The image editing unit changes the target speaker-related image used for editing according to the speaker state information.

  With the above-described configuration in which the target speaker-related image is uniformly edited through the speaker state information, the image editing unit can edit the image input from the imaging unit by decoding only the speaker state information. It becomes possible.

  According to a fourth aspect of the present invention, the videophone according to the third aspect further includes a voice detection unit that detects the presence or absence of the voice of the speaker input from the voice input unit, and the speaker state information generation unit Is characterized in that the speaker state information is generated including voice presence / absence information which is information related to the presence / absence of the detected voice.

  With this configuration, the image editing unit can change the target speaker-related image used for editing in accordance with the presence or absence of the voice of the speaker. In this way, by changing the image according to the presence or absence of voice, when there is voice, that is, when the speaker is imitating, emphasizing that imitation is being performed by changing the image related to the target speaker can do. Moreover, even when there is no sound, the other party's interest can be maintained without getting bored by displaying another image related to the target speaker.

  The video phone according to any one of claims 3 and 4, further comprising a voice recognition unit for recognizing the utterance content of the voice of the speaker input from the voice input unit, The speaker state information generation unit generates the speaker state information including utterance content information that is information related to the recognized utterance content.

  The image related to the target speaker stored in the telephone data storage unit is associated with arbitrary utterance content information in advance, and when the speech recognition unit recognizes the utterance content, the image editing unit The associated image is superimposed on the image to be transmitted to the other party. With this configuration, an image suitable for the content of the speaker's utterance can be transmitted to the other party, and a mimicry can be dynamically expressed.

  The invention described in claim 6 is the video phone according to any one of claims 3 to 5, further comprising a phoneme recognition unit that recognizes a phoneme of a speaker's voice input from the voice input unit. The speaker state information generation unit generates the speaker state information including phoneme information that is information related to the recognized phoneme type, and the image editing unit generates the target speaker according to the phoneme information. The opening / closing degree of the mouth of the related image is adjusted.

  The image related to the target speaker stored in the telephone data storage unit is associated with the phoneme information in advance, and when the phoneme recognition unit recognizes the phoneme of the speech, the image editing unit is associated with the phoneme information. Overlay the image on the image to be sent to the other party. By reflecting the movement of the speaker's mouth that can be represented by such phonemes in the target speaker's image, an image as if the target speaker is actually speaking can be transmitted to the other party. It is possible to express imitations.

  The invention described in claim 7 is the video phone according to any one of claims 3 to 6, further comprising an image recognition unit for recognizing a state of an image of the speaker input from the imaging unit, and the speaker The state information generation unit generates the speaker state information including the state of the recognized image.

  With this configuration, the image editing unit can change the target speaker-related image in conjunction with the movement of the speaker, particularly the actual movement of the face. Therefore, the speaker can visually and dynamically imitate the target speaker, and can emphasize the fact that the speaker is imitating the target speaker. .

  According to an eighth aspect of the present invention, in the video phone according to the seventh aspect, when the image recognition unit recognizes that there is a speaker's face, the position of the speaker is detected, and the speaker state information The generation unit generates the speaker state information including face position information that is information related to the detected face position, and the image editing unit sets the target position at the face position corresponding to the face position information. It is characterized by overlapping speaker-related images.

  With this configuration, the speaker can change the position of the target speaker-related image to be transmitted to the other party by moving his / her face.

  According to a ninth aspect of the present invention, in the video phone according to the eighth aspect, when the image recognition unit recognizes that there is a speaker's face, the image recognition unit detects the inclination of the speaker, and the speaker state information The generation unit generates the speaker state information including face inclination information that is information related to the detected face inclination, and the image editing unit generates the target speaker related image according to the face inclination information. It is characterized by rotating and overlapping.

  With this configuration, the speaker can rotate the target speaker-related image to be transmitted to the other party by changing the inclination of his / her face.

  According to a tenth aspect of the present invention, in the video phone according to the eighth or ninth aspect, when the image recognition unit recognizes that there is a face of a speaker, the size of the face is detected. The speaker state information generation unit generates the speaker state information including face size information that is information related to the detected face size, and the image editing unit is configured to respond to the face size information. The target speaker related image is enlarged or reduced.

  With such a configuration, the speaker can change the size of the target speaker-related image to be transmitted to the other party by changing the size of his / her face, that is, the distance from the imaging unit.

  According to an eleventh aspect of the present invention, in the video phone according to any one of the eighth to tenth aspects, when the image recognition unit recognizes that there is a speaker's face, the speaker's eyes are opened and closed. And the speaker state information generating unit generates the speaker state information including the eye opening / closing information which is information relating to the detected eye opening / closing, and the image editing unit is configured to respond to the eye opening / closing information. Then, the eyes in the target speaker related image are opened and closed.

  With the configuration that reflects the opening / closing operation of the speaker's eyes in the target speaker's image, it is possible to send an image as if the target speaker is actually blinking to the other party. It is possible to express imitations.

  According to a twelfth aspect of the present invention, in the video phone according to any one of the eighth to eleventh aspects, when the image recognition unit recognizes that there is a speaker's face, the mouth of the speaker is opened and closed. And the speaker state information generating unit generates the speaker state information including mouth opening / closing information which is information relating to the detected opening / closing of the mouth, and the image editing unit is configured to respond to the mouth opening / closing information. The mouth in the target speaker related image is opened and closed.

  With the configuration that reflects the opening and closing movements of the speaker's mouth in the target speaker's image, it is possible to send an image as if the target speaker is actually speaking to the other party, more imitating it. Can be expressed.

  According to a thirteenth aspect of the present invention, in the videophone according to any one of the first to twelfth aspects, the voice quality conversion filter is configured to convert a voice quality of each speaker into a common intermediate speaker voice quality. 1 voice quality conversion filter, and a second voice quality conversion filter for converting the voice quality of the intermediate speaker to the voice quality of each target speaker, and the data selection unit uses the first voice quality as the voice quality conversion filter. A conversion filter and the second voice quality conversion filter are selected, and the voice quality conversion unit uses the selected first voice quality conversion filter to convert the voice quality of the voice of the speaker input from the voice input unit to the intermediate speech. And converting the voice quality of the intermediate speaker into the voice quality of the target speaker using the selected second voice quality conversion filter.

  With such a two-stage filter configuration through the intermediate speaker, the speaker need only prepare the first voice quality conversion filter once, and does not need to generate a voice quality conversion filter each time the target speaker is changed. In addition, once the second voice quality conversion filter is generated on the service provider side providing the voice quality conversion filter to the target speaker, the common second voice quality conversion filter can be provided to a plurality of speakers. Thus, an efficient system can be built, and a large number of patterns of the speaker and the target speaker can be generated with a small load.

  According to a fourteenth aspect of the present invention, in the videophone according to the thirteenth aspect, an image related to a target speaker used in the image editing unit, a second voice quality conversion filter used in the voice quality conversion unit, and Is further provided with a receiving unit that receives the signal from an external electronic device.

  As described above, the service provider side that provides the voice quality conversion filter for the target speaker can provide the user with the second voice quality conversion filter common to the user. A related image and the second voice quality conversion filter may be combined and provided as package data. On the speaker side, it is possible to imitate an arbitrary target speaker immediately and simply by acquiring such package data.

  The invention according to claim 15 is the video phone according to claim 13 or 14, further comprising an utterance type selection section for allowing a speaker to select a clear utterance or a secret utterance, and the data selection section The first voice quality conversion filter that converts the clear voice quality of the speaker into the voice quality of the intermediate speaker according to the selected speech type, or converts the voice quality of the speaker into the voice quality of the intermediate speaker One of the first voice quality conversion filters is selected.

  With the configuration in which the first voice quality conversion filter corresponding to the two types of utterances is provided, the speaker can select an appropriate first voice quality conversion filter according to his / her situation.

  According to a sixteenth aspect of the present invention, in the video phone according to the fifteenth aspect, when a secret utterance type is selected in the utterance type selection unit, the data selection unit determines the voice quality of the intermediate speaker. The second voice quality conversion filter for converting to a clear voice quality is selected.

  With this configuration, the first voice quality that converts the voice quality of the speaker into the voice quality of the middle speaker even when the speaker is restricted to the situation in which he / she is placed, Through the conversion filter and the second voice quality conversion filter that converts the voice quality of the intermediate speaker into the clear voice quality of the speaker, the other party can surely grasp the voice uttered by the speaker. Become.

  According to a seventeenth aspect of the present invention, in the video phone according to the fifteenth or sixteenth aspect, when a dense utterance type is selected in the utterance type selection unit, the image editing unit A display image indicating that the user is speaking is overlaid.

  The first voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the middle speaker, and the voice quality of the middle speaker, even if the speaker is speaking dense Through the second voice quality conversion filter that converts the voice quality, the other party can hear the clear voice quality of the speaker. However, from the other party, it is impossible to grasp under what circumstances the call is being made. With the configuration in which the image indicating that the voice is spoken is overlaid, the call partner can grasp the situation of the speaker.

  According to an eighteenth aspect of the present invention, in the video phone according to any of the thirteenth to seventeenth aspects, the first voice quality conversion filter and the second voice quality conversion filter are synthesized to target the voice quality of the speaker. A voice quality conversion filter synthesis unit that generates a synthesis filter that directly converts the voice quality of the speaker is further provided.

  After the above two-stage filter is taken in at the stage of videophone call preparation, it is not necessary to keep the voice quality conversion filter in two stages. As described above, when the two-stage voice quality conversion filter is synthesized, it is not necessary to pass through the two-stage voice quality conversion filter for each utterance, and the processing load and power consumption for voice quality conversion can be reduced.

  According to a nineteenth aspect of the present invention, in the video phone according to any one of the first to eighteenth aspects, the image can be obtained only when an identifier that can identify the own video phone is transmitted to the video phone of the other party. The image processing apparatus further includes a function permission unit that permits the editing unit and the voice quality conversion unit to function.

  With such a configuration, it is possible to avoid disguise based on impersonation of the target speaker and the inability to identify the speaker. Here, the identifier to be transmitted may be the telephone number of the own videophone.

  Here, each of the above-described components is configured to be provided on the video phone, but such a component is provided on a server or a video phone of the other party in the path to the other party's video phone, and configured to operate in conjunction with the other party. You can also

  The videophone is represented by an assembly of a plurality of components, but each component need not belong to a single device. Further, the above components may function as an electric circuit or a functional module on a computer.

  In order to solve the above-mentioned problem, the call method according to claim 20 is a call method for making a call using a video phone that converts the voice quality of the voice of the speaker into the voice quality of the target speaker. A telephone data storage step of storing in the telephone data storage unit a voice quality conversion filter for converting an image related to the voice quality of the speaker to the voice quality of the target speaker, a speaker selection step of selecting a target speaker, A data selection step of selecting a target speaker related image, which is an image related to the target speaker selected in the speaker selection step, and a voice quality conversion filter corresponding to the target speaker from the telephone data storage unit, and an image of the speaker An imaging step for inputting the image, an image editing step for editing the image input in the imaging step based on the target speaker-related image, an audio input step for inputting the voice of the speaker, and an audio The voice quality of the speaker's voice input at the power step is converted to the voice quality of the target speaker using the selected voice quality conversion filter, and the image edited at the image editing step is converted at the voice quality conversion step. A telephone transmission step of transmitting the received voice to the other party.

  In order to solve the above problem, the program according to claim 21 stores an image related to the target speaker and a voice quality conversion filter for converting the voice quality of the speaker into the voice quality of the target speaker in the telephone data storage unit. Phone data storage step to be performed, speaker selection step for selecting a target speaker, target speaker related image that is an image related to the target speaker selected in the speaker selection step, and voice quality conversion corresponding to the target speaker A data selection step for selecting a filter from the telephone data storage unit, an imaging step for inputting an image of the speaker, and an image editing step for editing the image input in the imaging step based on the target speaker-related image; The voice input step of inputting the voice of the speaker, and the voice quality of the voice of the speaker input in the voice input step, the voice of the target speaker using the selected voice quality conversion filter And voice conversion step of converting into, characterized in that to execute a telephone transmission step of transmitting a voice to the other party that has been converted by the image and voice conversion step edited in the image editing step, to the computer.

  According to still another aspect of the present invention, in order to solve the above-described problem, a voice quality conversion / image editing service providing system according to another aspect of the present invention includes a server, a videophone connected to the server, and a videophone. A voice quality conversion / image editing service providing system for converting a voice quality of a speaker's voice into a voice quality of a target speaker and providing a voice quality conversion / image editing service for editing a speaker's image, the server comprising: A server data storage unit that stores an image related to the target speaker and a voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the target speaker; and a target speaker stored in the server data storage unit. A server transmitting unit that transmits the related image and the voice quality conversion filter to the video phone, and the video phone receives the image related to the target speaker and the voice quality conversion filter. The telephone data storage unit that stores the image and voice quality conversion filter related to the target speaker received by the receiving unit, the speaker selection unit that selects the target speaker, and the target story selected by the speaker selection unit A target speaker-related image that is an image related to the speaker, a voice quality conversion filter corresponding to the target speaker, a data selection unit that selects from the telephone data storage unit, an imaging unit that inputs an image of the speaker, and imaging The image editing unit that edits the image input from the unit based on the target speaker-related image, the voice input unit that inputs the voice of the speaker, and the voice quality of the voice of the speaker input from the voice input unit, A voice quality conversion unit that converts the voice quality of the target speaker using the selected voice quality conversion filter, an image edited by the image editing unit, and a telephone transmission unit that transmits the voice converted by the voice quality conversion unit to the other party; It is characterized by providing.

  With such a configuration, the server can store an image related to the target speaker and a voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the target speaker by the server data storage unit. Yes, the server transmission unit can transmit the image and the voice quality conversion filter related to the target speaker to the video phone.

  In addition, in the so-called “imitation”, the video phone can call not only the voice of the target speaker but also an image edited by the image editing unit based on an image related to the target speaker, for example, a face photograph of the target speaker. Thus, the other party can intuitively grasp who the speaker is imitating by hearing and visual sense.

  According to a twenty-third aspect of the present invention, in the voice quality conversion / image editing service providing system according to the twenty-second aspect, the voice quality conversion filter converts the voice quality of individual speakers into the voice quality of a common intermediate speaker. A first voice quality conversion filter; and a second voice quality conversion filter for converting the voice quality of the intermediate speaker into the voice quality of each target speaker. The server transmission unit may include the first voice quality conversion filter or the second voice quality conversion filter. Either or both of voice quality conversion filters can be transmitted, and the receiving unit can receive either one or both of the first voice quality conversion filter and the second voice quality conversion filter, and the telephone data storage The first voice quality conversion filter and the second voice include one or both of the first voice quality conversion filter and the second voice quality conversion filter received by the reception section. When the first voice quality conversion filter is designated in advance, the data selection section selects the second voice quality conversion filter from the telephone data storage section, and the first voice quality conversion filter selects the second voice quality conversion filter. If not specified in advance, the first voice quality conversion filter and the second voice quality conversion filter are selected from the telephone data storage section as the voice quality conversion filter, and the voice quality conversion section Is designated in advance, the voice quality of the intermediate speaker is converted using the designated first voice quality conversion filter. If the first voice quality conversion filter is not designated in advance, the voice input is performed. The voice quality of the voice of the speaker input from the unit is converted into the voice quality of the intermediate speaker using the selected first voice quality conversion filter, and the voice quality of the intermediate speaker is further converted into the voice quality of the intermediate speaker. And converting the voice quality of the target speaker by using a second voice conversion filters-option.

  With such a configuration, the two-stage filter configuration through the intermediate speaker allows the speaker to prepare the first voice conversion filter once and change the target speaker every time the target speaker is changed. Need not be generated. In addition, once the second voice quality conversion filter is generated in the server on the service provider side that provides the voice quality conversion filter to the target speaker, the common second voice quality conversion filter can be provided to a plurality of speakers. An efficient system can be built at a low cost, and a large number of patterns of a speaker and a target speaker can be generated with a small load.

  According to a twenty-fourth aspect of the present invention, there is provided a server used in the voice quality conversion / image editing service providing system according to the twenty-second or twenty-third aspect, comprising the server data storage unit and the server transmission unit.

  With such a configuration, operations and effects equivalent to those of the server in the voice quality conversion / image editing service providing system according to claim 22 or 23 can be obtained.

  The invention described in claim 25 is a video phone used in the voice quality conversion / image editing service providing system according to claim 22 or 23, wherein the receiving unit, the telephone data storage unit, the speaker selection unit, the data selection Unit, an imaging unit, an image editing unit, a voice input unit, a voice quality conversion unit, and a telephone transmission unit.

  With such a configuration, the same operation and effect as the video phone in the voice quality conversion / image editing service providing system according to claim 22 or 23 can be obtained.

  Further, the constituent elements corresponding to the dependent claims in the above-described videophone and the description thereof can be applied to a calling method, a program, a voice quality conversion / image editing service providing system, and a server.

  As described above, according to the video phone of the present invention, when performing so-called imitation, the voice of the target speaker whose voice quality has been converted and the image related to the target speaker are transmitted to the other party in conjunction with each other. The call partner can intuitively grasp who the speaker is imitating through hearing and vision.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  A portable mobile terminal such as a mobile phone is equipped with a camera that captures the face image of the user and a display that displays the face image of the other party. Using such a configuration, a so-called videophone call is possible in which a conversation is made while confirming the current call state of the other party.

  In such a videophone, a device for enjoying a telephone call by processing an image and sound is made, and entertainment is enhanced. For example, you can select an avatar that is your character as a substitute, and send that avatar image instead of your own image during a call, or convert your voice to another voice and tell it to the other party be able to.

  However, it is not possible to find out the relationship between voice quality conversion and the fact that an avatar is his or her partner's. Therefore, even if the so-called “imitation” is performed to convert the voice of the speaker into the voice of the known target speaker, the sense of reality is lacking. Furthermore, unless the speaker explicitly indicates who the target speaker is mimicking, the voice quality of the other party will be reduced. Is difficult to grasp.

  In the embodiment of the present invention, when voice quality conversion from the speaker to the target speaker is performed when transmitting an image and sound from the speaker to the call partner, the image of the speaker is also linked with the target speech. It is characterized by changing to an image related to the person. The timing at which this image changes is as follows: When the speaker operates, 2. 2. When the speaker's speech changes, When the speaker moves visually. Hereinafter, the three timings will be described in three embodiments. Further, further features regarding the conversion of voice quality between the speaker and the target speaker will be described in detail as fourth and fifth embodiments.

(First embodiment)
(Voice quality conversion / image editing service provision system)
FIG. 1 is an explanatory diagram for explaining a voice quality conversion / image editing service providing system using a video phone 100. An embodiment of the voice quality conversion / image editing service providing system will be described. The calling party's speaker 110 and the conversation partner 120 have a mobile phone as the video phone 100, and each other through the base station 130. Connected by a communication network 132. Here, the video phone 100 is formed of an information communication terminal such as a mobile phone, a PHS (Personal Handy phone System) terminal, a home phone, a PDA (Personal Digital Assistant), a mobile personal computer, a personal computer, or the like.

  Such a video phone 100 transmits an image and a voice uttered by a speaker 110 using a video phone to the call partner 120 in real time, and simultaneously receives an image and a sound of the call partner 120 in real time. Therefore, the speaker 110 can enjoy the call as if speaking in front of his eyes while scolding the current complexion of the other party 120. Here, speech is a speech sound that is uttered by a human through a vocal organ or reproduced by an electronic device, and is a set of pronunciations represented by a waveform obtained by synthesizing one or more fundamental frequencies. This refers to the tone and pitch of the voice.

  A server 140 is also connected to the communication network 132. The server 140 holds package data in which an image related to a target speaker used in the videophone 100 and a voice quality conversion filter corresponding to the target speaker are combined, and this package is received in response to a request from the videophone 100. Data can be provided (distributed). Hereinafter, the video phone 100 and the server 140 will be described in detail.

(Videophone 100)
FIG. 2 is a functional block diagram showing a schematic configuration of the video phone 100. The video phone 100 includes a central control unit 146, a telephone data storage unit 148, a display unit 150, a speaker selection switch 152, a speaker selection unit 154, a data selection unit 156, an imaging unit 160, an image Editing unit 162, voice input unit 164, voice quality conversion unit 166, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, imitation switch 174, reception demodulation unit 176, reception unit 180, The image display unit 182, the audio output unit 184, the speaker 186, and the function permission unit 188 are configured. Here, the user 110 speaking on the video phone 100 is called a speaker, and the user 120 listening to the speech is called a calling party.

  The central control unit 146 manages and controls the entire video phone 100 by a semiconductor integrated circuit including a central processing unit (CPU). Each component in the area 190 surrounded by a dotted line is under the control of the central control unit 146 and functions as an electric circuit or a program module. Accordingly, each component in the area 190 is normally held in a storage medium and read into the central processing unit to perform each function.

The telephone data storage unit 148 includes a storage medium such as RAM, E ² PROM, non-volatile RAM, flash memory, card memory, USB memory, HDD (Hard Disk Drive), etc., and is at least a target speaker to be imitated As one or more images 192 related to the target speaker, and a voice quality conversion filter 194 that enables voice quality conversion from the voice quality of the speaker to the voice quality of the target speaker. ing.

  The target speaker data may be a combination of an image 192 related to the target speaker and a second voice quality conversion filter described later, and is downloaded from the server 140 to the videophone 100 as package data based on such combination. It is good.

  The display unit 150 includes a liquid crystal display or the like, and performs selection screen output or result output of an application or the like used in the video phone 100. In FIG. 2, a target speaker list 196 for selecting a target speaker to be imitated is displayed. The target speaker list 196 is generated by a speaker selection unit 154 described later. The display unit 150 also displays an image of the other party received from the other party's video phone during a videophone call.

  The speaker selection switch 152 is formed of a cross key, a jog dial, a keyboard, or the like, and selects a specific target speaker from the target speaker list 196 displayed on the display unit 150 when pressed. It is also used for making calls to other parties and operating other applications.

  First, the speaker selection unit 154 refers to the image 192 or the voice quality conversion filter 194 related to the target speaker stored in the telephone data storage unit 148 via the data selection unit 156 and selects the target speaker. A target speaker list 196, which is a list of speakers that can be used, is generated and transmitted to the display unit 150. Here, when the speaker selects a target speaker through the speaker selection switch 152 from the target speaker list 196 displayed on the display unit 150, the speaker selection unit 154 displays the selected target speaker. As a target speaker to be imitated and transmitted to the data selection unit 156.

  The data selection unit 156 includes a target speaker related image, which is an image 192 related to the target speaker selected by the speaker selection unit 154, and a voice quality conversion filter 194 corresponding to the target speaker. Select from 148.

  The imaging unit 160 is formed to include an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), and captures an image input (captured) by the imaging element to the image editing unit 162. Send.

  The image editing unit 162 edits the image including the speaker and its periphery input from the imaging unit 160 based on the target speaker related image selected by the data selection unit 156. This editing includes superimposing an image related to the target speaker on part or all of the input image. The superimposed image may be the target speaker itself, such as a raw photo, or may be an image related to the target speaker, for example, an image associated with the target speaker.

  Here, editing based on the image 192 related to the target speaker is to change the target speaker related image in conjunction with the input image as will be described later, but regardless of the input image. In addition, the target speaker-related image is simply replaced with the target speaker-related image, such as using the target speaker-related image as a transmission image like an avatar.

  When the image editing unit 162 overwrites the target speaker on the image input from the image capturing unit 160, if the coordinate designation is associated with the target speaker related image, the image editing unit 162 follows the coordinate designation for each target speaker related image. The target speaker-related image may be displayed at the coordinate position, or may be displayed at the position of the speaker's face image in the manner of covering the face image of the speaker.

  The voice input unit 164 is configured by a device capable of inputting voice, such as a microphone, and converts a voice uttered by a speaker into an electrical signal and transmits the electrical signal to the voice quality conversion unit 166.

  The voice quality conversion unit 166 uses the voice quality conversion filter 194 selected by the data selection unit 156 to convert the voice quality of the speaker's voice input from the voice input unit 164 into the target story selected by the speaker selection unit 154. To voice quality In this embodiment, since the voice quality conversion unit 166 converts only the timbre, the speaker does not need to imitate the target speaker's wording, but automatically adjusts the pitch or filters each of the local voice and the back voice. It is also possible to improve the quality of imitation by making preparations.

  The voice quality conversion unit 166 is, for example, a feature value conversion method (for example, A. Kain and MWMacon, “Spectral voice conversion” that converts a feature value such as a spectrum series based on a Gaussian Mixture Model (GMM). for text-to-speech synthesis, "Proc.ICASSP, pp.285-288, Seattle, USAMay, 1998.), and any other known method can be used.

  The telephone transmission unit 168 transmits the image and sound of the speaker himself / herself to the modulation transmission unit 170 in a normal call, and modulates the image edited by the image editing unit 162 and the sound converted by the voice quality conversion unit 166. To 170.

  The modulation transmission unit 170 generates a modulation signal by modulating the frequency of the electrical signal received from the telephone transmission unit 168 to the transmission frequency, and transmits the modulation signal to the antenna unit 172.

  The antenna unit 172 changes the output of the modulation transmission unit 170 to a transmission radio wave and transmits it, and transmits the contents to the video phone of the other party. In addition, it receives radio waves received from the other party's video phone and outputs them to a reception demodulator 176 described later.

  The imitation switch 174 is pressed by a speaker at a desired timing, and while the imitation switch 174 is valid, an imitation function including the image editing unit 162 and the voice quality conversion unit 166 is executed. The imitation switch 174 is set to an imitation state, that is, a state in which the image editing unit 162 and the voice quality conversion unit 166 function, and may be switched between imitation release and imitation processing start each time it is pressed. The mimicking process only for the period may be executed. The mimic switch 174 may be a physical button, or may be a soft input for setting whether to mimic from the menu screen each time. In this way, the speaker can determine the timing of imitation based on his / her will by operating the imitation switch 174. Therefore, the speaker can perform the imitation only for the utterance he / she wants to imitate, and can convey the portion with emphasis to the other party.

  The reception demodulator 176 performs amplification of received radio waves and frequency tuning detection, and further transmits an electrical signal obtained through demodulation processing to the receiver 180.

  The receiving unit 180 receives the electrical signal from the reception demodulating unit 176 and transmits the image and sound of the other party to the image display unit 182 and the audio output unit 184. In addition, an image related to the target speaker used in the image editing unit 162 and a voice quality conversion filter used in the voice quality conversion unit 166 can be received from an external electronic device such as the server 140. Further, as the voice quality conversion filter 194, either or both of a first voice quality conversion filter and a second voice quality conversion filter described later can be received.

  The image display unit 182 displays the call partner image received by the receiving unit 180 on the display unit 150 in real time.

  The voice output unit 184 outputs the other party's voice received by the receiving unit 180 to the speaker 186 in real time.

  The speaker 186 receives the audio signal from the audio output unit 184 and transmits the voice of the other party to the speaker.

  The function permission unit 188 includes an identifier that can identify the own video phone 100, for example, the telephone number (number display) of the own video phone 100, identification information of a unique voice quality conversion filter used for voice quality conversion, The image editing unit 162 and the voice quality conversion unit 166 are allowed to function only when identification information of a unique first voice quality conversion filter, which will be described later, is transmitted to the other party's videophone. With this configuration, the other party can specify the speaker who made the call, and can avoid impersonation of the target speaker and obstacles based on the fact that the speaker cannot be specified.

  FIG. 3 is an external view showing an example of an interface arrangement with a speaker in the videophone 100 described above. The speaker acquires the image and sound of the other party through the display unit 150 and the speaker 186, and transmits his image and voice to the other party through the imaging unit 160 and the voice input unit 164. Then, the target speaker to be imitated is selected by the speaker selection switch 152, and the imitation switch 174 is pressed when actually imitating.

  In the present embodiment, the mimic switch 174 is provided on the lower side of the display unit 150. However, the present invention is not limited to such an arrangement, and if the input by the imaging unit 160 is not blocked, You may arrange | position in any of a back surface. Further, by integrating the function of the mimic switch 174 into the existing speaker selection switch 152, the speaker selection switch 152 may function as the mimic switch 174 during a call. Furthermore, the mimicking switch 174 can be realized by an existing pointing device.

  Also, a plurality of mimic switches 174, a plurality of selections in the jog dial format, a plurality of selections through the speaker selection switch 152 or a pointing device, a plurality of facial expressions such as emotions of the same target speaker, or A plurality of target speakers (Mr. A, Mr. B, etc.) can be selected at any time. Therefore, the speaker can decide at the same time the timing and which facial expression or which target speaker will imitate, and more clearly emphasize the imitation intended by the speaker to the other party. It becomes possible.

  In the present embodiment, such a video phone 100 is used to convert the voice quality of the speaker to the voice quality of the target speaker, and “imitate” is performed. In such “imitation”, not only the voice of the target speaker but also the image related to the target speaker is transmitted to the video phone of the other party, so that the other party hears and identifies who the target speaker is. It becomes possible to judge intuitively visually.

(Call method)
There is also provided a calling method in which the above-described videophone 100 is used to convert the voice and image of the speaker into the voice and image of the target speaker and transmit the voice and image to the other party.

  FIG. 4 is a flowchart showing the processing flow of the calling method. In such a call method, an image 192 related to the target speaker and a voice quality conversion filter 194 are stored in the telephone data storage unit 148 in advance.

  The speaker selection unit 154 of the video phone 100 selects a target speaker based on the selection action of the speaker, and the data selection unit 156 selects the target selected by the speaker selection unit 154 from the telephone data storage unit 148. A target speaker related image, which is an image 192 related to the speaker, and a voice quality conversion filter 194 corresponding to the target speaker are selected (S200). Thereafter, the telephone number of the other party is input to open the telephone line with the other party, and the call is started (S202).

  Next, the central control unit 146 of the video phone 100 determines whether or not the mimic switch 174 is valid by the imitation switch 174 being pressed by the speaker (S204), and if the mimic switch 174 is determined to be valid. Execute the mimicking process.

  As the imitation process, first, the image editing unit 162 edits the image input from the imaging unit 160 based on the image 192 related to the target speaker (S206).

  FIGS. 5, 6, and 7 are explanatory diagrams illustrating changes in the display unit 150 due to the image editing step (S206). The image editing unit 162 of the video phone 100 reads the target speaker related image from the telephone data storage unit 148 and superimposes it on an arbitrary position of the input image.

  For example, in FIG. 5, while the mimic switch 174 is valid, the image of the target speaker is overwritten on the entire image, and the image of the target speaker is represented on the entire display unit 150. In FIG. 6, the face image 252 of the target speaker is overwritten on the portion 250 corresponding to the speaker's face. Further, in FIG. 7, the target speaker 260 is displayed in an area where the recognition of the speaker is not hindered, for example, the background image of the speaker, while leaving the image of the speaker.

  The format of the image related to the target speaker used here may be a still image or a moving image, and the content of the image is a live-action image such as a real person such as an actor or a voice actor or a person who has passed away. It may be a character such as an animation or a computer graphic.

  Subsequently, the voice quality conversion unit 166 converts the voice quality of the speaker input from the voice input unit 164 into the voice quality of the target speaker using the voice quality conversion filter 194 (S208). If it is determined that the mimic switch 174 is invalid, the image editing unit 162 and the voice quality conversion unit 166 do not function, and the transmitted image and sound remain the speaker's image and sound.

  Finally, the telephone transmission unit 168 transmits the image and sound generated in the image editing step (S206) and the voice quality conversion step (S208) to the call partner (S210). Here, when it is determined that the call is continued (S212) and the call is not terminated, the processing from the validity determination step (S204) of the mimic switch 174 is repeated until the termination of the call is detected.

  Such a call method allows the speaker to intentionally control the timing of the imitation, so that the imitation can be executed only for the utterance that he / she wants to imitate, and that portion is emphasized and communicated to the other party. be able to.

  Also provided are a program that causes a computer to execute the communication method described above, and a computer-readable storage medium that stores the program.

(Second Embodiment)
In the first embodiment, the operation of converting the image of the speaker into the image of the target speaker at the timing when the speaker intentionally operates has been described. However, in the second embodiment, in addition to or independently of the operation, An operation for converting an image according to a change in the utterance of the speaker will be described in detail.

(Video phone 300)
FIG. 8 is a functional block diagram showing a schematic configuration of the video phone 300 in the second embodiment. The video phone 300 includes a central control unit 146, a telephone data storage unit 148, a display unit 150, a speaker selection switch 152, a speaker selection unit 154, a data selection unit 156, an imaging unit 160, and an image. Editing unit 162, voice input unit 164, speaker state information generation unit 310, voice detection unit 312, voice recognition unit 314, phoneme recognition unit 316, voice quality conversion unit 166, telephone transmission unit 168, Modulation transmission unit 170, antenna unit 172, imitation switch 174, reception demodulation unit 176, reception unit 180, image display unit 182, audio output unit 184, speaker 186, and function permission unit 188 are included. Consists of.

  The central control unit 146, the telephone data storage unit 148, the display unit 150, the speaker selection switch 152, the speaker selection unit 154, the data selection unit 156, which have already been described as the constituent elements in the first embodiment, Imaging unit 160, voice input unit 164, voice quality conversion unit 166, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, imitation switch 174, reception demodulation unit 176, reception unit 180, Since the image display unit 182, the audio output unit 184, the speaker 186, and the function permission unit 188 have substantially the same functions, redundant description is omitted, and here, the speaker state information generation unit 310 having a different configuration is used. The voice detection unit 312, the voice recognition unit 314, the phoneme recognition unit 316, and the image editing unit 162 will be mainly described.

  The speaker state information generation unit 310 generates speaker state information that is information related to the state of the speaker. Such speaker state information may be information indicating the voice state of the speaker input from the voice input unit 164, for example, voice presence / absence information, utterance content information, phoneme information, which will be described later, or other implementations. Information indicating the state of the speaker's image input from the imaging unit 160 described in the embodiment, for example, face position information, face tilt information, face size information, eye opening / closing information, and mouth opening / closing information may be used. The image editing unit 162 can change the target speaker related image used for editing in accordance with the speaker state information. Thus, by editing the target speaker-related image uniformly through the speaker state information, the image editing unit 162 edits the image input from the imaging unit 160 by decoding only the speaker state information. It becomes possible to do.

  In particular, in the present embodiment, speaker state information relating to a change in the speaker's utterance is generated, and the image editing unit 162 edits the image in accordance with the change in the speaker's utterance. Hereinafter, the operation of the videophone 300 will be described with a specific example of the speaker state information.

  The voice detection unit 312 determines the presence or absence of the voice of the speaker, and transmits to the speaker state information generation unit 310 that a voice of a predetermined level (threshold) or more has been detected. A plurality of such predetermined levels may be provided, and the presence / absence of sound can be detected stepwise, that is, the rough volume of the sound can be detected. The speaker state information generation unit 310 includes information related to the presence or absence of detected speech, for example, information indicating the presence or absence of speech as a flag or speech presence or absence information indicating the amplitude of speech as a numerical value at a predetermined level. The speaker state information is generated, and the image editing unit 162 grasps the presence / absence of the speaker's voice from the voice presence / absence information of the speaker state information, and changes the target speaker-related image used for editing.

  In this way, by changing the image according to the presence or absence of voice, when there is voice, that is, when the speaker is imitating, emphasizing that imitation is being performed by changing the image related to the target speaker can do. Moreover, even when there is no sound, the other party's interest can be maintained without getting bored by displaying another image related to the target speaker.

  For example, the image editing unit 162 may superimpose an image of the speaker when there is no sound, and an image related to the target speaker over the input image when there is sound, according to the sound presence / absence information. When there is no voice, a still image of the target speaker may be superimposed on the image, and when there is a voice, a moving image of the target speaker, for example, a moving image with an open / closed mouth (a mouth image) may be superimposed on the image.

  FIG. 9 is an explanatory diagram illustrating changes in the display unit 150 due to opening and closing of the mouth. Here, the target speaker is overwritten on the entire speaker, and the opening and closing of the target speaker's mouth is adjusted. For example, the image editing unit 162 changes the state in which the mouth is completely closed 330, the half-open state 332, the open state 334, and the like according to the presence or absence of the speaker's speech. As for the opening and closing images of the mouth, an arbitrary number of still images of target speakers with different opening and closing degrees of the mouth may be prepared, or the entire image of the target speaker is fixed and only the opening and closing portions of the mouth May be prepared by overwriting the target speaker.

  FIG. 10 is a timing chart for explaining the determination of the presence or absence of sound by the sound detection unit 312. In FIG. 10, it is detected whether or not the amplitude of the sound exceeds a predetermined level, and “no sound” is determined in the region 350 where the sound amplitude is equal to or lower than the predetermined level, and “sound is present” in the region 352 where the sound amplitude is equal to or higher than the predetermined level. Is done. Further, the determination criterion may be to detect not only the amplitude of the voice but also whether it is a human voice from the waveform of the voice, and to determine “sound is present” only when the voice is a human voice.

  Here, when it is strictly determined whether the amplitude of the voice is equal to or higher than a predetermined level, the display of the target speaker displayed on the display unit 150 frequently changes according to the fluctuation of the amplitude. In order to avoid such a phenomenon, the voice detection unit 312 may be provided with a minimum execution time of image change or a hysteresis characteristic. For example, once the amplitude of the sound exceeds a predetermined level, the “sound is present” is maintained for a predetermined time regardless of the fluctuation of the amplitude, and when the predetermined level is exceeded again during that time, the predetermined time is counted from that point. It is possible to fix it. Thus, the problem that the image on the display unit 150 changes complicatedly can be solved.

  The determination of the presence / absence of sound by the sound detection unit 312 may be performed only while the mimic switch 174 is valid. That is, the speaker starts voice conversion by pressing the mimic switch 174, and changes the image to the target speaker only when the voice of a predetermined level or higher is emitted. On the other hand, when the mimic switch 174 is invalid, neither image nor audio conversion is performed. Thus, the speaker can save the trouble of pressing the mimic switch 174 for each utterance.

  Further, the image editing unit 162 does not always edit an image only when “sound is present”, but may edit an image even when “sound is not present”. For example, if “sound is present”, the target speaker's face image is superimposed on the position of the speaker's face, and if “sound is not present”, the target speaker can be hung around the speaker's face, It may be possible to make them amiable by shaking them. That is, even in the case of “no sound”, it is possible to express an image that is linked to the sound to be imitated.

  In this way, even when there is no sound, it is possible to maintain the other party's interest without overwhelming the other party by overlaying other images, and call who imitates what is being imitated from now on The other party can be notified and the image related to the target speaker when the voice is present, that is, when the speaker is imitating, can be emphasized.

  The voice recognition unit 314 extracts information (language information) related to the semantic content included in the voice (speech waveform) of the speaker input from the voice input unit 164, recognizes the semantic content, and is set in advance, for example. A word or sentence such as a keyword or the like is compared with the utterance content of the utterer, and if they match, the utterance state information generation unit 310 is notified of the match. The speaker state information generation unit 310 generates speaker state information including information related to the recognized utterance content, for example, utterance content information such as a keyword in which the spoken voice is replaced with a character string and an identifier associated with the keyword. Then, the image editing unit 162 recognizes that the keyword matches from the utterance content information of the speaker state information, and superimposes the images related to the target speaker linked with the keyword. For example, when a target speaker uses a keyword together with a predetermined determined pose, an image representing the predetermined determined pose is overlaid in conjunction with the keyword.

  Further, the voice recognition unit 314 recognizes the meaning content of the voice related to the operation of the speaker, for example, sneezing, laughing voice, yawning, tongue-brushing, bruxing, whistling, etc. You may tell. In this case, the image editing unit 162 superimposes an image related to the target speaker corresponding to the operation of the speaker, for example, a moving image that is sneezing, while maintaining the imitation state, and the speaker It is possible to transmit the status of the call to the other party in real time.

  The phoneme recognition unit 316 performs speech recognition on the voice of the speaker input from the voice input unit 164 and grasps the phoneme of the voice. For example, the phoneme recognition unit 316 determines the type of a single sound of the voice, that is, a consonant or a vowel. This is notified to the speaker status information generation unit 310. The speaker state information generating unit 310 generates speaker state information including information related to the recognized utterance content, for example, utterance content information such as a consonant flag or a phoneme replaced with characters. Based on the utterance content information of the speaker state information, the image editing unit 162 determines whether the utterance being performed is a consonant or a vowel, and the degree of opening and closing of the mouth of the image related to the target speaker depending on what the consonant and the vowel are. The degree of opening / closing of the mouth can be adjusted according to the volume of the sound. Since the image of the degree of opening and closing of the mouth has already been described with reference to FIG. 9, it is omitted here.

  With the configuration that reflects the movement of the speaker's mouth in the target speaker's image, it is possible to send an image as if the target speaker is actually speaking to the other party. It becomes possible to express.

  As described above, the image editing unit 162 associates an arbitrary sound and an image in advance, and when the arbitrary sound is detected by the sound detection unit 312, the sound recognition unit 314, or the phoneme recognition unit 316, the associated sound is associated. Edit using images. With this configuration, it is possible to transmit an image suitable for the utterance of the speaker to the other party, and to dynamically imitate.

(Third embodiment)
In the third embodiment, the operation of converting an image when a speaker moves visually will be described in detail in addition to or independently of the operation of the first embodiment and / or the operation of the second embodiment. .

(Video phone 400)
FIG. 11 is a functional block diagram showing a schematic configuration of the video phone 400 according to the third embodiment. The video phone 400 includes a central control unit 146, a telephone data storage unit 148, a display unit 150, a speaker selection switch 152, a speaker selection unit 154, a data selection unit 156, an imaging unit 160, an utterance, Person status information generation unit 310, image recognition unit 410, image editing unit 162, voice input unit 164, voice quality conversion unit 166, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, and imitation The switch 174, the reception demodulation unit 176, the reception unit 180, the image display unit 182, the audio output unit 184, the speaker 186, and the function permission unit 188 are configured.

  The central control unit 146, the telephone data storage unit 148, the display unit 150, the speaker selection switch 152, the speaker selection unit 154, the data selection unit 156, which have already been described as the constituent elements in the first embodiment, Imaging unit 160, speaker state information generation unit 310, voice input unit 164, voice quality conversion unit 166, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, mimicking switch 174, and reception demodulation The unit 176, the receiving unit 180, the image display unit 182, the audio output unit 184, the speaker 186, and the function permission unit 188 are substantially the same in function, so redundant description is omitted here. The different image recognition unit 410 and image editing unit 162 will be mainly described.

  The image recognition unit 410 recognizes the state of the speaker's image input from the imaging unit 160 and transmits the image to the speaker state information generation unit 310. The speaker state information generation unit 310 generates speaker state information including the recognized image state. Then, the image editing unit 162 changes the target speaker related image used for editing in accordance with the speaker state information. Here, the image recognition is a system that performs recognition processing on an image input from the imaging unit 160 and recognizes the meaning and content of the image. Here, the existing image recognition technology is used to recognize the coordinates in the captured image of a characteristic part such as the speaker's face and its eyes and mouth, and to move left and right, up and down, blink, open and close the mouth Such changes can be grasped.

  With this configuration, the image editing unit 162 can change the target speaker-related image in conjunction with the movement of the speaker, particularly the actual movement of the face. Therefore, the speaker can visually and dynamically imitate the target speaker, and can emphasize the fact that the speaker is imitating the target speaker. . Hereinafter, the operation of the video phone 400 will be described with a specific example of the speaker state information.

  In addition, when the image recognition unit 410 recognizes that there is a speaker's face, the image recognition unit 410 detects the position of the face, and the speaker state information generation unit 310 includes information related to the detected face position, for example, a display unit. The speaker state information including face position information such as absolute or relative coordinates in 150 or absolute or relative coordinates with respect to the speaker's image is generated, and the image editing unit 162 sets the target position to the face position corresponding to the face position information. Speaker related images can be overlaid. With this configuration, the speaker can change the position of the target speaker-related image to be transmitted to the other party by moving his / her face.

  Further, when the image recognition unit 410 recognizes that there is a speaker's face, the image recognition unit 410 detects the inclination of the face, and the speaker state information generation unit 310 includes information relating to the detected face inclination, for example, a display unit It is also possible to generate speaker state information including face tilt information such as an absolute angle with respect to 150 and a relative angle with respect to the speaker, and the image editing unit 162 rotates and overlaps the target speaker related image according to the face tilt information. Good. Here, the inclination of the face may be calculated from the arrangement of both eyes of the speaker. Further, the direction in which the eyes are facing and the blink may be changed in conjunction with each other. With this configuration, the speaker can rotate the target speaker-related image to be transmitted to the other party by changing the inclination of his / her face.

  When the image recognizing unit 410 recognizes that there is a speaker's face, the image recognizing unit 410 detects the size of the face. The speaker state information generating unit 310 includes information related to the detected face size, for example, a display unit. The speaker state information including face size information such as the area occupied by the speaker's face with respect to 150 and the area ratio with respect to the entire display unit 150 is generated, and the image editing unit 162 determines the target speaker according to the face size information. The related image may be enlarged or reduced. With this configuration, the speaker can change the size of the target speaker-related image to be transmitted to the other party by changing the size of his / her face, that is, the distance to the imaging unit 160.

  When the image recognition unit 410 recognizes that there is a speaker's face, the image recognition unit 410 detects opening / closing of the eyes of the speaker, and the speaker state information generation unit 310 detects information relating to the detected opening / closing of the eyes, for example, eyes Utterance state information is generated including eye opening / closing information indicating the opening / closing of the eye and the eye opening / closing information indicating the degree of opening / closing of the eyes with a numerical value at a predetermined stage, and the image editing unit 162 determines the target story according to the eye opening / closing information. The eyes in the person-related image may be opened and closed. With the configuration that reflects the opening / closing operation of the speaker's eyes in the target speaker's image, it is possible to send an image as if the target speaker is actually blinking to the other party. It is possible to express imitations.

  When the image recognition unit 410 recognizes that there is a speaker's face, the image recognition unit 410 detects opening / closing of the mouth of the speaker, and the speaker state information generation unit 310 detects information related to opening / closing of the mouth, for example, mouth Utterance state information is generated including mouth opening / closing information indicating the opening / closing of the mouth as a flag and mouth opening / closing information indicating the opening / closing degree of the mouth by a numerical value at a predetermined stage, and the image editing unit 162 generates the target story according to the mouth opening / closing information. The mouth in the person-related image may be opened and closed. With the configuration that reflects the opening and closing movements of the speaker's mouth in the target speaker's image, it is possible to send an image as if the target speaker is actually speaking to the other party, more imitating it. Can be expressed.

  With such a configuration, it is possible to transmit to the call partner in correspondence with the target speaker up to the detailed operation of the speaker's face, and it is possible to express imitation more dynamically.

(Fourth embodiment)
In the fourth embodiment, further characteristics relating to the conversion of voice quality between the speaker and the target speaker will be described.

(Voice quality conversion unit 166)
When converting the voice quality of the speaker to the voice quality of the target speaker, the voice quality conversion filter 194 is usually used as a conversion function for converting from a specific speaker to a specific target speaker. Here, the voice quality conversion filter 194 records and accumulates the speech of the speaker and the target speaker, and is updated with a learning function for learning the correspondence between the speech of the speaker and the target speaker from the accumulated speech. It may be done.

FIG. 12 is an explanatory diagram for explaining a voice quality conversion filter 194 used for voice quality conversion. In FIG. 12, a voice quality conversion filter 452 that changes the voice quality of M (M is an integer) speakers 110A, 110B, 110C to the voice quality of N (N is an integer) target speakers 450A, 450B, 450C is shown. Has been. As can be seen with reference to the figure, in order to cover all voice quality conversions, M × N voice quality conversion filters f _AA , f _AB , f _AC , f _BA , f _BB , f _BC , f _CA , f _CB and _fCC are required.

  In order to realize such voice quality conversion, a unique conversion function corresponding to the combination of the voice quality of the speaker and the voice quality of the target speaker must be generated for M × N described above. Therefore, the preparation takes time, and an arbitrary target speaker cannot be easily selected.

  The voice quality conversion unit 166 according to the present embodiment is not a direct voice quality conversion filter 194 from the voice quality of the speaker to the voice quality of the target speaker as described above, but an intermediate story that is a speaker provided in common between them. It is possible to take a configuration of a two-stage voice quality conversion filter that relays a person's voice. That is, the voice quality conversion unit 166 converts the voice quality of each speaker into a common intermediate speaker voice quality, and converts the voice quality of the intermediate speaker into the voice quality of each target speaker. The voice quality is converted using the second voice quality conversion filter.

  Further, the data selection unit 156 selects the first voice quality conversion filter and the second voice quality conversion filter as the voice quality conversion filter 194, and the voice quality conversion unit 166 determines the voice quality of the voice of the speaker input from the voice input unit. Converting the voice quality of the intermediate speaker using the selected first voice quality conversion filter, and further converting the voice quality of the intermediate speaker to the voice quality of the target speaker using the selected second voice quality conversion filter. it can.

  Here, the intermediate speaker can be a person or TTS (Text-to-Speech), and is constructed by a service provider that generates and provides a second voice quality conversion filter. For the basic concept of such technology, the technical content of Japanese Patent Application No. 2005-349754 by the present applicant can be referred to.

  FIG. 13 is an explanatory diagram for explaining voice quality conversion by the first voice quality conversion filter 462 and the second voice quality conversion filter 464. In FIG. 13, intermediate speakers 460 are provided between M (M is an integer) speakers 110A, 110B, 110C and N (N is an integer) target speakers 450A, 450B, 450C. The voice quality of the speakers 110A, 110B, and 110C is once converted to the voice quality of the intermediate speaker 460 and then converted to the voice quality of the target speakers 450A, 450B, and 450C.

Therefore, in order to realize voice quality conversion, M first voice quality conversion filters f _AM , f _BM , f _CM from the voice quality of the speakers 110A, 110B, 110C to the voice quality of the intermediate speaker, and the voice quality of the intermediate speaker Need only to prepare N second voice quality conversion filters f _MA , f _MB , f _MC to voice quality of target speakers 450A, 450B, 450C, and to cover all voice quality conversions, A voice quality conversion filter is sufficient. Accordingly, it is possible to reduce the production cost and the storage capacity. Also, when the first and second voice quality conversion filters are accompanied by a learning function, the learning function of the first voice quality conversion filter only needs to be ensured in the video phone owned by the speaker, and the learning burden is reduced.

  With this two-stage voice quality conversion filter configuration via the intermediate speaker, once the speaker prepares the first voice quality conversion filter 462, it is not necessary to generate a voice quality conversion filter every time the target speaker is changed. This is because a large number of second voice quality conversion filters 464 having a common intermediate speaker 460 as a voice quality conversion source can be used. Therefore, if only the second voice quality conversion filter 464 of an arbitrary target speaker is downloaded, it can be imitated immediately together with the first voice quality conversion filter 462.

  Also, the service provider who provides the second voice quality conversion filter 464 to such a speaker does not need to prepare a voice quality conversion filter for each speaker, and at least one second voice quality conversion filter 464 is provided for each target speaker. The second voice quality conversion filter 464 can be applied to a plurality of speakers. Therefore, it is possible to build an efficient system at a low cost, and it is possible to form a large number of patterns of a speaker and a target speaker with a small load.

  The first voice quality conversion filter 462 may be created by registering the voice of the speaker 110 in advance and associating it with the voice of the intermediate speaker 460 in units of segments. This filter creation function may be provided in the video phone itself, or may be provided in a separate device such as the server 140. When the filter creation function is generated by the server 140, a communication network such as the Internet, wireless communication, infrared communication It may be downloaded to a video phone via a recording medium.

(Server 140)
The first voice quality conversion filter 462 or the second voice quality conversion filter 464 can be freely downloaded from the server 140 used in the voice quality conversion / image editing service providing system described above, together with the image 192 related to the target speaker. .

  FIG. 14 is a functional block diagram illustrating a schematic configuration of the server 140 according to the fourth embodiment. The server 140 includes a server data storage unit 480 and a server transmission unit 482.

  The server data storage unit 480 stores an image 192 related to the target speaker and a voice quality conversion filter 194 that converts the voice quality of the speaker into the voice quality of the target speaker. The image 192 and the voice quality conversion filter 194 related to the target speaker may be integrally stored as package data.

  Further, when the voice quality conversion filter 194 includes the first voice quality conversion filter 462 and the second voice quality conversion filter 464 as the two-stage voice quality conversion filter including the intermediate speaker as described above, the server data storage unit 480 may store either one or both of the first voice quality conversion filter 462 and the second voice quality conversion filter 464.

  The server transmission unit 482 receives the image 192 related to the target speaker desired by the speaker and the voice quality conversion filter 194 stored in the server data storage unit 480 in response to the request of the speaker. Send. Here, the server transmission unit 482 may transmit the video data to the videophone 100 in units of package data described above.

  Further, when the voice quality conversion filter 194 includes the first voice quality conversion filter 462 and the second voice quality conversion filter 464, the server transmission unit 482 has either the first voice quality conversion filter 462 or the second voice quality conversion filter 464. Alternatively, both are read from the server data storage unit 480 and transmitted to the videophone 100.

  In the videophone 100, for example, when a speaker is fixed and the first voice quality conversion filter 462 stored in the telephone data storage unit 148 is designated in advance, the designated first voice quality conversion filter 462 is used. In use, the data selection unit 156 selects the second voice quality conversion filter 464 from the data storage unit 148. The voice quality conversion unit 166 uses the designated first voice quality conversion filter 462 and second voice quality conversion filter 464 to convert the voice quality of the speaker's voice into the voice quality of the target speaker selected by the speaker selection unit 154. Convert to

  Downloading such package data is not limited to the use of the voice quality conversion / image editing service providing system, but can be executed via a communication network such as the Internet, wireless communication, infrared communication, and a recording medium. Furthermore, addition, update, and deletion can be freely performed in the video phone. When there are a plurality of downloaded package data, the target speaker selected by the speaker selection unit 154 becomes the target of imitation.

  Also, a plurality of package data may be prepared for the same target speaker, although the voice (voice quality) is common but the images are different. For example, when the speaker utters “Good morning”, an image related to the target speaker of one package can be opened and closed (mouth packed), and the other package can be bowed. In addition, various modifications are possible, such as including the same target speaker but different images such as clothes and hairstyle.

(Fifth embodiment)
In the fifth embodiment, further characteristics relating to voice quality conversion between the speaker and the target speaker based on the voice quality conversion unit 166 in the fourth embodiment will be described.

(Voice quality conversion unit 166)
FIG. 15 is a functional block diagram showing a schematic configuration of a video phone 500 according to the fifth embodiment. The video phone 500 includes a central control unit 146, a telephone data storage unit 148, a display unit 150, a speaker selection switch 152, a speaker selection unit 154, a data selection unit 156, an imaging unit 160, an image, Editing unit 162, voice input unit 164, speech type selection unit 510, voice quality conversion unit 166, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, mimic switch 174, reception demodulation unit 176 A receiving unit 180, an image display unit 182, an audio output unit 184, a speaker 186, a function permission unit 188, and a voice quality conversion filter synthesis unit 512.

  The central control unit 146, the display unit 150, the speaker selection switch 152, the speaker selection unit 154, the imaging unit 160, the image editing unit 162, and the voice input already described as the constituent elements in the first embodiment Unit 164, telephone transmission unit 168, modulation transmission unit 170, antenna unit 172, imitation switch 174, reception demodulation unit 176, reception unit 180, image display unit 182, audio output unit 184, speaker 186 and the function permission unit 188 have substantially the same functions, and thus a duplicate description is omitted. Here, the telephone data storage unit 148, the utterance type selection unit 510, the voice quality conversion unit 166, which have different configurations, The image editing unit 162 and the voice quality conversion filter synthesis unit 512 will be mainly described.

  The telephone data storage unit 148 stores two first voice quality conversion filters 514 that convert a clear voice quality of a speaker and a dense voice quality (whisper voice: NAM) into a voice quality of an intermediate speaker. Here, NAM (Non-Auditable Murmur), which is defined as a dense voice quality, is an unvoiced exhalation of soft tissue transmission that is not accompanied by vocal cord vibration that is tuned by utterances that are not audible to the surrounding people and filter characteristics of the speech organs. Say the sound.

  The utterance type selection unit 510 first refers to the first voice quality conversion filter 514 with different utterance types stored in the telephone data storage unit 148 via the data selection unit 156, and determines the voice quality of the speaker. An utterance type list 516 of the first voice quality conversion filter 514 that can be selected is created and transmitted to the display unit 150. Here, when the speaker selects clear voice quality or dense voice quality from the speech type list 516 displayed on the display unit 150 through the speaker selection switch 152, the speaker selection unit 154 selects the selected voice quality. The transmitted utterance type is transmitted to the data selection unit 156.

  The data selection unit 156 converts the clear voice quality of the speaker into the voice quality of the intermediate speaker according to the speech type selected by the speech type selection unit 510 (speaker's speech status). Either the first voice quality conversion filter 514 or the first voice quality conversion filter 514 that converts the voice quality of the speaker into the voice quality of the intermediate speaker is selected, and the voice quality conversion unit 166 selects the first voice quality selected. The conversion filter 514 is used to convert the voice quality of the speaker's voice to the voice quality of the target speaker.

  As described above, when a dense utterance type is selected by the utterance type selection unit 510, the data selection unit 156 converts the voice quality of the intermediate speaker into the clear voice quality of the speaker himself as the second voice quality conversion filter 464. A filter may be automatically selected. In this case, both the speaker and the target speaker are themselves.

  With such a configuration, the other party can surely grasp the voice intended by the speaker regardless of the situation where the speaker is placed and the size of the speaker's utterance. It is also possible to transmit utterances on the train to the other party.

  When the utterance type selection unit 510 selects a secret utterance type, the image editing unit 162 can superimpose a display image indicating that the speaker is speaking secretly.

  FIG. 16 is an external view showing overwriting of an image by the image editing unit 162. Here, it shows that the speaker is secretly speaking in an area that does not interfere with the recognition of the speaker in the display unit 150, for example, the background image of the speaker, while leaving the image of the speaker. An image, for example, a symbol 520 indicating a secret story with an index finger applied to B is displayed.

  The first voice quality conversion filter 462 that converts the voice quality of the speaker into the voice quality of the intermediate speaker, and the voice quality of the intermediate speaker, even if the speaker is speaking densely, Through the second voice quality conversion filter 464 that converts the voice quality, the other party can hear the clear voice of the speaker. Therefore, although the speaker is speaking with a voice quality that is denser than the speaking environment, the other party cannot grasp it, but by displaying the symbol 520 as described above, the other party can It becomes possible to grasp the situation.

  The voice quality conversion filter synthesis unit 512 synthesizes the first voice quality conversion filter 462 and the second voice quality conversion filter 464 to generate a synthesis filter that directly converts the voice quality of the speaker into the voice quality of the target speaker. Then, after the synthesis is completed, the voice quality conversion unit 166 performs voice quality conversion using the synthesized synthesis filter.

  In the video phone 500, the first voice quality conversion filter 462 and the second voice quality conversion filter 464 are individually downloaded. However, after that, it is not necessary to maintain the voice quality conversion filter in two stages. Therefore, it is possible to reduce the load and power consumption for voice quality conversion and speed up voice quality conversion by using only the synthesized filter instead of the two-stage voice quality conversion filter for each utterance. . Naturally, the voice quality conversion filter synthesis unit 512 can also be applied to the first voice quality conversion filter 462 and the second voice quality conversion filter 464 of the fourth embodiment.

  As mentioned above, although preferred embodiment of this invention was described referring an accompanying drawing, it cannot be overemphasized that this invention is not limited to the example which concerns. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  For example, in the embodiment described above, all the components are provided in the video phone of the speaker. However, the present invention is not limited to this, and some of the components are included in a server that can be accessed via the communication network. May be transmitted to the other party through voice and image conversion by this server during a call. Also, the above-mentioned components are provided on the other party's video phone, and the voice and image from the speaker are sent to the other party. It is also possible to convert in a video phone.

  In the above-described embodiment, the function permission unit is provided in the video phone on the speaker side. However, the function permitting unit may be arranged on the video phone on the call partner side to limit the imitation of the speaker on the call partner side. Is possible.

  In the above-described embodiment, an example in which an image related to the target speaker is changed based on the image and sound of the speaker has been described. However, the present invention is not limited to this example. When detected as a keyword, an image related to the target speaker may be displayed in response to the detection, and pinpoint imitation may be performed in which the characteristic way of speaking is represented by the voice of the target speaker again.

  Furthermore, in the above-described embodiments, human voices such as actors and voice actors are described as target speakers, but the present invention is not limited to such cases, and various sounds such as animal calls and sounds generated from inanimate objects are used. It is also possible to adapt. In addition, the video phone is not limited to being wireless, and may be connected to a communication network via a wired line.

  Note that each step in the calling method of the present specification does not necessarily have to be processed in chronological order according to the order described in the flowchart, but is performed in parallel or individually (for example, parallel processing or object processing). ) May also be included.

  The present invention can be applied to a video phone, a telephone call method, a program, a voice quality conversion / image editing service providing system, and a server that change an image together with voice quality conversion in a video phone.

It is explanatory drawing for demonstrating the voice quality conversion and image editing service provision system which uses the video telephone in 1st Embodiment. It is a functional block diagram showing a schematic configuration of a video phone. It is the external view which showed the example of interface arrangement with the speaker in a videophone. It is the flowchart which showed the flow of the process of a telephone call method. It is explanatory drawing explaining the change of the display part by an image editing step. It is explanatory drawing explaining the change of the display part by an image editing step. It is explanatory drawing explaining the change of the display part by an image editing step. FIG. 5 is a functional block diagram illustrating a schematic configuration of a video phone according to a second embodiment. It is explanatory drawing explaining the change of the display part by a mouth pack. It is a timing chart for demonstrating the judgment of the audio | voice by a speech recognition part. It is the functional block diagram which showed the schematic structure of the video telephone in 3rd Embodiment. It is explanatory drawing for demonstrating the voice quality conversion filter utilized for conversion of a voice quality. It is explanatory drawing for demonstrating the voice quality conversion by the 1st voice quality conversion filter and 2nd voice quality conversion filter in 4th Embodiment. It is the functional block diagram which showed the schematic structure of the server in 4th Embodiment. It is the functional block diagram which showed the schematic structure of the video telephone in 5th Embodiment. It is the external view which showed overwriting of the image of the said image editing part.

Explanation of symbols

100, 300, 400, 500 Videophone 148 Telephone data storage unit 152 Speaker selection switch 154 Speaker selection unit 156 Data selection unit 160 Imaging unit 162 Image editing unit 164 Voice input unit 166 Voice quality conversion unit 168 Telephone transmission unit 170 Modulated transmission Unit 174 imitation switch 176 reception demodulation unit 180 reception unit 188 function permission unit 192 images 194 and 452 related to the target speaker voice quality conversion filter 310 speaker state information generation unit 312 voice detection unit 314 voice recognition unit 316 phoneme recognition unit 410 image Recognition unit 460 Intermediate speaker 480 Server data storage unit 482 Server transmission unit 462, 514 First voice quality conversion filter 464 Second voice quality conversion filter 510 Utterance type selection unit 512 Voice quality conversion filter synthesis unit

Claims (25)

A video phone comprising an imaging unit for inputting an image of a speaker and a voice input unit for inputting the voice of the speaker, and converting a voice quality of the voice of the speaker into a voice quality of a target speaker,
A telephone data storage unit that prestores an image associated with the target speaker and a voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the target speaker;
A speaker selection unit for selecting the target speaker;
A data selection unit for selecting a target speaker-related image, which is an image related to the target speaker selected by the speaker selection unit, and a voice quality conversion filter corresponding to the target speaker from the telephone data storage unit;
An image editing unit that edits an image input from the imaging unit based on the target speaker-related image;
A voice quality conversion unit that converts the voice quality of the voice of the speaker input from the voice input unit to the voice quality of the target speaker using the selected voice quality conversion filter;
A telephone transmission unit that transmits the image edited by the image editing unit and the voice converted by the voice quality conversion unit to a call partner;
A video phone, comprising: It also has a mimic switch,
The video phone according to claim 1, wherein the image editing unit and the voice quality conversion unit function while the imitation switch is valid. A speaker state information generating unit for generating speaker state information which is information relating to the state of the speaker;
The video phone according to claim 1, wherein the image editing unit changes the target speaker-related image used for editing in accordance with the speaker state information. A voice detection unit for detecting the presence or absence of the voice of the speaker input from the voice input unit;
4. The videophone according to claim 3, wherein the speaker status information generation unit generates the speaker status information including voice presence / absence information that is information related to the presence / absence of the detected voice. A speech recognition unit for recognizing the speech content of the speech of the speaker input from the speech input unit;
The said speaker state information generation part produces | generates the said speaker state information including the speech content information which is the information which concerns on the said recognized speech content, Either of Claim 3 or 4 characterized by the above-mentioned. Video phone. Further comprising a phoneme recognition unit for recognizing a phoneme of a speaker's voice input from the voice input unit;
The speaker state information generation unit generates the speaker state information including phoneme information that is information related to the recognized phoneme type,
6. The video phone according to claim 3, wherein the image editing unit adjusts a degree of opening and closing of the mouth of the target speaker related image according to the phoneme information. An image recognition unit for recognizing the state of the image of the speaker input from the imaging unit;
The videophone according to any one of claims 3 to 6, wherein the speaker state information generation unit generates the speaker state information including a state of the recognized image. When the image recognition unit recognizes that there is a speaker's face, it detects the position of the face,
The speaker state information generation unit generates the speaker state information including face position information which is information related to the detected face position,
The video phone according to claim 7, wherein the image editing unit superimposes the target speaker-related image on a face position corresponding to the face position information. When the image recognition unit recognizes that there is a speaker's face, it detects the inclination of the face,
The speaker state information generation unit generates the speaker state information including face inclination information that is information relating to the detected face inclination,
The video phone according to claim 8, wherein the image editing unit rotates and superimposes the target speaker related images according to the face tilt information. When the image recognition unit recognizes that there is a speaker's face, it detects the size of the face,
The speaker state information generation unit generates the speaker state information including face size information which is information related to the detected face size,
10. The video phone according to claim 8, wherein the image editing unit enlarges or reduces the target speaker-related image in accordance with the face size information. When the image recognition unit recognizes that there is a speaker's face, the image recognition unit detects opening / closing of the speaker's eyes,
The speaker state information generation unit generates the speaker state information including eye opening / closing information that is information relating to the detected opening / closing of the eyes,
11. The video phone according to claim 8, wherein the image editing unit opens and closes eyes in the target speaker related image according to the eye opening and closing information. When the image recognition unit recognizes that there is a speaker's face, it detects the opening / closing of the speaker's mouth,
The speaker state information generation unit generates the speaker state information including mouth opening / closing information which is information relating to the detected opening / closing of the mouth,
The video phone according to claim 8, wherein the image editing unit opens and closes a mouth in the target speaker-related image according to the mouth opening and closing information. The voice quality conversion filter has a first voice quality conversion filter for converting the voice quality of individual speakers into a voice quality of a common intermediate speaker, and a voice quality conversion filter for converting the voice quality of the intermediate speaker into voice quality of each target speaker. Of the second voice quality conversion filter,
The data selection unit selects the first voice conversion filter and the second voice conversion filter as the voice conversion filter,
The voice quality conversion unit converts the voice quality of the voice of the speaker input from the voice input unit to the voice quality of the intermediate speaker using the selected first voice quality conversion filter, and further The video phone according to any one of claims 1 to 12, wherein voice quality is converted into voice quality of the target speaker using the selected second voice quality conversion filter.   The image processing device further includes a receiving unit that receives an image related to the target speaker used in the image editing unit and a second voice quality conversion filter used in the voice quality conversion unit from an external electronic device. The video phone according to claim 13. It further includes an utterance type selection unit that allows the utterer to select clear or secret utterance,
The data selection unit is configured to convert a clear voice quality of a speaker into a voice quality of an intermediate speaker according to the selected speech type, or a voice quality of the middle speaker is converted to a middle voice of the speaker. 15. The video phone according to claim 13, wherein any one of the first voice quality conversion filters for conversion into voice quality is selected. When a dense utterance type is selected in the utterance type selection unit,
16. The video phone according to claim 15, wherein the data selection unit selects a second voice quality conversion filter that converts the voice quality of the intermediate speaker into a clear voice quality of the speaker. When a dense utterance type is selected in the utterance type selection unit,
The video phone according to claim 15, wherein the image editing unit superimposes a display image indicating that a speaker is speaking secretly.   A voice quality conversion filter synthesizing unit that synthesizes the first voice quality conversion filter and the second voice quality conversion filter to generate a synthesis filter that directly converts the voice quality of the speaker into the voice quality of the target speaker. The video phone according to claim 13.   And a function permission unit that permits the image editing unit and the voice quality conversion unit to function only when an identifier that can identify the own video phone is transmitted to the video phone of the other party. The video phone according to any one of claims 1 to 18. A call method for making a call using a video phone that converts the voice quality of a speaker's voice into the voice quality of a target speaker,
A telephone data storage step of storing an image related to the target speaker and a voice quality conversion filter for converting the voice quality of the speaker into the voice quality of the target speaker in a telephone data storage unit;
A speaker selection step of selecting the target speaker;
A data selection step of selecting a target speaker related image that is an image related to the target speaker selected in the speaker selection step and a voice quality conversion filter corresponding to the target speaker from the telephone data storage unit;
An imaging step of inputting an image of the speaker;
An image editing step of editing the image input in the imaging step based on the target speaker-related image;
A voice input step for inputting the voice of the speaker;
A voice quality conversion step of converting the voice quality of the voice of the speaker input in the voice input step into the voice quality of the target speaker using the selected voice quality conversion filter;
A telephone transmission step of transmitting the image edited in the image editing step and the voice converted in the voice quality conversion step to a call partner;
A call method comprising the steps of: A telephone data storage step of storing an image related to the target speaker and a voice quality conversion filter for converting the voice quality of the speaker into the voice quality of the target speaker in a telephone data storage unit;
A speaker selection step of selecting the target speaker;
A data selection step of selecting a target speaker related image that is an image related to the target speaker selected in the speaker selection step and a voice quality conversion filter corresponding to the target speaker from the telephone data storage unit;
An imaging step of inputting an image of the speaker;
An image editing step of editing the image input in the imaging step based on the target speaker-related image;
A voice input step for inputting the voice of the speaker;
A voice quality conversion step of converting the voice quality of the voice of the speaker input in the voice input step into the voice quality of the target speaker using the selected voice quality conversion filter;
A telephone transmission step of transmitting the image edited in the image editing step and the voice converted in the voice quality conversion step to a call partner;
A program that causes a computer to execute. Consists of a server and a videophone that is communicably connected to the server, and provides a voice quality conversion / image editing service that converts the voice quality of the speaker's voice into the voice quality of the target speaker and edits the image of the speaker A voice quality conversion / image editing service providing system,
The server
A server data storage unit that stores an image related to the target speaker and a voice quality conversion filter that converts the voice quality of the speaker into the voice quality of the target speaker;
A server transmission unit for transmitting an image related to the target speaker and a voice quality conversion filter stored in the server data storage unit to the videophone;
With
The video phone is
A receiver that receives an image associated with the target speaker and a voice quality conversion filter;
A telephone data storage unit that stores an image related to the target speaker and a voice quality conversion filter received by the reception unit;
A speaker selection unit for selecting the target speaker;
A data selection unit for selecting a target speaker-related image, which is an image related to the target speaker selected by the speaker selection unit, and a voice quality conversion filter corresponding to the target speaker from the telephone data storage unit;
An imaging unit for inputting an image of the speaker;
An image editing unit that edits an image input from the imaging unit based on the target speaker-related image;
A voice input unit for inputting the voice of the speaker;
A voice quality conversion unit that converts the voice quality of the voice of the speaker input from the voice input unit to the voice quality of the target speaker using the selected voice quality conversion filter;
A telephone transmission unit that transmits the image edited by the image editing unit and the voice converted by the voice quality conversion unit to a call partner;
A system for providing a voice quality conversion / image editing service, comprising: The voice quality conversion filter includes a first voice quality conversion filter for converting the voice quality of individual speakers to a voice quality of a common intermediate speaker, and a voice quality conversion filter for converting the voice quality of the intermediate speaker to the voice quality of each target speaker. Of the second voice quality conversion filter,
The server transmission unit can transmit one or both of the first voice quality conversion filter and the second voice quality conversion filter,
The receiving unit can receive one or both of the first voice quality conversion filter and the second voice quality conversion filter,
The telephone data storage unit stores a first voice conversion filter and a second voice conversion filter including one or both of the first voice conversion filter and the second voice conversion filter received by the reception unit;
The data selection unit
If the first voice quality conversion filter is designated in advance, the second voice quality conversion filter is selected from the telephone data storage unit,
If the first voice quality conversion filter is not designated in advance, the first voice quality conversion filter and the second voice quality conversion filter are selected from the telephone data storage unit as the voice quality conversion filter;
The voice quality conversion unit
When the first voice quality conversion filter is designated in advance, it is converted to the voice quality of the intermediate speaker using the designated first voice quality conversion filter,
If the first voice quality conversion filter is not designated in advance, the voice quality of the speaker's voice input from the voice input unit is converted to the voice quality of the intermediate speaker using the selected first voice quality conversion filter. Converted to
The voice quality conversion / image editing service providing system according to claim 22, further comprising: converting the voice quality of the intermediate speaker into the voice quality of the target speaker using the selected second voice quality conversion filter. . A server used in the voice quality conversion / image editing service providing system according to claim 22 or 23,
A server comprising the server data storage unit and a server transmission unit. A video phone used in the voice quality conversion / image editing service providing system according to claim 22 or 23,
A video phone comprising the receiving unit, a telephone data storage unit, a speaker selection unit, a data selection unit, an imaging unit, an image editing unit, a voice input unit, a voice quality conversion unit, and a telephone transmission unit.

Images