JP2003248841A - Virtual television intercom - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a communication terminal with a display function which displays a virtual three-dimensional CG character that a receiver selects as a communication party and enables a voice conversation through the CG character. <P>SOLUTION: A communication part 1 performs voice communication and a CG character corresponding to a communication party is selected through a character background selective input part 2. A voice processing part 5 performs voice processing needed for a telephone call. A voice conversion part 6 performs voice conversion and a voice output part 7 outputs a voice. A voice input part 8 obtains the voice. A voice analysis part 9 takes a voice analysis and a feeling estimation part 10 estimates a feeling from a voice analysis result. A lips movement control part 11, a body movement control part 12, and a feeling control part 13 send control information to a three-dimensional drawing part 14 to generate an image, which is displayed at a display part 15. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention
The present invention relates to a virtual video call by a communication terminal device with a display device for the purpose of visually enjoying a voice conversation through a three-dimensional CG (Computer Graphics) character.

[0002]

2. Description of the Related Art Conventionally, what has been called television communication is a device in which a camera and a telephone device equipped with a display device communicate with each other while looking at the face image of the other party taken by the camera. Generally, in order to reduce the amount of transmission, the captured face image data is compressed, multiplexed with audio data, and transmitted to the receiver. On the receiver side, the multiplexed data is separated into audio data and compressed image data, the image data is expanded, and audio output and image display are performed in synchronization with the audio data. Recently, MPEG-4 (Moving Picture
A mobile phone called a videophone for next-generation mobile communication (IMT-2000) based on the standard image compression standard of Experts Group Phase 4) has been developed (see Non-Patent Document 1).

On the other hand, in the case of multiplexing and transmitting images as described above, a broadband communication standard that deviates from the conventional voice communication framework and an infrastructure for realizing it are necessary. Therefore, there is one that attempts to realize a function similar to that of television communication in a pseudo manner only by audio data communication without using such an image compression method (Patent Document 1).
reference. ). This invention is a still image of a face obtained by processing the face image of the other party into a mouthless state in advance, and "A" and "I".
Hold a still image of the mouth shape with the pronunciation of the vowel "u" ... on the phone. The voice data transmitted from the other party is analyzed by vowels of the voice transmitted using the voice recognition technology, and the image in which the mouth shape data corresponding to the analysis result is combined with the face image is displayed at any time to speak the other person. It is to display the state of being. An advantage of the present invention is that pseudo TV communication can be realized within the framework of normal voice communication, but the user feels uncomfortable with a still image that does not move except the mouth, and also has a conversation with the person. I doubt if it feels like they are doing it.

Although it is out of the framework of voice communication, there is also an invention combined with image recognition technology because the amount of data can be reduced rather than transmitting an image (see Patent Document 2). In the present invention, facial expressions and mouth shapes are recognized by image recognition, parameterized, and transmitted together with voice data. The receiving side holds the other party's three-dimensional model in advance, and when the voice is output, the three-dimensional model is transformed and displayed based on the received parameters.

All of the above three techniques are intended to talk while looking at the face of the other party of communication, and not to enhance the entertainment of the conversation itself. The above is related to so-called telephone technology, and although the character-based conversation is mainly due to the spread of the Internet, the conversation by a personal computer is also possible. In such a situation, my proxy CG (Com
There is also a technique in which a putter Graphics) character is allowed to participate and enjoy a conversation with a CG character acting on behalf of another person who participates in the character via a CG character acting on behalf of the character (see Patent Document 3). The object of the present invention is to talk with the other party in a state of having anonymity and to participate in a state of being separated from the actual one, so that the user often enjoys fictitious conversation including falsehood.

Since the proxy CG character works as a virtual body selected by the person who communicates, the avatar (A
vatar). This avatar is selected by the participant himself, and the conversation partner cannot change the character of the avatar. Further, since this avatar itself is for the other participants only to identify the other party, there is no need to change it. In terms of feasibility, the participant's terminal computer (client computer)
In addition to the above, a server computer for performing management control of a common virtual space inviting participants, status control of participants, and notification of information about them is required.

A technique for having a conversation with a virtual CG character is also disclosed on the website of Extempo Systems on the Internet, for example. Here, the user has a conversation with a specialized character on the Internet, but this is a character-based conversation, not a voice conversation.

Further, technically, a dictionary in which the conversation dictionary is classified by keywords is prepared in advance, and the suitability of the conversation contents of the other party to the classified keywords is analyzed,
By displaying the most suitable conversation document, the conversation between the CG character and the human being is established. Since human comprehension is high, even a proper document can be formed as a conversation as it is, but since the number of conversation documents is at most registered, the document is gradually repeated while exchanging conversations several times. Although there is a new entertainment feature of talking with a virtual CG character, there is a big difference from the conversation with an actual human being in terms of flexibility, variety, suitability, and individuality. The aim of such technology is how to approach the conversation ability of a real person.

[0009]

[Patent Document 1] Japanese Unexamined Patent Publication No. 62-274962

[0010]

[Patent Document 2] Japanese Patent Laid-Open No. 05-153581

[0011]

[Patent Document 3] US Pat. No. 5,880,731

[0012]

[Non-Patent Document 1] NIKKEI ELECTRONICS 1999.11.1 (No.75
6), pp99-117

[0013]

Summarizing the features of the above-mentioned conventional techniques, all of the above three techniques were invented in response to a request to have a conversation while looking at the face of a communication partner. The purpose is to have a conversation while checking the facial expressions and appearance of the communication partner. Therefore, it is not intended for the listener to perform some action by himself or herself to process the display video and audio to enhance the entertainment, and the technical disclosure thereof is not made.

Next, in the fourth conventional example, the CG character selected by the user is allowed to participate in the virtual community space, and has anonymity, and conversely, since it has anonymity, it is not a problem. It is a place to enjoy true conversation, fictitious conversation, and false conversation. Therefore, the CG character of the conversation partner is only required to be identified, and the CG character or voice of the conversation partner is not allowed to have any action to enjoy the conversation with enhanced entertainment. The fifth conventional example has the aspect of enjoying a conversation with a virtual CG character having an artificial intelligence conversation function, but it is not similar to a conversation with a real person, Conversations with humans differ greatly in terms of flexibility, diversity, suitability, and individuality.

In view of the above problems, the present invention displays the communication partner as a virtual three-dimensional CG character selected by the listener and uses the conversation of the communication partner to create a virtual three-dimensional character.
The purpose of the present invention is to provide a communication terminal with a display function that allows a voice conversation with a three-dimensional CG character. This allows
Realization of a new communication terminal that enhances the entertainment of voice conversations by a method different from the functions of "you can see the face of the communication partner or a video similar to it" and "impersonate a fictional character" Is possible.

Another object of the present invention is to provide a communication device with a display device for realizing a conversation in a virtual space without using a device such as a server as in the above-mentioned prior art. further,
It is also an object of the present invention to provide a new call device in which a three-dimensional CG character expresses emotions suitable for a conversation during a call.

[0017]

In order to achieve the above object, a virtual video call device of the present invention selects a communication means for performing voice communication and selection of CG character shape data of at least one of a user himself or a communication partner. Character selection means, a voice input means for inputting the voice of the user himself, a voice output means for outputting the voice of the communication partner, voice data of the communication partner received by the communication means, or the received communication A voice analysis unit for performing voice analysis on both voice data of the other party and voice data of the user himself / herself input by the voice input unit, and a communication partner or a communication partner by using the voice analysis result of the voice analysis unit. Emotion estimation means for estimating the emotional state of the user himself, movement control means for controlling the movement of the CG character based on the emotion estimation means, and the CG character The image forming apparatus includes a drawing unit that performs drawing processing using the shape data and the motion data generated based on the control information of the motion control unit to generate an image, and a display unit that displays the image generated by the drawing unit. It is characterized by

Further, in the virtual video call device according to the present invention, the emotion estimation means notifies the operation control means of the estimation result of the emotion estimation means, and the operation control means
The operation data is specified based on the notification result.

The present invention can be realized not only as the above-described virtual TV call device, but also as a virtual TV communication method using the steps of the virtual TV call device as steps, or using the steps. It can also be realized as a television communication system.

Further, it goes without saying that the virtual television communication method can be realized as a program realized by a computer or the like, or the program can be distributed via a recording medium such as a CD-ROM or a transmission medium such as a communication network. Yes.

[0021]

BEST MODE FOR CARRYING OUT THE INVENTION (First Embodiment) A virtual video call device according to a first embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 shows the configuration of a virtual video call apparatus according to the first embodiment of the present invention. This virtual video call device includes a communication unit 1, a character background selection input unit 2, a data management unit 3, a voice selection input unit 4, a voice processing unit 5, a voice conversion unit 6, a voice output unit 7, a voice input unit 8, a voice. Analysis unit 9, emotion estimation unit 10, lip motion control unit 1
1, body motion control unit 12, facial expression control unit 13, three-dimensional drawing unit 14, display unit 15, motion facial expression input unit 16, viewpoint change input unit 17, character shape data storage unit 18, character motion data storage unit 19, background. The data storage unit 20, the texture data storage unit 21, and the music data storage unit 22 are included.

The virtual video call apparatus according to the first embodiment of the present invention configured as described above will be described in detail below. In the first embodiment of the present invention, since the operation at the time of setting and the operation at the time of sending / receiving conversation can be divided, each will be described in order. However, the common point to them is that the operation is first saved in the device. Explain the data and its management.

(Explanation of Saved Data and Management) The character shape data storage unit 18 manages and saves the shape data of the CG character and the corresponding thumbnail data (image data showing the appearance of the CG character) by address. Character shape data is generally composed of parts such as the head, upper limbs, trunk, and lower limbs, and each part further includes sub-parts such as eyes, nose, mouth, and hair for the head, and hands for the upper limbs, It consists of sub-parts such as the forearm and upper arm. If the character shape is more detailed, the hand is made up of sub-parts such as fingers and palm. These hierarchical relationships
It shows a character-shaped structure, and is generally called a scene graph.

Each of the parts and sub-parts is usually expressed as a surface model by approximating only the surface of the object, which is called a surface model, and is represented by a set of surfaces. The vertex coordinates in the three-dimensional space and the normal vector component at the same vertex. (Required for light source brightness calculation),
Indexed point sequence data of texture coordinates (required when performing texture mapping) and topological data indicating how they are connected (for example, if vertex indices are written in the order of 1, 2, 3 then point 1 2,
3 represents the triangle having the apex), and further includes attribute data such as reflectance (diffuse reflectance, specular reflectance) of each surface, ambient light intensity, and object color. When a CG character wears clothing or the like by texture mapping, the corresponding part of the shape data of the CG character has an address of the texture to be used in the texture data storage unit 21 or an ID of an identifier corresponding to the address. Is clearly stated.

The character motion data storage unit 19 stores CG
The body motion data of the character, body motion pattern data that is transition graph data of body motion, facial expression data and facial expression pattern data, lip motion data and lip motion pattern data are managed and stored by addresses.

The body movement data includes the amount of parallel movement representing the movement of the whole body of the route, which is the representative point of the body in the three-dimensional space, and the posture state of the whole body, as is done in normal CG character animation. Amounts of rotation around three coordinate axes in the three-dimensional space that represents the vector, or amounts of rotation around a vector of components of a vector that represents the center axis of rotation, and amounts of rotation around the coordinate axes of the local coordinate system defined by each joint. Is the time series data of. By converting the root position and the joint coordinate system in the local coordinate system, the CG character shape data is converted to generate the position and orientation of the CG character at each time, the pose of the CG character's body, and three-dimensional drawing. A CG animation can be realized by processing and performing this continuously in time. When using the key frame animation technology, the body motion data does not have for all frames, it is time-series data that is discontinuous in time, and the motion state at the time during that time is calculated by interpolation calculation. The above-mentioned time-series data of the amount of parallel movement and the amount of angle, which are discontinuous in time, are body movement data.

The physical motion pattern data is graph data in a finite state as shown in FIG. 6B, and the relationship between motions that can be transferred from a certain motion and substance motion information (motion ID, data type, each substance). It is data consisting of addresses of body movements, the number of frames, and transition probability of each transition). For example, in FIG.
In (b), the motion is calculated from the physical motion data representing the standard state.
It can be seen that the transition to A, operation C, operation D, and operation E is possible. When a predetermined event occurs while the CG character is in the standard state, by the selection process based on the transition probability described in the substantive motion information, motion A, motion C,
The action is selected from the action D and the action E, and the substance of the action is acquired by the address.

Further, in the case of the present embodiment, the body movement pattern data after the start of the call includes the estimation result of the normal state, the laughing state, the crying state, the angry state, the worried state, the convincing state, etc. Although it is described that a transition occurs as an event based on the input result of the motion facial expression input unit 16, the present invention can be similarly performed even when the transition occurs due to a more complicated estimation result or an event by another input unit.

The body motion depends on the structure of the shape data (skeletal structure, hierarchical structure) (for example, the motion of an insect with 6 legs cannot be applied to a human with 2 legs), and all body motions are Since it is not applicable to the shape data of No. 3, they are classified as applicable shape data according to the data type of the substantive motion information. Further, regarding the above-mentioned physical motion pattern data, the physical motion pattern data is newly provided in the upper layer, and the addresses of the entities of the plurality of physical motion pattern data are managed by this physical motion pattern data of the upper layer. It is also possible that there is only one body movement pattern of the layers. For example, it is very effective to switch and use a body movement pattern like a scene change.

The facial expression motion data is as shown in FIG.
This is data for generating facial expressions of CG characters. The facial expression is generated by using a facial animation technique that is usually used, but for example, there are a method of deforming the shape of the face and a method of pasting the texture of the face. When the face shape is deformed, the facial movement data is the time series data of the movement amount of the vertex coordinates corresponding to the end points of the eyebrows, the eyes, the mouth, etc., which generate the facial expression, among the facial shape data. These movement amounts can also be calculated by simulation calculation based on the facial muscle model. If the vertices to be converted span multiple conversion systems, give weighting to each conversion to the vertices,
An envelope method is also used in which a plurality of vertices whose vertices are once transformed are calculated in each transformation system and the vertices are transformed into averaged coordinates in consideration of weighting.

In FIG. 6A, each emotion is expressed by deforming the shape of eyes, the size of nose, the shape of ears, the shape of faces, and the like. When the textures are pasted together, the laughing expression, the crying expression, and the intermediate texture are the expression data. Facial expression pattern data is transition graph data of these facial expression data. Similar to the transition graph data of body movement data, a finite state graph that allows transition from certain facial expression data to certain facial expression data and actual facial expression information (facial expression ID, data) The type, the address and the number of frames of each actual facial expression motion data, and the transition probability of each transition). For example, as shown in FIG. 6- (a), this example shows that the face cannot be transferred to another face without passing through the normal face, and the selection of the transfer destination is based on the transfer probability of the actual facial expression information. Is done. Similar to the case of the physical movement, the applicable shape is identified depending on the data type of the actual facial expression information, whether it is a facial movement or a texture. For example, the first digit of the data type is used for classification of facial expression or texture, and the second digit or more is used as the shape identification number. Note that the facial expression pattern data described above may have a plurality of facial expression patterns by providing upper-layer facial expression pattern data, as in the case of the body movement pattern data.

In the case of the present embodiment, the emotion estimation unit 10 estimates the normal state, the laughing state, the crying state, the angry state, and the worried state for the facial expression pattern data after a call, and the estimation result and the action facial expression. Although the description will be made assuming that the transition occurs as an event based on the input result of the input unit 16, the present invention can be similarly implemented even when the transition occurs due to a more complicated estimation result or an event by another input unit.

The lip motion data is similar to the facial expression motion data and the facial expression motion pattern data, and there are a method of deforming the shape of the mouth and a method of pasting the texture.
However, the lip motion data depends on the content of the voice analysis process, and if the lip motion is generated based on the sound intensity analysis result as described later, the motion data corresponding to the amount of opening of the mouth is simply generated. It has been saved (see FIG. 5A). When processing up to phoneme analysis is possible, for example, when vowel analysis and “n” sound analysis can be performed, shape deformation data and texture data of the lip to generate a lip shape matching the sound operate. It is stored as data (see FIG. 5B).

The lip pattern data represents a set of several types of lip motion data as described above, and is data including real lip information (each lip ID, data type, each real lip movement address and the number of frames). is there. Each real lip ID is
For example, in the case of controlling by sound intensity as shown in FIG. 5A, the identifier corresponding to the level is used, and 0 is given as level 0, ..., 3 as level 3. If the identifier is based on the phoneme analysis as shown in FIG. 5B, the identifiers corresponding to “n”, “a”, ..., “O” are given as 0, 1, ..., 5 respectively. It is an identifier. Furthermore, it is possible to combine sound intensity analysis and phoneme analysis, and even for the same "a" sound, a large "a" and a small "a" are provided. In this case, the vertical direction of FIG.
The levels in (a) are arranged side by side, and the lip ID may be defined as a two-dimensional identifier.

The background data storage unit 20 manages the address of the background shape data or the background image and the thumbnail image corresponding to the background shape data as the background data when the CG character is displayed, and stores it. The background shape data is the same as the shape data of the CG character, and is an object that serves as a background as a shape. The background image data is, for example, image data of a sky or a distant view, and can be used in combination with a background object. When adding a pattern or the like to the shape data of the background object by texture mapping, the address in the texture data storage unit 21 to be used or the ID of the identifier corresponding thereto is specified.

The texture data storage unit 21 is a CG used for texture mapping in the three-dimensional drawing unit 14.
Image data of textures such as clothes worn by the character and image data for texture mapping of objects used in the background are address-managed and stored.

The music data storage unit 22 stores the music data by address management. This is used as a signal by ringing when an incoming call is received from the other party. The data management unit 3 manages stored data, manages stored setting data, and notifies setting data. Here, first, the character shape data storage unit 18, the character motion data storage unit 19, the background data storage unit 20,
The management of the data stored in the texture data storage unit 21 and the music data storage unit 22 will be described.

FIG. 3 is one of the tables held by the data management unit 3 and shows the CG character data management table 3a. The CG character data is based on the user's designation with respect to the name of the CG character, the address in the character shape data storage unit 18 having the substance of the CG character shape data, and the texture of clothing etc. specified in the CG character shape data. The address of the clothing texture before the exchange and the address of the clothing texture data after the exchange (a plurality of addresses can be described) in the texture data storage unit 21 of the clothing texture data when performing the exchange, and the facial expression pattern stored in the character motion data storage unit 19. It consists of two addresses before and after the call of data, the address of the lip movement pattern, and the address of the thumbnail image stored in the character shape data storage unit 18, and these are tabulated by the CG character ID identifier. Is the CG character data management table 3a.

As other stored data management tables, there are three types of background data management table, motion pattern management table, and voice management table, and there are a total of four types including the CG character data management table 3a. In the background data management table, the name of the background object or the image data of the distant view and the address in the background data storage unit 20 are set as the background ID.
It is made into a table with an identifier by. The motion pattern management table contains the names of the physical motion pattern data,
The addresses in the character motion data storage unit 19 are tabulated by the motion pattern ID. The music data management table is a table in which the names of music data and the addresses in the music data storage unit 22 are tabulated by the identifiers of music IDs.

(Operation at the time of setting)
As shown in (a), the correspondent management table 1a is stored. The correspondent management table 1a manages the contents of the sender ID, telephone number, name, and display mode of the communication partner. The display mode is a non-display mode when talking in normal voice communication without displaying the CG character, a display mode in which only the communication partner is displayed as the CG character and the call is performed as virtual TV communication, not only the other party There is a simultaneous display mode in which the user himself / herself also displays it as a CG character and makes a call as virtual television communication, and this is managed by an identifier. In this embodiment,
The description will be given assuming that the non-display mode is 0, the partner display mode is 1, the principal simultaneous display mode is 2, and the identifiers are assigned.

It is assumed that the sender ID number 0 is predetermined as an identification of the person. In this embodiment, since telephone communication is basically considered, it is assumed that transmission / reception is managed by telephone number.For example, in the case of the Internet, IP based on TCP / IP
It may be an address or an email address for the user. Since these are identifiers that are determined depending on the communication infrastructure and are used to identify a correspondent, any identifier that satisfies such conditions can be used.

CG data management table 3b of FIG. 4 (a)
Is a table stored in the data management unit 3 for storing and managing CG data settings for a communication partner. The CG character ID in the CG character data management table 3a, the background ID in the background data management table, and the physical motion pattern IDs before and after the call start in the motion pattern management table, which are determined for the sender.
Items consisting of are managed by sender ID.

The voice management table 3c of FIG. 4 (a) is also a table stored in the data management section 3, and is a voice conversion numerical parameter for the communication partner determined for the sender,
The voice conversion numerical parameter for managing the item consisting of the music data ID at the time of incoming call by the sender ID is
It is used by the voice conversion unit 6, and is an identifier assigned to each bandpass filter when voice conversion is performed by a bandpass filter. For example, 0 is no filter,
1 is a filter of 1 kHz or less, 2 is a filter of 1 to 5 kHz,
3 assigns an identifier such as a filter of 5 kHz or more.

As described above, since the parameters required for conversion are identified, the conversion method (for example, even when voice conversion is performed by pitch conversion, the set of parameters required for conversion may be identified). Does not depend on.
The voice conversion numerical parameter is an identifier for determining the level of voice, and has an effect like a voice changer when the user changes the setting. The music data ID serves as an identifier for determining a so-called incoming melody.

The operation at the time of setting will be described with reference to FIG. When the user operates the setting state transition input section provided in the character background selection input section 2, the data management section 3 is notified that the setting state transition input section is entered, and the data management section 3 is saved in the communication section 1. The contents of the correspondent management table 1a are read and sent to the three-dimensional drawing unit 14 (S
401). The three-dimensional drawing unit 14 sends the correspondent management table 1a based on the setting screen data stored in advance.
A setting screen reflecting the contents of is generated and the setting screen is displayed on the display unit 15. The character background selection input unit 2 selects a correspondent (S402) and inputs the display mode according to the above-mentioned identifier for the correspondent. If the selection is 0 indicating the non-display mode (S403), the setting ends.

Next, when the display mode is the display mode 1 in which only the other party is displayed as a CG character or the display mode 2 in which the user himself is also displayed as a CG character, the result is the communication unit via the data management unit 3. 1 and 3D drawing unit 1
4 will be notified. In the communication unit 1, the correspondent management table 1
The display mode of the selection result is entered and saved in a. The three-dimensional drawing unit 14 sequentially generates a predetermined character selection setting screen as shown in FIG. 3, a clothing texture setting screen, and a body motion pattern setting screen and displays them on the display unit 15.

On the character selection screen, the image and name of the CG character are drawn as shown in FIG. 3 based on the thumbnail address and name shown in the CG character data management table 3a. Then, a CG character selection setting screen, a clothing texture setting screen, and a body movement pattern setting screen are sequentially displayed. Among them, the CG character selection setting screen for the default or specific correspondent selected and input by the character background selection input unit 2 is displayed. The selection result on the physical movement pattern setting screen is the data management unit 3
The CG data management table 3b stored in
The ID is recorded. The selection result on the clothing texture setting screen is recorded in the corresponding column of the CG character management table 3a stored in the data management unit 3.

For selecting the physical movement pattern, two types, one before the call start and one after the call start, are selected. At that time, the name described in the motion pattern management table is displayed on the setting screen. Is also possible. With this display, the user can easily grasp the image of the body motion, and thus can easily select the image. For example, mambo dance, waltz dance, announcer movement, famous talent movement, etc. (S404).

Similarly, the voice selection input unit 4 inputs and sets voice conversion parameters and music data. The transition to the input mode is made by the preset state transition input unit preset in the voice selection input unit 4. When the user operates, the transition is made through the communication unit 1 and the data management unit 3
The dimension drawing unit 14 is notified. The three-dimensional drawing unit 14 generates a predetermined setting screen and displays it on the display unit 15.
Based on the displayed setting screen, the user selects and inputs a voice conversion parameter and music data using the voice selection input unit 4. The input selection result is recorded in the voice management table 3c stored in the data management unit 3 (S40).
4).

In the case of the partner display mode, the process proceeds to the background selection setting (S405). When the person simultaneous display mode is selected, the selection of the CG character, clothing texture, and motion pattern for the user is performed by the character background selection input unit 2 in the same manner as above (S406), and then the background is selected. Move to.

Regarding the selection of the background, a predetermined background setting screen is displayed, and the background is selected by the character background selection input unit 2 (S407). The selection result is stored in the CG data management table 3b stored in the data management unit 3.

Finally, at the time of setting the CG character and the physical motion pattern, the address of the specific facial motion data in the facial pattern data and the address of the specific physical motion data in the physical motion pattern data are set. The motion facial expression input unit 16 is notified. The motion facial expression input unit 16 holds the notified address of the physical motion data and the notified address of the facial motion data, and associates it with the input button prepared in advance in the motion facial expression input unit 16. When the user presses the input button, the address of the body motion data or facial expression data corresponding to the input button is notified to the data management unit 3, and if the notification result is the address of the body motion data, the body motion control unit 12 If it is the address of the facial expression motion data, the facial expression control unit 13 is notified. By preparing a plurality of input buttons, a plurality of addresses of body motion data and facial motion data that can be held can be held.

Further, the addresses of the body motion data and the addresses of the facial motion data before and after the start of the call are clearly known. In the present embodiment, the button input is described, but any input unit (for example, keyboard, mouse, etc.) that can be specified may be used. Therefore, the user can select his own character and
You can freely select the character of the other party,
Further, since the communication device on the user side has data necessary for the virtual video call, the user can make a virtual video call even if the other party does not necessarily use the virtual communication device.

The above-described graphical setting is generally performed on a PC and can be realized by existing software technology. (Operation at the time of sending and receiving) At the time of sending, by inputting the telephone number in the communication unit 1 and checking with the contents of the telephone number column recorded in the saved correspondent management table 1a, the sender's ID and display Specify the mode. When a call arrives, the telephone number of the called party is usually notified before the call. Therefore, by collating the telephone number with the telephone number column of the correspondent management table 1a,
Specify the sender ID and display mode. Note that the communication unit 1 has a normal voice communication function (so-called baseband processing in the case of a mobile phone).

When the specified display mode is the non-display mode, the voice call processing generally performed is performed. That is,
When the voice data is sent from the sender after the approval of the communication with the communication partner is obtained, the voice processing unit 5 performs a normal voice process such as a decoding process and passes the voice conversion unit 6. Then, the audio is output to the audio output unit 7, and the audio is output. In addition, the voice of the user himself is input from the voice input unit 8, the voice processing unit 5 performs voice processing such as compression of voice data that is normally performed, and transmits the voice data to the communication partner via the communication unit 1. .

The operation will be described for the case where the specified display mode is the other party display mode in which only the other party is displayed as a CG character. The operation is divided before and after the call is started, and the call is started by the communication unit 1. Notify the data management unit 3 of the start.

Since the telephone number of the transmission partner can be specified as described above before the call is started at the time of transmission / reception, the sender ID of the communication partner is specified by the communication unit 1 from the communication party management table 1a, Send to the data management unit 3. Data management unit 3
Is the sender from the saved CG data management table 3b.
CG character ID corresponding to ID, background ID, motion pattern ID
Identify (two IDs of the physical movement pattern before and after the call). If the one corresponding to the sent sender ID is not in the CG data management table 3b, it is set as default.
A CG character ID, a background ID, and a motion pattern ID (two IDs of a physical motion pattern before and after a call) are specified.

In the data management unit 3, the CG character data management table 3a is used for the specified CG character ID,
The address of the CG character shape data, the address of the clothing texture before the exchange and the address of the clothing texture after the exchange, the addresses of the two facial expression pattern data before and after the call start, and the address of the lip movement pattern are specified. The background data management table is stored to specify the address of the background data from the specified background ID.
Further, the stored motion pattern management table identifies the addresses of the two physical motion patterns before and after the call start from the motion pattern IDs (two IDs of the physical motion patterns before and after the call).

The data management unit 3 notifies the specified three-dimensional drawing unit 14 of the address of the CG character shape data, the address of the clothing texture before the exchange, the address of the clothing texture after the exchange, and the address of the background data. In addition, the data management unit 3 stores, from the character motion data storage unit 19, addresses of two specified physical motion patterns before and after the start of the call, addresses of two facial expression pattern data before and after the start of the call, Based on the address of the lip motion pattern data, two physical motion pattern data before the start of the call and after the start of the call are read and the physical motion control unit 1
2, the two facial expression pattern data before and after the start of the call are read and sent to the facial expression control unit 13, and the lip movement pattern data is read and sent to the lip movement control unit 11.

The lip motion control unit 11 selects an appropriate lip motion data address from the lip motion pattern data and sequentially notifies the three-dimensional drawing unit 14 of the address from the frame number 0 to the number of frames. As a method of selecting an appropriate lip motion data address from the lip motion pattern data, there is a method of using a random number. In addition to selection with equal probability, lip weighting is performed to perform selection control. This process is repeated until the call starts. It is also possible to predefine a fixed transition without using random numbers and notify the 3D drawing unit 14 of the address and frame number of the lip motion data according to the sequence of the transition. However, in this case, the user will see regular repeated lip movements. For example, it is possible to repeatedly display the lip movement in accordance with the word "I'm on the phone".

The physical motion control section 12 first sets the address and frame number of the physical motion data corresponding to the standard state to 0 from the physical motion pattern data before the start of the call, as shown in FIG. 6B. The three-dimensional drawing unit 14 is sequentially notified of the number of frames. After the notification of the number of frames, a random number based on the transition probability of each transition is generated to select the next physical action data, and the address and frame number of the physical action data of the transition destination are set from 0 to the number of frames for the three-dimensional drawing unit 14. To notify. After the end, a transition is performed again by generating a random number based on each transition probability. This process is repeated until the call starts.

Note that fixed transitions may be defined in advance in the body movement pattern without using random numbers, and the address and frame number of the body movement data may be notified to the three-dimensional drawing unit 14 according to the sequence of the transitions. it can. However, in this case, the user will see regular repeated physical movements. For example, a physical action such as "pick up the telephone receiver" can be repeatedly displayed.

First, the facial expression control unit 13 sets the address and the frame number of the facial expression action data corresponding to the normal face from 0 from the facial action pattern data before the start of the call, as shown in FIG. 6A. Three-dimensional drawing unit 14 for the number of frames in sequence
To notify. After the notification of the number of frames, a random number based on the transition probability of each transition is generated to select the next facial expression motion data, and the address and frame number of the facial expression motion data of the transition destination are set from 0 to the number of frames for the three-dimensional drawing unit 14. To notify. After the end, a transition is performed again by generating a random number based on each transition probability. This process is repeated until the call starts.

Note that it is also possible to predefine a fixed transition in the facial expression action pattern without using random numbers, and notify the three-dimensional drawing unit 14 of the address and frame number of the facial action data in accordance with the sequence of the transition. it can. However, in this case, the user will see a regular repeated facial expression motion. For example, facial expressions such as “normal face and troubled face” can be repeatedly displayed.

The basic operation of three-dimensional drawing in the three-dimensional drawing unit 14 will be described. The three-dimensional drawing unit 14 uses the address of the CG character shape data, the address of the clothing texture before the exchange and the address of the clothing texture after the exchange, and the address of the background data notified from the data management unit 3 to store the character shape data storage unit. Draw from 18
CG character shape data, texture data storage unit 21 to clothing texture data, background data storage unit 20
First, load the background data from.

Next, the lip motion data address and frame number notified by the lip motion control unit 11, the physical motion data address and frame number notified by the physical motion control unit 12, and the facial expression control unit 13 are notified. Receive the address and frame number of the facial expression motion data. The lip movement data, the body movement data, and the facial movement data are loaded from the character movement data storage unit according to the received addresses of the lip movement data, the addresses of the body movement data, and the addresses of the facial movement data. This load is performed by the lip motion control unit 11, the body motion control unit 12, and the facial expression control unit 13.
Unless the address of each operation notified from is updated,
Only once at the beginning of the notification. Since a character corresponding to a specific communication partner is displayed on the screen when an incoming call is received, the user can understand who the incoming call is from just looking at the character displayed on the screen.

The motion data of the frame number notified from the lip motion control unit 11 is generated from the loaded lip motion data. This is a key frame that is usually used when the lip motion data is a shape deformation. Similar to the animation technique, it is generated by interpolation of key motion data, and even in the case of texture, it is generated by interpolation of key texture. In the case of shape transformation, the shape of the mouth of the CG character shape data is transformed using the generated motion data of the frame number. In the case of a texture, a texture mapping technique that is usually used is used to perform mapping on the mouth portion, which is performed at the time of three-dimensional drawing processing.

Similarly, in the case of facial expression motion data, the motion data of the notified frame number is generated, and in the case of shape deformation, the face is deformed based on the motion data. In the case of a texture, the face part is drawn by texture mapping, which is performed during the three-dimensional drawing process. In addition, the motion data of the body motion data of the notified frame number,
It is generated by interpolation of the physical motion data that is a key, and based on the physical motion data, the aforementioned conversion is applied to the CG character to determine the position and physical condition of the CG character.

After that, if the background data, the clothing texture data, and the lip movement data are textures, the textures are used. If the facial expression movement data are textures, the textures are used to perform the three-dimensional drawing processing (modeling conversion, visual field conversion) normally performed. Conversion, perspective conversion, screen conversion, and pixel processing to the screen are performed in this order, but texture mapping is performed at the time of pixel processing to the screen) to generate an image. At that time, camera data (required for view conversion and screen conversion depending on the position and direction of the camera and angle of view)
First uses the default one. For example, by setting the whole body facing the front of the CG character so as to be in the center of the generated image, such setting finds the smallest rectangular parallelepiped including the CG character, and determines the direction vector of the root of the CG character. The angle of view may be set so that the optical axis is in the direction opposite to the direction corresponding to the front part of the optical axis, its center of gravity is on the optical axis, and each vertex is included in the screen.

Further, by inputting camera data from the viewpoint changing input unit 17 and notifying the 3D drawing unit 14 and performing 3D drawing processing based on this camera data, an image whose viewpoint has been changed is displayed. Can be generated. In addition, camera data preset in the viewpoint change input unit 17 is prepared,
The preset data is notified to the three-dimensional drawing unit 14 to change the viewpoint.

When the user presses the input button which is set in advance as described above, the action facial expression input unit 16 sends the address of the physical action data or the address of the facial action data via the data management unit 3. The address of the body motion data is notified to the body motion control unit 12, and the address of the facial motion data is notified to the facial expression control unit 13. In the case of the address of the body movement data, when the body movement control unit 12 receives this notification, when the notification of the last frame number of the body movement data currently notified to the three-dimensional drawing unit 14 ends, the next unit is normally set. The body movement data of the transition destination is selected as described above, but the address and frame number of the body movement data forcibly notified are notified to the three-dimensional drawing unit 14. Similarly, in the case of the address of the facial expression motion data, the facial expression control unit 13 compulsorily determines the address and the frame number of the facial expression motion data notified from the data management unit 3 after the notification of the facial expression motion data currently notified is completed. Notify the three-dimensional drawing unit 14. This allows the user to forcibly display the action selected by the user, which is automatically selected and normally used for animation.

The image generated as described above after the three-dimensional drawing is completed is transferred to the display unit 15 and displayed. It should be noted that the three-dimensional drawing process in the three-dimensional drawing unit 14 is usually performed according to the refresh rate of the display unit 15. The address and frame number of the motion notified from the lip motion control unit 11, the body motion control unit 12, and the facial expression control unit 13 are notified during the 3D drawing process of the 3D drawing unit 14,
It is set as the data to be used next. When performing the three-dimensional drawing process of the next frame, the address and frame number of each set operation data are used. From the above, the lip motion control unit 11, the body motion control unit 12,
Synchronous control is performed regarding the notification from the facial expression control unit 13.

The music data will be described. The data management unit 3 specifies the value of the voice conversion numerical parameter corresponding to the sender ID and the music data ID by the voice management table 3c. When the one corresponding to the sent sender ID is not in the voice management table 3c, the voice conversion numerical parameter and the music data ID set as default are specified.
The address of the music data is acquired from the music data management table by the music ID. The corresponding music data is loaded from the music data storage unit 22 according to the obtained address of the music data and transferred to the voice processing unit 5.

The voice processing unit 5 performs a decompression process for compressed music data and a sound generation process for stored sound source data in the case of encoded music data such as MIDI data. Then, the audio output unit 7 outputs music through the audio conversion unit 6. In this way, when an incoming call arrives, the voice output unit 7 outputs the incoming melody linked to the character of the communication partner, so that the communication partner can be easily confirmed.

By the above operation, while the music is playing,
It is possible to display CG characters, but music and CG
The movements of the character are basically not synchronized (if the movement data is created in advance so as to be synchronized with the music data, the movement can be synchronized, so at least the first output can be synchronized).

Here, the synchronization of music and CG characters will be described. Music data including time management data corresponding to a time stamp used for video data or the like is used. This is MPEG-4 (Moving Picture Experts Group
Phase 4) has a time stamp in the audio, and if it is MIDI data, it is called Delta Time, which can be substituted by integrating the time increment data. The voice processing unit 5 manages the time stamp when transferring the music data to the voice output unit 7, and uses the type stamp of what is to be used for music output as a time synchronization signal, the lip motion control unit 11, the body motion control unit 12 , To the facial expression control unit 13. The lip movement data, facial expression movement data, and body movement data with time stamps starting from 0 are also used. For this, a time stamp is assigned in advance according to the music.

Lip motion controller 11, body motion controller 1
2. The facial expression control unit 13 compares the sent time stamp with the time stamp number of the motion data controlled by each, and at that time, the time stamp of the motion data that has been three-dimensionally rendered so far. It is used that when the cumulative number of is added to the time stamp of each operation, it matches the music time stamp. The frame number and the address of the motion data that match this collation are given to the three-dimensional drawing unit 14
Send to at the same time. With the above processing, operation control in synchronization with music data can be performed.

Next, the operation after the call is started will be described. The communication unit 1 determines that the call with the communication partner has been started. For normal telephone communication, if you make a call from yourself, when the other party picks up the receiver, you will receive an accept signal, and when you are called from the other party, The communication establishment can be recognized by returning an accept signal to the other party by taking it. The basic mechanism is the same for wireless communication such as mobile phones and communication over the Internet, and it is possible to recognize the establishment of communication start. The communication unit 1 notifies the data management unit 3 that the call has been established.

When the data management unit 3 receives the notification of the establishment of the call, it stops the transfer of the music data to the voice processing unit 5 and notifies that the call is started. The voice conversion numerical parameter is read from 3c and notified to the voice conversion unit 6 via the voice processing unit 5. In parallel, the lip movement control unit 11, the body movement control unit 12, and the facial expression control unit 13 are notified that the call is started.

Lip motion controller 11, body motion controller 1
2. Upon receiving the notification, the facial expression control unit 13 stops the transfer to the three-dimensional drawing unit 14. When the voice analysis unit 9 described later performs only the sound intensity analysis process, the lip motion control unit 11 only addresses and frame numbers of the lip motion data in the level 0 state shown in FIG. When both the sound intensity analysis and the phoneme analysis are performed, the address and the frame number of the lip motion data of the “n” sound shown in FIG. 5B are sent to the three-dimensional drawing unit 14.

The body movement control unit 12 sends the address and frame number of the body movement data in the standard state of the body movement pattern data after the start of the call to the three-dimensional drawing unit 14. The facial expression control unit 13 sends the address and frame number of the facial expression action data of the normal face of the facial action pattern data after the start of the call to the three-dimensional drawing unit 14. Upon receiving the address and frame number of the motion data sent from the lip motion control unit 11, the body motion control unit 12, and the facial expression control unit 13, the three-dimensional drawing unit 14 performs the three-dimensional processing by the same motion as described above. Then, the generated image is sent to the display unit 15 and displayed.

When the voice processing unit 5 receives the notification of the start of the call, the voice processing unit 5 performs voice processing of the voice data (decoding of voice data, noise cancellation, etc.) according to the communication medium sent from the communication unit 1, and the voice processing is performed. Data to voice converter 6
To the voice analysis unit 9.

The voice conversion unit 6 converts the voice based on the voice variation or the numerical parameter that has been sent (for example, when the filter processing is performed as described above, applies the corresponding filter), and sends the voice output unit 7. . Therefore, the voice of the caller is converted into another voice and output.

The voice analysis unit 9 performs sound intensity analysis, phoneme analysis, or both analysis on the sent voice data. In the sound intensity analysis, as shown in FIG. 5A, the absolute value of the amplitude of the audio data is integrated (addition of sampling values) with respect to a predetermined period (for example, display rate time), The level value of the integrated value is determined according to a predetermined segment value.

In the phoneme analysis, a process usually performed by speech recognition is performed so that each phoneme is “n”, “a”, ..., “O”.
Any of the above is classified, or the ratio thereof is output.
Basically, statistically collected "n", "a", "i", ...
..A template is created by normalizing the voice data of the "O" sound and template matching is performed with the input voice data that is phoneme decomposed and normalized, and the one with the highest degree of matching is selected. Output the matching rate. As for the matching degree, an appropriate distance function (Euclidean distance, Hilbert, Mahalanobis) is defined, and the one having the smallest distance measured by the distance function is selected, or “n”, “a”, ...・, Calculate the value obtained by dividing each distance by the sum of all measured distances of "O" sound as a ratio. The above voice analysis result is the emotion estimation unit 1.
Sent to 0. In addition, the lip ID is determined from the voice analysis result as described above, and the determined lip ID is used as the lip motion control unit 1.
Send to 1.

The lip movement control section 11 determines the address of the lip movement data corresponding to the lip movement pattern data from the lip movement pattern data based on the lip ID sent from the voice analysis section 9, and obtains the address and frame number of the lip movement data. It is sent to the three-dimensional drawing unit 14.

The emotion estimation unit 10 stores the voice analysis result sent from the voice analysis unit 9 for a predetermined period of time, and estimates the emotional state of the caller based on the stored result. For example, the emotions to be classified are "normal", "laughter",
It is defined as "anger", "crying", and "worry". Regarding the sound intensity level, it has a level pattern for a certain period as a template of each emotion. For a certain period, for example, 3 for voice analysis
If it is a batch, "normal" if "level 2, level 2, level 2", "laugh" if "level 3, level 2, level 3""anger" if "level 3, level 3, level 3" , "Level 1, Level 2, Level 1" have "crying", and "Level 0, Level 1, Level 0" have "worry" as templates.

On the other hand, the sum of the absolute values of the differences between the level values (Hilbert distance) and the sum of the squares of the differences between the levels (Euclidean distance) are calculated for the three stored voice results. , The closest one is determined as the emotional state at that time. Alternatively, the sum of distances for each emotion is calculated, and the sum of the distances for each emotion is divided by the sum to calculate the emotional state. When the phoneme analysis result is sent, the keyword is held as a dictionary template and template matching with the keyword is performed.

However, in this embodiment, since the phoneme analysis is only the vowel analysis, the following method is used. For example, in the case of anger, angry words such as “angry”, “indignation”, and “hit” are displayed as vowels and expressed as “Ia-i-i-u”, “Ioi-i”, “Au-u”. , Assuming that the fixed period is three times of the voice analysis result, a dictionary of three letters from the beginning of the sequence is created. Similarly, create dictionaries for other emotional states. Of these dictionaries, the same ones will naturally appear, but everyday conversations are analyzed, and the dictionary template is generated in advance by including them in the dictionary of the emotional state with the higher frequency. If the fixed period is three times, there are 216 combinations of vowels, so this dictionary template has 2
The 16 vocabularies are classified according to emotional states.

The stored phoneme analysis results for three times are subjected to template matching with the dictionary template to determine the emotional state. When these are combined with the case of sound intensity and the case of phoneme, when both have the same emotional state, the emotional state is selected, and when different, the emotional state is stochastically selected by a random number. State. The emotional state calculated as described above is sent to the body movement control unit 12 and the facial expression control unit 13.

On the other hand, the conversation generated by the user is input to the voice input unit 8 and the voice data input to the voice processing unit 5 is sent. A microphone is used as the voice input unit 8. The voice processing unit 5 performs noise cancellation processing, echo removal processing, etc., which are usually performed on the input voice data that has been sent, and sends the processed voice data to the voice analysis unit 9. The processed voice data is subjected to processing depending on the communication method, for example, the encoding processing is streamed and packetized, and is transmitted to the communication partner via the communication unit 1.
The voice analysis unit 9 also performs the above-described sound intensity analysis and phoneme analysis on the input voice data that has been sent, and a voice analysis result for the input voice and an identifier indicating that it is the input voice, It is sent to the emotion estimation unit 10.

The emotion estimation unit 10 stores the voice analysis result in the dedicated storage area for the input voice for a certain period as described above, and performs the same emotion estimation process as described above on the storage result. However, the emotion estimation is performed by adding a peculiar state at the time of the listener, for example, a "convincing state". That is, the emotion estimation for the voice data of the transmission partner and the voice data of the user himself / herself may be different. The emotion estimation result is sent to the body motion control unit 12 and the facial expression control unit 13.

As another emotion estimation method, there is a method using a frequency signal of voice data such as prosody, amplitude, stress and the like. FIG. 9 is a flowchart showing a processing procedure of an emotion estimation method using a frequency signal. In this emotion estimation method, the most basic emotion classification is "anger",
The description will be given on the assumption that four types of emotions of “sadness”, “joy”, and “standard” are estimated.

First, the voice of the user himself is input to the voice input unit 8 as voice data and then sent to the voice processing unit 5. On the other hand, the voice of the other party is input to the voice processing unit 5 via the communication unit 1 (S901). The voice processing unit 5
The noise canceling process and the echo removing process that are normally performed are performed on the sent voice data, and the processed voice data is sent to the voice analysis unit 9.

The voice analysis unit 9 extracts a feature amount by processing using a frequency signal of voice data such as prosody, amplitude and stress. The feature amount is based on a fundamental frequency that well reflects differences between emotions, and for example, FOmax (maximum fundamental frequency (FO) [Hz] during utterance) [Hz], Amax.
(Maximum amplitude [Hz] during utterance), T (time length from start to end of utterance [sec]), FOinit (fundamental frequency [Hz] immediately after start of utterance), FOrange (maximum fundamental during utterance) Frequency-minimum fundamental frequency [Hz]) or the like is used. Further, it is possible to add another parameter such as gender difference correction to the feature amount.

As a method of extracting the fundamental frequency in the voice analysis unit 9, a method based on DP matching considering the continuity of the entire utterance is used. To briefly explain this extraction method, the voice data input to the voice input unit 8 is once converted into frequency domain data in the voice analysis unit 9, and then converted into time domain data by a predetermined operation. From this data, select a certain number in descending order of peak value,
The fundamental frequency is extracted by connecting the peaks by a predetermined process (S902).

Next, the emotion estimation unit 10 calculates statistics based on the feature amount extracted by the voice analysis unit 9 (S9).
By doing so, it is estimated to which emotion group each voice data belongs (S904). According to this emotion estimation method, it is possible to estimate the emotion of the speaker with high probability. Next, the emotion estimation unit 10 uses the emotion estimation result as the lip movement control unit 11, the body movement control unit 12, and the facial expression control unit 13.
Send to.

Therefore, the character displayed on the screen of the virtual video call device moves by estimating the emotions of the user and the other party of the call, so that the virtual video call device with higher entertainment can be realized.

Then, the body movement control unit 12 determines (predetermines) the next movement transition as the body movement data corresponding to the sent emotion estimation result, and sends it to the three-dimensional drawing unit 14 at present. When the address and frame number of the body motion data corresponding to the number of frames have been completed, the determined address and frame number of the body motion data are sent to the three-dimensional drawing unit 14. In order to control the transition determination of the body motion data stochastically, the probability of causing or not causing the transition corresponding to the emotion estimation result (because it is a binomial distribution, if one probability is determined, the rest is inevitably determined) is determined. Then, the transition is determined by using a random number according to the distribution. The facial expression control unit 13 also performs the transition determination process by the same process, and calculates the address and frame number of the facial expression motion data by the three-dimensional drawing unit 14.
Send to.

In the three-dimensional drawing unit 14, the lip motion control unit 1
1, the address and frame number of the above-mentioned lip motion data sent from 1, the address and frame number of the physical motion data sent from the physical motion control unit 12, the facial expression control unit 1
An image is generated using the address and frame number of the facial expression motion data sent from 3 by the same process as before the start of the call and sent to the display unit 15. The display unit 15 displays the sent image.

Motion facial expression input unit 16 and viewpoint change input unit 17
When there is an input from, the motion and facial expression according to the input are reflected in the CG character, and the viewpoint is changed, as in the case before the call is started.

Even in the simultaneous display mode, the basic operation is
The operation is the same as the above, except that it is necessary to add the person. That is, the person's own data is added to the data notified from the data management unit 3 before and after the start of the call. In addition, in the lip motion control unit 11, the body motion control unit 12, and the facial expression control unit 13, in addition to the address and frame number of the motion data of the opponent CG character, the address and frame number of the motion data of the CG character of the other party are three-dimensionally set. In the drawing unit 14,
Send with an identifier that indicates the other party and the person.

In the three-dimensional drawing unit 14, based on the identifier, the physical condition, facial expression, lip condition,
The physical condition, facial expression, and lip condition of the CG character of the person himself / herself are determined, the same processing is performed as described above to generate an image, and the generated image is sent to the display unit 15 and displayed. The voice data sent from the voice processing unit 5 is sent to the voice analysis unit 9 with the identifier of the partner or the person being attached. In the voice analysis unit 9,
Although the same processing as described above is performed, the voice analysis result is sent to the lip movement control unit 11 and the emotion estimation unit 10 with an identifier of the partner or the person.

The lip motion control unit 11 determines the address and frame number of the lip motion data from the transition of the lip motion of the partner or the principal or the lip motion pattern according to the identifier of the partner or the principal. The emotion estimation unit 10 performs the same emotion estimation as described above, but performs emotion estimation corresponding to each of the other party and the person, and the result is sent to the body motion control section 12 and the facial expression control section 13 together with the identifier of the other party or the person. send. The body movement control unit 12 determines the transition destination of the body movement of the other party and the transition destination of the person according to the identifier of the other party or the person, and the address and frame number of each body movement data are sent to the three-dimensional drawing unit 14 together with the identifier. send. Similarly, the facial expression control unit 13 sends the address and frame number of each facial expression action data to the three-dimensional drawing unit 14 together with the identifier while determining the transition destination of the facial expression action of the other party and the transition destination of the facial expression action of the person.

The emotion estimation result in the emotion estimation unit 10 is that the conversation is basically alternately performed, so that the emotions of the other party and the person with respect to the conversation content of the other party are estimated, and the estimation result is obtained between the other party and the person. Alternately, it is reflected in the physical and facial expressions of the CG character, and the emotion estimation result of the conversation content of the person who received it next is also reflected in the physical and facial expressions of the CG character of the other person and himself. Will be repeated.

When an input is made in the viewpoint change input section 17, an image in which the viewpoint is changed is generated and displayed on the display section 15 as in the above case. Regarding the motion facial expression input unit 16, in the present embodiment, the motion of the other party and the operation for changing the facial expression have been described. However, when the input buttons for the other party and the person are provided and the input button is pressed, By performing the same processing from the data management unit 3 except that the identifier of the other party or the person is added, both the CG character of the other party and the CG character of the person can be changed according to the action facial expression input unit 16.

FIG. 7 shows a pipeline of the series of operations from the voice input to the image display. The processing result of the voice processing unit 5 is output as voice conversion, and a double buffer is used for drawing. As can be seen from FIG. 7, the lip movement of the CG character displayed as the voice conversion output is delayed by 2 frames at the display rate.
If the display rate is, for example, 30 frames / second, it is about 66 ms, which cannot be seen visually. In addition, the emotion estimation result is delayed by one frame added during a certain period of storing the voice analysis result. As shown in Fig. 7, the storage period is 3
If it is a frame, a delay of 4 frames occurs (about 134 ms if the display rate is 30 frames / sec). However, it takes a considerable amount of time for an actual person to generate emotions when they say something (it depends on the recognition content, but after understanding the other person's words, it is estimated to be around several hundred ms). ), This delay doesn't matter unless the storage period is very large.

(Second Embodiment) A virtual video call device according to a second embodiment of the present invention will be described below with reference to the drawings. FIG. 2 shows the configuration of a virtual video call device according to the second embodiment of the present invention.
Communication unit 101, data download unit 102, communication data determination unit 103, character background selection input unit 2, data management unit 104, voice selection input unit 4, voice processing unit 5, voice conversion unit 6, voice output unit 7, voice input. Unit 8, voice analysis unit 9, emotion estimation unit 10, lip motion control unit 11, body motion control unit 12, facial expression control unit 13, three-dimensional drawing unit 14, display unit 15, motion facial expression input unit 16, viewpoint change input unit. 17, a character shape data storage unit 18, a character motion data storage unit 19, a background data storage unit 20, a texture data storage unit 21, and a music data storage unit 22.

The virtual video call apparatus according to the second embodiment of the present invention configured as described above will be described in detail below. The first embodiment of the present invention is
Only the points of downloading CG data are different, so
Only the operation of downloading CG data will be described.

In this embodiment, the data to be downloaded are CG character data (shape data, clothing texture data, facial expression pattern data and facial expression movement data, lip movement pattern data and lip movement data, thumbnail image data), body movement patterns. Data and physical activity data,
Although it is background data and music data, it can be similarly performed when each data is individually downloaded.

The data download unit 102 accesses the data storage server via the communication unit 101.
This access is the same as that performed when downloading with a normal mobile phone or downloading with a personal computer. For example, specify the server by IP address, notify the server machine of access, and
The procedure according to the P / IP protocol may be performed.

Next, ask the list of the above data stored in the server to be transmitted by http or ftp protocol,
The data download unit 102 receives the transmission result. The user selects the data to download from the list. For example, the list is sent to the communication data determination unit 103 via the communication unit 101, the communication data determination unit 103 determines that the data is in the list, and the three-dimensional drawing unit 14 via the data management unit 104. Send to. The three-dimensional drawing unit 14 converts the list into an image and displays it in the display unit 15.
The user can confirm the content by sending it to and displaying it.

The user's data selection is performed via the data download unit 102. The communication unit 101 transmits the name or identifier of the selected data to the server according to the protocol of the above-mentioned protocol. On the server side, the file of the selected data is transmitted to the communication unit 101 in accordance with the protocol of the protocol described above, the communication data determination unit 103 determines that the communication content is a data file, and sends it to the data management unit 104. .

In the data management unit 104, the data is
CG character data, physical motion pattern data and physical motion data, background data or music data are discriminated and the data size is specified. This determination is not necessary when the selection result in the data download unit 102 is notified to the data management unit 104 via the communication unit 101 and the communication data determination unit 103, since it is known in advance. Next, the data management unit 104, according to the data content, the character shape data storage unit 18, the character motion data storage unit 19,
Background data storage unit 20, texture data storage unit 21,
The music data storage unit 22 is inquired about a free space for storage, and if there is a free space, a data file is sent to the storage unit. The corresponding storage unit stores the data file, and stores the stored address in the data management unit 104.
Send to.

The data management unit 104 adds the data to be stored in the management table to the management table according to the contents of the data. For example, in the case of the CG character data of FIG. 3, 4 is added as the CG character ID, and the address returned from the storage unit is entered in the corresponding column. The same applies to other data. After adding the management table,
The notification of completion is sent to the data download unit 102 via the communication data determination unit 103 and the communication unit 101, and the data download end is sent to the server via the communication unit 101, and the download processing ends.

If there is no data storage area, it is determined that there is no data storage area.
The data download unit 102 is notified via 1. The data download unit 102 notifies the user that there is no storage area (such as displaying on the display unit 15 as described above).
Then, in the same manner as above, the download processing is notified of completion by sending a notification of completion to the data download section 102 via the communication data determination section 103 and the communication section 101, and sending a data download end to the server via the communication section 101 to download processing. Ends.

At the time of communication of voice data, the communication data determination unit 103 determines that the data is voice data and sends it to the voice processing unit 5. The first and second embodiments of the present invention can be implemented as a program for a device having a voice communication unit, a display unit, a voice input / output unit, a central processing unit and a memory. For example, a mobile phone, a pocket computer, a stationary telephone with a display device, a vehicle-mounted terminal device with a communication function, a personal computer, and the like. However,
The processing can be speeded up by having a dedicated three-dimensional processing device, a voice input / output device, and a voice processing device. In the case of a personal computer, it is effective to use one having a three-dimensional graphics board and a sound blaster board. The display unit 15 can use CRT, liquid crystal, organic EL, or the like, and its type does not matter.

FIGS. 8 (a) and 8 (b) are schematic views of the virtual television communication of the present invention. The CG character corresponding to the recipient selected by the above configuration is displayed as a CG character. Enjoy the conversation. In addition, some users can simultaneously display and enjoy conversation in a virtual space. The operation at the time of setting can be performed before and after the call starts.

FIG. 10A is a diagram showing a personal computer (hereinafter referred to as a PC) 1001 having a virtual video call function according to the present invention.
002 and a microphone 1003.

When the user selects at least one character of himself or the other party to start a call, the emotion estimation unit 10 estimates the emotion based on the voice during the call.
In accordance with this emotion estimation, the CG character displayed on the screen 1004 changes its motion and facial expression, so that the virtual television call device can be made more entertaining. Also, PC
Since the user of 1001 can freely select the character and voice of the other party, for example, in the setting of the boss, a virtual TV with enhanced entertainment, such as forest setting background setting, bear setting character setting, and cute voice setting. The PC 1001 having a call function can be used.

FIG. 10B is a diagram showing a mobile phone 1005 having a virtual video call function of the present invention.
The mobile phone 1005 has a hands-free function, and the selected character is displayed on the screen 1006 while performing the motion for which the emotion is estimated. Therefore, a mobile phone 1005 having a virtual video call function with enhanced entertainment can be obtained.

Further, in order to improve the emotion estimation function of the present invention, it is possible to add a new sensor unit to the virtual video call device. FIG. 11 corresponds to FIG. 1 or FIG.
A block diagram in which a sensor unit 1101 is added to the functional block diagram of the virtual video call device in FIG. The sensor unit 1101 serves as a processing unit that detects changes in the body temperature and heartbeat of the user, grip strength of a portable device, and the like, and notifies the emotion estimation unit 10 of the changes. For example, the sensor unit 1101 is
When the change in the body temperature of the user is detected by the thermistor and the result is passed to the emotion estimation unit 10, the emotion estimation unit 10 can more reliably perform the emotion estimation using the body temperature change that is a new parameter for emotion estimation. Can be considered.

FIG. 12A is a diagram showing an example of use of a mobile phone equipped with various sensor units for emotion estimation, which is provided with a grip strength measurement unit 1201 for detecting changes in the grip strength of the user. FIG. 12B is a reference diagram showing a mobile phone including various sensor units for emotion estimation, which includes a grip strength measuring unit 1201 and a thermistor 1202 for measuring a change in user's body temperature. Therefore, it is possible to perform a more reliable emotion estimation function by using a new parameter other than the voice data.

The present invention is not limited to the above-mentioned respective embodiments, but can be carried out within the range in which it can be used. In the above-mentioned embodiments, at least one character of the user and the communication partner is used. Although the virtual video call device displayed on the screen has been described, for example, a PC
Emotional estimation is performed in the communication where a large number of people gather through communication,
It can be considered that the virtual video call device displays a large number of characters with emotion estimation on the screen.

It is also conceivable to reflect the result of emotion estimation in music data and output dark, bright, fun, rhythmic or other music to control the facial expression motion and physical motion of the CG character.

[0127]

As described above, according to the present invention, the communication partner is displayed as a virtual three-dimensional CG character selected by the listener, and the conversation of the communication partner is used to generate a virtual three-dimensional character.
You can have a voice conversation with a CG character. As a result, a new type of communication that enhances the entertainment of voice conversations is provided in a different method from the functions of "You can see the face of the communication partner or a video similar to it" and "Spoof a fictional character". A terminal can be realized. Further, the present invention can realize a communication device with a display device that realizes a conversation in a virtual space without using a device such as a server as in the above-described conventional technique.
Also, since it can be downloaded, the CG data can be updated with new data. You can enjoy conversations with various CG characters by exchanging CG characters or changing voices by voice conversion even if the person you talk to is the same.

In addition, the listener can select his or her own character and the character of the other party of the call, and the character is capable of expressing emotions suitable for the conversation during the call by using the emotion estimation function. It becomes a virtual video call device.

From the above, it is considered that the effect of the present invention is great and brings a new enjoyment and joy to the conversation by the voice conversation device.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a virtual video call device according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a virtual video call device according to a second embodiment of the present invention.

FIG. 3 is a CG character data management table and CG of the present invention.
It is an explanatory view of a character selection screen.

FIG. 4A is an explanatory diagram of a communication management table, a CG data management table, and a voice management table according to the present invention. (B) It is a flowchart of the operation | movement at the time of setting of this invention.

FIG. 5 (a) is an explanatory diagram of sound intensity analysis and lip movement of the present invention. (B) It is explanatory drawing of the phoneme analysis and lip movement of this invention.

FIG. 6 (a) is an explanatory diagram of transition of facial expression motion according to the present invention. (B) It is explanatory drawing of the transition of the body motion of this invention.

FIG. 7 is an explanatory diagram relating to pipeline processing and delay of the present invention.

FIG. 8 (a) is a schematic view of virtual television communication according to the present invention. (B) A schematic view of the virtual television communication of the present invention is shown.

FIG. 9 is a flowchart showing a processing procedure of an emotion estimation method using a frequency signal.

FIG. 10 (a) is a reference diagram showing another mode of use of the first and second embodiments of the present invention. (B) It is a reference drawing showing another mode of use of the first and second embodiments of the present invention.

FIG. 11 is a block diagram in which a sensor unit is added to the functional block diagram of the virtual video call device of the present invention.

FIG. 12A is a diagram showing a usage example of a mobile phone including various sensor units for emotion estimation. (B) It is a reference diagram showing a mobile phone including various sensor units for emotion estimation.

[Explanation of symbols]

1 Communication unit 2 Character background input section 3 Data management department 4 Voice selection input section 5 Voice processing unit 6 Voice converter 7 Audio output section 8 voice input section 9 Speech analysis section 10 emotion estimation part 11 Lip motion controller 12 Physical movement control unit 13 Expression control unit 14 3D drawing unit 15 Display 16 Motion facial expression input section 17 Viewpoint change input section 18 Character shape data storage 19 Character motion data storage 20 Background data storage 21 Texture data storage 22 Music data storage 101 Communication unit 102 communication data determination unit 103 Data download section 104 Data Management Department

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) G10L 3/00 551G (72) Inventor Riki Hijiri 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Inventor Naoki Otani 1006, Kadoma, Kadoma-shi, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Toshiya Naka Naka, 1006, Kadoma, Kadoma-shi, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. (72) Goji Yamamoto, Kanagawa Kanagawa Matsushita Communication Industrial Co., Ltd., 3-1, Tsunashimahigashi, 3-chome, Kohoku-ku, Yokohama, Japan (72) Inventor Shigeo Asahara 1006, Kadoma, Kadoma, Osaka Prefecture Matsushita Electric Industrial Co., Ltd. F-term (reference) 5B050 AA08 BA08 BA09 BA12 CA08 EA24 EA28 FA02 FA05 5D015 AA05 AA06 KK02 5E501 AC16 BA17 CA08 CB02 CB03 CB09 CB14 CB15 DA11 EA21 EB05 FA14 FA27 FA32

Claims (35)

[Claims] 1. A communication means for performing voice communication, a character selection means for selecting CG character shape data of at least one of a user himself or a communication partner, and a voice input means for inputting a voice of the user himself. The voice output means for outputting the voice of the communication partner, the voice data of the communication partner received by the communication means or both the voice data of the communication partner received and the voice data of the user himself / herself input by the voice input means. A voice analysis unit that performs voice analysis on the other hand, an emotion estimation unit that estimates the emotional state of the communication partner or the communication partner and the user himself using the voice analysis result of the voice analysis unit, and control of the motion of the CG character Motion control means for performing the motion estimation based on the emotion estimation means, and motion data generated based on the CG character shape data and the control information of the motion control means. Virtual video call device, it characterized in that it comprises a drawing means for generating an image by performing the drawing process had, and display means for displaying the image generated by the drawing means. 2. The emotion estimating means notifies the motion control means of the estimation result of the emotion estimating means, and the motion control means specifies the motion data based on the notification result. The virtual video call device according to claim 1. 3. The motion control means includes a lip motion control means for generating lip motion control information of the CG character data based on a voice analysis result of the voice analysis means, and the drawing means has the CG character shape. 2. The virtual video call apparatus according to claim 1, wherein an image is generated by performing a drawing process using the lip movement data generated based on the data and the control information of the lip movement control means. 4. The emotion estimation means notifies the lip movement control means of the estimation result of the emotion estimation means, and the lip movement control means specifies the lip movement data based on the notification result. The virtual video call device according to claim 3, which is characterized. 5. The movement control means includes a body movement control means for controlling a body movement of the CG character, and the drawing means is a body movement based on the body movement control information generated by the body movement control means. The virtual video call device according to claim 1, wherein drawing processing is performed using data. 6. The emotion estimation means notifies the physical movement control means of the estimation result of the emotion estimation means, and the physical movement control means specifies the physical movement data based on the notification result. The virtual video call device according to claim 5, which is characterized. 7. The virtual video call device further comprises a body motion pattern data selecting unit that defines a specific body motion, and the body motion controlling unit sets the body motion pattern data selected by the selecting unit. The virtual video call device according to claim 5, wherein the physical control is performed based on the physical control. 8. The motion control means includes a facial expression control means for controlling a facial expression motion of the CG character, and the drawing means stores facial expression motion data based on the facial expression motion control information generated by the facial expression control means. The virtual video call device according to claim 1, wherein the drawing process is performed using the virtual video call device. 9. The emotion estimating means notifies the facial expression control means of the estimation result of the emotion estimating means, and the facial expression control means specifies the facial expression motion data based on the notification result. The virtual video call device according to claim 8. 10. The virtual video call device further comprises voice conversion means for converting the received voice of the communication partner into another voice, according to any one of claims 1 to 9. Virtual video call device. 11. The virtual video call device further comprises voice selection input means capable of selecting the voice quality when converting the voice of the communication partner received by the voice conversion means into another voice. The virtual video call device according to claim 10. 12. The drawing means generates an image of a CG character of the communication partner when the communication is received from the communication partner, and the display means displays the image of the CG character when the communication is received and before voice communication is started. Is displayed to indicate a voice communication waiting state. 13. The voice output means outputs music data corresponding to each of the communication partners when the communication is received from the communication partner to indicate a voice communication waiting state. The virtual video call device according to item 1. 14. The virtual video call device according to claim 1, wherein the drawing means performs a drawing process using background data to generate an image. 15. The virtual video call device according to claim 14, wherein the virtual video call device further includes a background selection unit capable of selecting background data. 16. The drawing means performs a three-dimensional drawing process to generate a three-dimensional image.
16. The virtual video call device according to any one of 1 to 15. 17. The virtual video call device comprises means for storing lip motion data and means for downloading lip motion data from an external device and storing it in the storage means. Alternatively, the virtual video call device according to item 4. 18. The virtual video call device comprises means for storing lip movement pattern data, and means for downloading lip movement pattern data from an external device and storing it in the storage means. Item 15. The virtual video call device according to Item 17. 19. The virtual video call device comprises means for storing physical movement data, and means for downloading the physical movement data from an external device and storing it in the storage means. Or claim 6
The virtual video call device described in. 20. The virtual video call device comprises means for storing physical movement pattern data, and means for downloading physical movement pattern data from an external device and storing it in the storage means. The virtual video call device according to item 7. 21. The virtual video call device comprises a storage unit for facial expression motion data, and a unit for downloading the facial expression motion data from an external device and storing it in the storage unit. Alternatively, the virtual video call device according to item 9. 22. The virtual video call device comprises: storage means for storing facial expression pattern data; and means for downloading the facial expression pattern data from an external device and storing it in the storing means. The described virtual video call device. 23. The virtual video call device comprises: means for storing music data; and means for downloading music data from an external device and storing it in the storage means. Virtual video call device. 24. The virtual video call device comprises: background data storage means; and means for downloading the background data from an external device and storing the background data in the storage means. The described virtual video call device. 25. The virtual video call device is a CG
25. The clothing texture data of a character is stored, and the clothing texture data of a CG character is downloaded from an external device, and means for storing the data in the storage means is provided. The virtual video call device described in. 26. The virtual video call device is a CG
Character shape data storage means and CG from external device
26. The virtual video call device according to claim 1, further comprising means for downloading the character shape data and storing it in the storage means. 27. The virtual video call device is a CG
2. The virtual video call apparatus according to claim 1, further comprising a display mode selection unit that determines whether or not to display a character. 28. The display mode is a communication partner display mode in which only the CG character of the communication partner is displayed, a simultaneous display mode in which CG characters of the communication partner and the user himself are displayed, and no CG character is displayed. 28. The virtual video call device according to claim 27, which is in one of the display modes. 29. The virtual video call device is a CG
8. The virtual video call device according to claim 5, further comprising means capable of controlling designation and start of a physical action of the character. 30. The virtual video call device is a CG
10. The virtual video call device according to claim 8, further comprising means for controlling designation and start of facial expression motion of the character. 31. The virtual video call device is a CG
31. The virtual video call device according to claim 1, further comprising a viewpoint changing unit for displaying a character from a viewpoint direction according to a user's intention. 32. A virtual video call system for making a call between at least a call device between a user and a communication partner, wherein the virtual video call system comprises at least a user's call device and a communication device of a communication partner. The communication device includes a communication means for performing voice communication, a character selection means for selecting CG character shape data of at least one of the user himself or the communication partner, and a voice input means for inputting the voice of the user himself. A voice output means for outputting the voice of the communication partner, voice data of the communication partner received by the communication means or voice data of the received communication partner and voice data of the user himself / herself input by the voice input means. A voice analysis means for performing voice analysis on both, and a voice analysis result of the voice analysis means are used to communicate with the communication partner or the communication partner. Emotion estimation means for estimating the emotional state of the user himself, movement control means for controlling the movement of the CG character based on the emotion estimation means, based on the CG character shape data and control information of the movement control means A virtual video call system comprising: a drawing unit that performs an image drawing process by using the operation data generated by the above process to generate an image; and a display unit that displays the image generated by the drawing unit. 33. The emotion estimating means notifies the motion control means of the estimation result of the emotion estimating means, and the motion control means specifies the motion data based on the notification result. The virtual video call system according to claim 32. 34. A program for performing a virtual video call between a communication partner device and its own device by performing communication between at least a communication partner and a user, the communication step of performing voice communication, and the user himself / herself. Alternatively, a character selection step of selecting CG character shape data of at least one of the communication partners, a voice input step of inputting the voice of the user himself, a voice output step of outputting the voice of the communication partner, the communication step In the voice analysis step of performing voice analysis on both the voice data of the communication partner received in step 1 or both the voice data of the received communication partner and the voice data of the user himself / herself input in the voice input step; An emotion estimation step that estimates the emotional states of the communication partner or the communication partner and the user himself using the voice analysis result. A motion control step of controlling the motion of the CG character based on the emotion estimation step; and a drawing process using motion data generated based on the CG character shape data and control information of the motion control step. A program, comprising: a drawing step of performing an image to generate an image; and a display step of displaying the image generated in the drawing step. 35. The program according to claim 34, wherein the emotion estimation step specifies the motion data based on the estimation result obtained in the emotion estimation step.