JP4896118B2 - Video phone terminal - Google Patents

Description

  The present invention relates to a videophone terminal that can easily convey a user's feelings and impressions to a partner.

  In a videophone system composed of a plurality of videophone terminals and a network or the like, video shot by each videophone terminal is sent to the other terminal via the network together with sound. For this reason, it is possible to have a conversation while looking at the face even with a remote party. If you can talk while looking at the other party's face, you can visually check the other party's facial expression as well as the tone of the voice.

  However, there are some users who do not like the video of themselves being sent to the partner terminal as it is. In addition, there are users who feel that there is no interest in talking while watching video shot by themselves or the other party. For this reason, feature points of each part such as eyebrows, eyes, nose, and mouth are extracted from an image of the user's face, and a virtual character (hereinafter referred to as “virtual character”) resembling the user's face from the feature points. This is a technology that sends the video of the virtual character to the other terminal.

  In this technology, first, after recognizing which area of an image obtained by photographing a user's face (hereinafter referred to as “face image”) is a feature of each part such as eyebrows, eyes, nose, and mouth. Points (hereinafter referred to as “feature points”) are extracted. FIG. 1 is an explanatory diagram showing a face image and each feature point. Next, based on the feature points of each part, a virtual character resembling the user's face is generated from the average face character obtained by averaging the features of each part. More specifically, a difference between the extracted feature point and the feature point of the average face character is calculated, and the difference data is reflected in the average face character, thereby generating a virtual character resembling the user's face. . FIG. 2 is an explanatory diagram showing a virtual character imitating a user's face.

  Then, each feature point in the user's face image is tracked, and the movement of each feature point is reflected in the virtual character. By doing so, the movement of each part accompanying the change in the facial expression of the user is linked with the movement of each part of the virtual character, so that the facial expression of the virtual character also changes in accordance with the change in the facial expression of the user. In addition, by changing the movement of each feature point in the user's face image to a completely different character without making the virtual character resemble the user's face, the expression of the virtual character is changed according to the change of the user's expression. You can also.

  Furthermore, if all the parts forming the face move in the same direction on the coordinate axis of the face image, it can be considered that the entire face has moved. Therefore, when the user crawls, crawls his head or shakes his head, this action can be reflected in the virtual character.

Japanese translation of PCT publication No. 2002-511617 Japanese translation of PCT publication No. 2002-511620

  As described above, in the above-described conventional technology, the movement of each part accompanying the change in the facial expression of the user is linked to the movement of each part of the virtual character, so the facial expression of the virtual character changes in accordance with the facial expression of the user. Furthermore, since the movement of the user's head is reflected in the virtual character, the virtual character moves in the same manner when the user crawls, crawls his head or shakes his head.

  However, since the conventional technique only reflects the change in the facial expression of the user and the movement of the head as it is in the virtual character, it can be expressed only by the facial expression and movement of the virtual character as described above, except for voice. It cannot express emotions or impressions that cannot be expressed. However, in order to convey the user's emotions or impressions to the other party in an easy-to-understand manner, it is better to emphasize changes in facial expressions, etc. or to express them with marks etc. It is thought that it is easy to communicate. In addition, this is more interesting in terms of entertainment because it adds more fun to the expression and movement of the virtual character.

  The present invention has been made in view of the above-described conventional demands, and an object of the present invention is to provide a videophone terminal that can easily convey a user's feelings and impressions to the other party.

In order to achieve the above object, a videophone terminal according to the present invention is provided via a network by video and audio including a virtual character (hereinafter referred to as “virtual character”) generated based on a human face. A videophone terminal that communicates with the other terminal, and a transmission / reception unit that transmits and receives video and audio data; an input unit that includes a plurality of buttons; When the first button of the plurality of buttons is operated, before disconnecting the line, the first video including the virtual character disappears from the screen and is transmitted to the transmission / reception unit. When the second button of the plurality of buttons of the input unit is operated, before the line is disconnected, the second video including the virtual character disappears from the screen to the transmission / reception unit. System to send The first button and the second button are different buttons, and the first video including the virtual character and the second video including the virtual character are different. It is a video.

The program according to the present invention includes a transmission / reception unit that transmits and receives video and audio data, and includes video and audio including a virtual character (hereinafter referred to as “virtual character”) generated based on a human face. A step of accepting input by a plurality of buttons to a videophone terminal which is a computer that communicates with another terminal via a network; and when the communication with the other terminal is terminated, When the button 1 is operated, before the line is disconnected, the first video including the virtual character disappears from the screen is transmitted to the transmission / reception unit, and the first of the plurality of buttons is transmitted. When the operation of the button 2 is performed, before the line is disconnected, the second video including the virtual character disappears from the screen and is transmitted to the transmitting / receiving unit. A program for execution, wherein the first button and the second button are different buttons, a first video including the virtual character, a second video including the virtual character, Is a program characterized by different images.

  As described above, according to the videophone terminal of the present invention, it is possible to convey the user's feelings and impressions to the other party in an easily understandable manner. In particular, it is possible to convey the impression of the conversation the user has to the other party according to the content of the video.

  Embodiments of a videophone terminal according to the present invention will be described below with reference to the drawings.

  The video phone terminal according to the present embodiment is a communication terminal such as a mobile phone, a PHS, or a PDA equipped with a camera capable of shooting a moving image or a still image (hereinafter collectively referred to as “video”). The videophone terminal can be used as a videophone by transmitting and receiving video and audio. However, a video transmitted / received between videophone terminals during a videophone call is a virtual character (hereinafter referred to as a “virtual character”) generated based on a user's face shot with a camera in addition to a video shot with a camera. It may be a video. This embodiment demonstrates the case where the image | video of the said virtual character is transmitted / received.

  Hereinafter, generation of a virtual character will be described. The videophone terminal according to the present embodiment recognizes which region is a face from the face image of the user taken by the camera. Next, points (hereinafter referred to as “feature points”) that are features of each part such as eyebrows, eyes, nose, and mouth are extracted from the face image. FIG. 1 is an explanatory diagram showing a face image and each feature point. The eyebrows, eyes, nose, and mouth, which are the main parts that make up the face, change slightly depending on the facial expression. As these parts change, the relative position with other feature points changes when the facial expression changes. Extracted as

  Next, based on the feature points of each part, a virtual character close to the user's face is generated from the average face character obtained by averaging the features of each part. More specifically, a difference between the extracted feature point and the feature point of the average face character is calculated, and a virtual character close to the user's face is generated by reflecting the difference data on the average face character. FIG. 2 is an explanatory diagram showing a virtual character imitating a user's face.

  Then, each feature point in the user's face image is tracked, and the movement of each feature point is reflected in the virtual character. Also, if all the parts that form the face move in the same direction on the coordinate axis of the face image, it can be considered that the entire face has moved, so when the user scolds, crawls his head, shakes his head, This action is reflected in the virtual character.

  Hereinafter, the configuration of the videophone terminal of the present embodiment will be described with reference to FIG. As shown in the figure, the videophone terminal of the present embodiment includes a camera 101, a video processing unit 103, a microphone 105, a speaker 107, an audio processing unit 109, a virtual character generation unit 111, and a display unit 113. A keyboard 115, a storage unit 117, a central processing unit 119, a wireless unit 121, and an antenna 123.

  The video processing unit 103 analyzes a video taken by the camera 101 to recognize a face from the video and extract a feature point. The voice processing unit 109 performs predetermined processing on the own voice input from the microphone 105, processes the other party's voice data received from the other party's videophone terminal, and outputs the processed data from the speaker 107. is there. Note that the processing performed by the speech processing unit 109 includes analysis of elements that are features of speech such as volume, phoneme, and pitch, and the analysis is performed on the speech of the user and the other party.

  The virtual character generation unit 111 generates a virtual character based on the feature points extracted by the video processing unit 103, and reflects the user's facial expression and action taken by the camera 101 on the virtual character. Note that the virtual character generation unit 111 may change the generated virtual character partially or entirely based on an instruction from the central processing unit 119. Note that the virtual character generation unit 111 sets the background of the virtual character displayed on the display unit 113 to a predetermined image based on the schedule information and date information stored in the storage unit 117. For example, a cake image is used as the background for the user's birthday, and the background is changed according to the date or period, such as a doll image on March 3 and a carp streamer image on May 5.

  In addition, the storage unit 117 stores a program related to changes and actions of the virtual character's facial expression, predetermined video and audio data, user schedule information, date information, and the like. Note that the virtual character generation unit 111 sets the background of the virtual character displayed on the display unit 113 to a predetermined image based on the schedule information and date information stored in the storage unit 117. For example, a cake image is used as the background for the user's birthday, and the background is changed according to the date or period, such as a doll image on March 3 and a carp streamer image on May 5.

  The keyboard 115 is used for performing a transition to a hold mode (to be described later), disconnecting a line, and other instructions to the central processing unit 119. In addition, the central processing unit 119 performs video and audio processing according to the operation of the keyboard 115 and keywords, predetermined processing at the time of connection / disconnection of a line, start / release of a hold mode, and compression / decompression of video data and audio data. Processing is performed. The wireless unit 121 performs modulation and demodulation of video and audio data, and transmits and receives signals via the antenna 123.

  Based on the above description, an embodiment of the videophone terminal according to the present invention will be described in detail in the order of [First Embodiment], [Second Embodiment], and [Third Embodiment].

[First Embodiment]
In the first embodiment, when a video phone conversation using the above-mentioned virtual character is performed, the user performs a predetermined operation using the keyboard 115 shown in FIG. When a predetermined keyword is recognized from the uttered voice, a video in which the virtual character is changed or a completely different video is obtained.

  To change the virtual character is, for example, a pattern that expresses emotions such as changing the size of each part of the virtual character's face or the entire face, putting vertical lines in the virtual character's eyes, or blushing the cheeks. Is to add. If only the virtual character's eyes are made larger than usual, a surprise can be expressed, and if the entire face is made larger than usual and turned red, anger can be expressed.

  Further, what can be considered as completely different images include, for example, an exclamation mark (!) And a question mark (?). The exclamation mark (!) Can express exclamation, and the question mark (?) Can indicate doubt.

  In addition, the image with the thumb raised is stored in the storage unit 117 in advance, and the image is associated with the keyword “well done!”. In this way, when the voice processing unit 109 recognizes the keyword from the utterance that it has uttered, the central processing unit 119 reads the image with the thumb raised from the storage unit 117 and displays it instead of or superimposed on the virtual character video. . Note that the image is not limited to an image and may be a moving image.

  Similarly, a predetermined sound effect is stored in the storage unit 117 in advance, and the sound effect is associated with a predetermined keyboard operation. In this way, when a predetermined operation is performed using the keyboard 115, the central processing unit 119 reads out the sound effect data from the storage unit 117 and reproduces it instead of or overlying the sound of the user and the other party.

  As described above, in the present embodiment, when a user performs a predetermined operation using the keyboard 115 or when the voice processing unit 109 recognizes a predetermined keyword, the expression and action different from those of a normal virtual character are completely different. Therefore, it is possible to express an emotion or impression that cannot be expressed only by the facial expression and movement of the virtual character. Further, in this case, since the interestingness is added to the facial expression and movement of the virtual character, it is possible to improve the entertainment characteristics of the communication using the video using the virtual character. In addition, since a predetermined sound effect can be reproduced in accordance with a keyboard operation, it is possible to express a user's emotion or impression that cannot be expressed only by voice or video with sound.

[Second Embodiment]
In the second embodiment, the virtual character returns to the screen along with the reproduction of the melody when shifting to the hold mode during the videophone conversation using the virtual character and releasing the hold mode. To do. When the user presses the hold button on the keyboard 115 shown in FIG. 3 during the hold, the central processing unit 119 detects this and releases the hold mode. Alternatively, if the feature point extracted by the video processing unit 103 from the video captured by the camera 101 during the hold corresponds to the user who had a conversation before the hold, the central processing unit 119 releases the hold mode.

  At this time, the central processing unit 119 executes a predetermined program, reads predetermined melody data from the storage unit 117, and displays the virtual character that was displayed before the suspension along with the reproduction of the melody. However, between the release of the hold mode and the actual display of the virtual character, the virtual character generation unit 111 is based on the feature points extracted by the video processing unit 103 from the video captured by the camera 101. Since it is necessary to reflect the expression and movement of the character in the virtual character, it takes some time. For this reason, the screen displayed during this time is an image of a virtual character that performs a predetermined action together with the melody without changing the expression. The predetermined movement is, for example, an action such as a virtual character opening a door and entering.

  As described above, according to the present embodiment, when the hold mode is canceled, an image in which the virtual character performs a predetermined action in conjunction with a predetermined melody is displayed, so that the opponent has returned from the hold state. You can know visually.

[Third Embodiment]
In the third embodiment, when the videophone conversation using the above-mentioned virtual character is terminated, the virtual character disappears from the screen in response to the user selecting the button on the keyboard 115 shown in FIG. The video is displayed last and the line is disconnected. The predetermined video is a video in which the virtual character disappears from the screen with a flower, a video in which the virtual character is crushed from the head with a heavy load, or the like.

  However, the impression given to the other party depends on the content of the video. For this reason, for example, when the user feels that the conversation is enjoyable, the user presses a predetermined button to end the image in which the virtual character disappears from the screen with a flower. Press another button to end the video with the virtual character crushed from the head under heavy load. As described above, when a predetermined button is pressed at the end of the conversation, the central processing unit 119 reads out and transmits video data corresponding to the pressed button from the storage unit 117 before disconnecting the line.

  As described above, according to the present embodiment, when a conversation is ended, a different video is displayed according to the button selection, so that the user can convey the impression of the conversation that the user has depending on the content of the video. it can.

  Note that the video processing unit 103, the audio processing unit 109, the virtual character generation unit 111, and the central processing unit 119 of the videophone terminal of each embodiment described above may operate by executing a program.

Explanatory drawing showing face image and each feature point Explanatory drawing which shows the virtual character imitating the user's face The block diagram which shows the structure of the videophone terminal of one Embodiment which concerns on this invention

Explanation of symbols

101 Camera 103 Video Processing Unit 105 Microphone 107 Speaker 109 Audio Processing Unit 111 Virtual Character Generation Unit 113 Display Unit 115 Keyboard 117 Storage Unit 119 Central Processing Unit 121 Radio Unit 123 Antenna

Claims (2)

A videophone terminal that communicates with other terminals via a network by video and audio including a virtual character (hereinafter referred to as “virtual character”) generated based on a human face,
A transmission / reception unit for transmitting and receiving video and audio data;
An input unit having a plurality of buttons;
When the first button of the plurality of buttons of the input unit is operated when ending communication with the other terminal, the first character including the virtual character is disconnected before disconnecting the line. When the video disappearing from the screen is transmitted to the transmission / reception unit and the second button among the plurality of buttons of the input unit is operated, the virtual character is included before the line is disconnected. A control unit that causes the transmission / reception unit to transmit a video in which the second video disappears from the screen,
The first button and the second button are different buttons,
The videophone terminal , wherein the first video including the virtual character and the second video including the virtual character are different from each other . A transmission / reception unit that transmits and receives video and audio data, and communicates with other terminals via a network using video and audio including virtual characters (hereinafter referred to as “virtual characters”) generated based on human faces. To a videophone terminal that is a computer that performs
Accepting input from multiple buttons;
When the first button of the plurality of buttons is operated when the communication with the other terminal is terminated, the first video including the virtual character is displayed on the screen before the line is disconnected. When the video disappearing from the screen is transmitted to the transmission / reception unit and the second button among the plurality of buttons is operated, the second video including the virtual character is displayed on the screen before the line is disconnected. A step of causing the transmitter / receiver to transmit the video disappearing from the program, and a program for executing the video,
The first button and the second button are different buttons,
The first image including the virtual character and the second image including the virtual character are different images.
A program characterized by that.