JP2004287558A - Video phone terminal, virtual character forming device, and virtual character movement control device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video phone terminal, a virtual character forming device and a virtual character movement control device capable of utilizing a virtual character of high entertaining nature. <P>SOLUTION: The virtual character formed based on a user's face photographed by a camera 101 is not formed to completely look like the user's face, but a virtual character forming part 111 forms the virtual character delicately or moderately resembling the user's face by emphasizing the features of previously specified parts. The variation of user's expression or motion is not reflected as it is in the expression or motion of the virtual character, but the virtual character forming part 111 provides the virtual character with original individuality of having expression or motion which the user does not actually have. Further, the neck motion of the user is not reflected as it is in the motion of the virtual character, but the virtual character forming part 111 moves the neck of the virtual character also according to other factors. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a videophone terminal capable of utilizing virtual characters having high entertainment properties, a virtual character generation device, and a virtual character operation control device.
[0002]
[Prior art]
In a videophone system including a plurality of videophone terminals, a network, and the like, video captured by each videophone terminal is transmitted to a partner terminal via a network together with audio. For this reason, it is possible to have a conversation with a remote party while looking at the face. If you can talk while looking at the other person's face, you can visually confirm not only the tone of the voice but also the expression of the other person, so that there is an advantage that a more realistic high-level communication can be taken.
[0003]
However, some users do not like to send a video of themselves to the partner terminal as it is. In addition, some users find it interesting to have a conversation while watching the video taken by themselves or the other party. For this reason, feature points of each part such as eyebrows, eyes, nose, mouth, etc. are extracted from an image of the user's face, and a virtual character (hereinafter, “virtual character”) similar to the user's face is extracted from the feature points ) Is generated, and the image of the virtual character is transmitted to the partner terminal as an alter ego.
[0004]
In this technique, first, after recognizing which region of an image of a user's face (hereinafter, referred to as “face image”) is a face, the face image is used to characterize each part such as eyebrows, eyes, nose, and mouth. Points (hereinafter referred to as “feature points”) are extracted. FIG. 1 is an explanatory diagram showing a face image and each feature point. Next, based on the feature points of each part, a virtual character that resembles the user's face is generated from the average face character in which the features of each part are averaged. More specifically, a virtual character that resembles a user's face is generated by calculating a difference between the extracted feature point and the feature point of the average face character, and reflecting the difference data on the average face character. . FIG. 4 is an explanatory diagram illustrating a virtual character that resembles a user's face.
[0005]
Then, each feature point in the user's face image is tracked, and the movement of each feature point is reflected on the virtual character. By doing so, the movement of each part associated with the change of the user's facial expression cooperates with the movement of each part of the virtual character, so that the facial expression of the virtual character also changes in accordance with the change of the user's facial expression. By changing the movement of each feature point in the user's face image to a completely different character without making the virtual character resemble the user's face, the expression of the virtual character is changed according to the change of the user's expression. You can also.
[0006]
Furthermore, if all of the parts forming the face move in the same direction on the coordinate axis of the face image, it can be considered that the entire face has moved. Thus, when the user nods, bows, or shakes his head, this action can be reflected on the virtual character.
[0007]
[Patent Document 1]
Japanese Patent Publication No. 2002-511617 [Patent Document 2]
Japanese Unexamined Patent Publication No. 2002-511620
[Problems to be solved by the invention]
As described above, in the above-described conventional technology, the virtual character is made to resemble the face of the user, or to be a completely different character. However, considering a virtual character from the viewpoint of entertainment, it is considered that the virtual character is more subtle or more similar than the user's face. On the other hand, if the virtual character is a completely different character, it is not reminiscent of the face of the conversation partner.
[0009]
Further, in the above-described conventional technology, since the movement of each part associated with the change of the user's facial expression is linked to the movement of each part of the virtual character, the facial expression of the virtual character changes according to the user's facial expression. However, when considering virtual characters from the point of view of entertainment, it is thought that those who have surprising expressions and movements of the virtual characters are more interesting than reflecting the changes in the user's facial expressions directly on the expressions of the virtual characters. Can be
[0010]
Further, in the above-described conventional technology, the movement of the user's neck is reflected on the virtual character. Therefore, when the user nods, bows, or shakes the head, the virtual character performs the same movement. However, considering a virtual character from the viewpoint of entertainment, it is considered that a person who has unexpectedness in the movement of the virtual character is more interesting than a function of directly reflecting the movement of the user's neck in the movement of the virtual character.
[0011]
Therefore, when a videophone is used as a communication tool including an entertaining element, it is desirable to be able to utilize a virtual character having high entertainment properties.
[0012]
The present invention has been made in view of the above-mentioned conventional needs, and provides a videophone terminal capable of utilizing virtual characters having high entertainment properties, and a virtual character generation device and a virtual character operation control device. It is an object.
[0013]
[Means for Solving the Problems]
In order to achieve the above object, a virtual character generation device according to the present invention is a virtual character generation device that generates a virtual character (hereinafter, referred to as a “virtual character”) based on a human face. Feature point extracting means for extracting the feature points of each part from the image of the captured image, the feature points of each part extracted by the feature point extracting means, and the character of the average face in which the features of each part are averaged. Feature point difference calculation means for calculating a difference from a feature point; virtual character generation means for generating the virtual character of the user by reflecting the difference calculated by the feature point difference calculation means on the character of the average face. Wherein the virtual character generation means changes a difference calculated by the feature point difference calculation means for a predetermined part so as to emphasize characteristics of the predetermined part. In to be reflected in the character of the average face.
[0014]
In this way, by emphasizing the features of the predetermined part, it is possible to generate a virtual character that is subtle or moderately similar to the user's face. The virtual character is considered to be more subtle or more similar than the user's face, so it is possible to utilize a highly entertaining virtual character.
[0015]
Further, the virtual character operation control device according to the present invention provides a virtual character (hereinafter referred to as a “virtual character”) generated based on a human face to a unique character that is not related to the actual expression or movement of the user. A virtual character operation control device having personality, wherein a desired program is downloaded from a program storage device storing at least one program for realizing the unique personality, and the downloaded program is executed to execute the virtual program. The character is controlled to perform a predetermined operation according to the personality corresponding to the program.
[0016]
As described above, by downloading and executing the program for realizing the unique personality from the program storage device, it is possible to cause the virtual character to have an expression or movement different from the actual expression or movement of the user. Therefore, a change in the expression or movement of the virtual character can be enjoyed, so that the virtual character having high entertainment properties can be utilized.
[0017]
Further, the virtual character operation control device according to the present invention is a virtual character operation control device that controls the movement of the neck of a virtual character (hereinafter, referred to as “virtual character”) generated based on a human face, Apart from the actual neck movement of the user as the basis of the virtual character, the virtual character is controlled so as to perform a predetermined neck movement according to a keyword, a feature of audio or video, or randomly.
[0018]
In this manner, the virtual character performs a neck movement different from the actual movement of the user. It is considered that the movement of the virtual character is more surprising and interesting than the reflection of the movement of the user's neck as it is on the movement of the virtual character. Therefore, a virtual character having high entertainment properties can be utilized.
[0019]
Furthermore, a videophone terminal according to the present invention includes the virtual character generation device according to claim 1 or the virtual character operation control device according to claim 2 or 3, and is configured to communicate with another terminal and the virtual character via a network. Performs video and audio communication. Therefore, it is possible to enhance the entertainment property of the communication using the virtual character.
[0020]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of a videophone terminal according to the present invention will be described with reference to the drawings.
[0021]
The videophone terminal according to the present embodiment is a communication terminal such as a mobile phone, a PHS, or a PDA equipped with a camera capable of capturing a moving image or a still image (hereinafter, collectively referred to as “video”). It can be used as a videophone by transmitting and receiving video and audio to and from the videophone terminal. However, the video transmitted and received between terminals during a videophone call is a video of a virtual character (hereinafter, referred to as a “virtual character”) generated based on a user's face captured by the camera, in addition to a video captured by the camera. It may be. In the present embodiment, a case where the video of the virtual character is transmitted and received will be described.
[0022]
Hereinafter, generation of a virtual character will be described. The videophone terminal according to the present embodiment recognizes which area is a face from a user's face image captured by a camera. Next, points (hereinafter, referred to as “feature points”) that are features of each part such as eyebrows, eyes, nose, and mouth are extracted from the face image. FIG. 1 is an explanatory diagram showing a face image and each feature point. The major parts of the face, the eyebrows, eyes, nose, and mouth, change subtly depending on the facial expression.These parts, such as these parts, change their relative position with other characteristic points when the facial expression changes. Is extracted as
[0023]
Next, based on the feature points of each part, a virtual character close to the user's face is generated from the average face character in which the features of each part are averaged. More specifically, a virtual character close to the user's face is generated by calculating a difference between the extracted feature point and the feature point of the average face character, and reflecting the difference data on the average face character.
[0024]
Then, each feature point in the user's face image is tracked, and the movement of each feature point is reflected on the virtual character. Also, if all parts forming the face move in the same direction on the coordinate axis of the face image, the entire face can be regarded as moving, so when the user nods, bows, or shakes his head, This operation is reflected on the virtual character.
[0025]
Hereinafter, the configuration of the videophone terminal of the present embodiment will be described with reference to FIG. FIG. 2 is a block diagram illustrating a configuration of the videophone terminal of the present embodiment. As shown in the figure, the videophone terminal of this embodiment includes a camera 101, a video processing unit 103, a microphone 105, a speaker 107, an audio processing unit 109, a virtual character generation unit 111, a display unit 113 , A keyboard 115, a central processing unit 117, a wireless unit 119, and an antenna 121.
[0026]
The video processing unit 103 analyzes a video captured by the camera 101 to recognize a face from the video and extract feature points. The audio processing unit 109 performs predetermined processing on own audio input from the microphone 105, processes audio data of the other party received from the other videophone terminal, and outputs the processed data from the speaker 107. is there. Note that the processing performed by the voice processing unit 109 includes analysis of elements that are characteristics of the voice such as volume, phoneme, and pitch, and the analysis is performed on the voice of the user and the partner.
[0027]
Further, the virtual character generation unit 111 generates a virtual character based on the feature points and the like extracted by the video processing unit 103, and reflects the user's facial expressions and actions taken by the camera 101 on the virtual character. The virtual character generation unit 111 may change the generated virtual character partially or entirely based on an instruction from the central processing unit 117. The display unit 113 displays a virtual character generated by the virtual character generation unit 111, a virtual character transmitted from the other party's videophone terminal, and the like.
[0028]
The keyboard 115 is for issuing an instruction relating to virtual character generation, which will be described later, to the virtual character generation unit 111, and for issuing other instructions, etc., to the central processing unit 117. The central processing unit 117 performs, for example, MPEG compression of the virtual character generated by the virtual character generation unit 111, synchronization of the movement of the virtual character with the audio, and compression / decompression processing of video data and audio data. It is. The radio section 119 performs modulation and demodulation of video and audio data and transmits and receives signals via the antenna 121.
[0029]
Based on the above description, the embodiment of the videophone terminal according to the present invention will be described in detail in the order of [First Embodiment], [Second Embodiment], and [Third Embodiment].
[0030]
[First Embodiment]
In the first embodiment, the virtual character does not completely resemble the face of the user, but subtly or moderately resembles. As described above, when generating the virtual character, the difference between the extracted feature point and the feature point of the average face character is calculated, and the difference data is reflected on the average face character. In the present embodiment, the difference data is not directly reflected on the character of the average face, but is reflected on a part specified in advance by the user after changing the difference data so as to emphasize the feature.
[0031]
It should be noted that the designation of the part to emphasize the feature is performed from the keyboard 115 shown in FIG. Then, the change of the difference data of the designated part is performed by the virtual character generation unit 111. The feature point extracting means in the claims corresponds to the video processing unit 103, and the feature point difference calculating means and the virtual character generating means correspond to the virtual character generating unit 111.
[0032]
FIG. 3 is an explanatory diagram illustrating an example of (a) a virtual character that resembles a user's face and (b) an example of a virtual character that subtly or moderately resembles a user's face. For example, when the user has instructed to emphasize the size of the eyes, the difference data of the feature points of the eyes is changed so that the eyes become larger as a whole. Then, by reflecting the changed difference data of each feature point of the eyes and the difference data of each feature point of the eyebrows, the nose, and the mouth to the character of the average face, the eyes shown in FIG. A virtual character in which the size of is emphasized is generated.
[0033]
As described above, according to the present embodiment, it is possible to generate a virtual character that is delicate or slightly similar to the user's face by emphasizing the features of the part specified in advance. Since the virtual character is considered to be more subtle or slightly more interesting than the user's face too much, according to the present embodiment, it is possible to enhance the entertainment of communication using the virtual character. it can. In addition, since the feature is emphasized for the designated part, it is possible to generate a virtual character in which only the part designated by the user is emphasized.
[0034]
In the present embodiment, the features of the face parts are emphasized, but the parts that the user does not want to emphasize may be deformed so as to be inconspicuous.
[0035]
[Second embodiment]
In the second embodiment, a change in the expression or movement of the user is not directly reflected on the expression or movement of the virtual character, but a unique character such as an expression or movement that the user does not actually perform is added to the virtual character. I have it. In the present embodiment, a character server (not shown) provided with a plurality of programs for realizing various personalities is separately provided. The user accesses the character server using the videophone terminal and downloads a program for realizing the desired personality to the videophone terminal. Then, the virtual character is set to have the personality. The character server corresponds to a program storage device in the claims.
[0036]
As described above, if a program corresponding to a desired personality is downloaded in advance from the character server and set, the program for realizing the personality is executed by the virtual character generation unit 111 in the next and subsequent videophones, and the virtual character is generated. The character performs a predetermined operation according to this personality.
[0037]
In addition, the personality that performs an operation completely unrelated to the expression or movement of the user, the personality that performs a predetermined operation in accordance with the movement of each part of the face, or the user (self) or a partner (speaks) A personality that performs a predetermined expression or action in response to a specific keyword can be considered. For example, a personality that starts dancing once every 10 minutes, a personality that simulates the behavior of the other party's virtual character, a personality that sucks out cigarettes when the conversation is interrupted, a personality that performs like a kabuki actor, There are a variety of personalities, such as personality that can be added, and a question mark in response to words like “why?”
[0038]
As described above, according to the present embodiment, a desired personality is selected from a plurality of personalities prepared in the character server, and the virtual character is given a facial expression or movement different from the actual facial expression or movement of the user. Can be made. Therefore, since it is possible to have a conversation with the other party while enjoying the change of the expression or movement of the virtual character, it is possible to enhance the entertainment of the communication using the virtual character.
[0039]
[Third embodiment]
In the third embodiment, the movement of the virtual character's neck is not directly reflected in the movement of the virtual character, but is moved according to other factors. Note that "the neck moves" refers to an operation in which the head moves three-dimensionally with respect to the body, such as nodding, shaking the head, or shaking the head. In the present embodiment, the virtual character is generated in accordance with the characteristics of a specific keyword or sound emitted (speaked) by the user or the other party, or the characteristics of the video taken by the camera 101 shown in FIG. Move your neck.
[0040]
As described above, the audio processing unit 109 illustrated in FIG. 2 analyzes the voice of the user or the other party, and the video processing unit 103 analyzes the video captured by the camera 101. Therefore, when moving the head of the virtual character according to the keyword or the feature of the sound, the analysis result of the sound processing unit 109 is obtained. When moving the head of the virtual character according to the feature of the video, the analysis result of the video processing unit 103 is obtained. Based on this, the central processing unit 117 instructs the virtual character generation unit 111 to make the virtual character make a predetermined neck movement. In addition, when moving the head of the virtual character randomly, the central processing unit 117 instructs the virtual character generation unit 111 to make the virtual character make a predetermined head movement by using a random number or the like.
[0041]
As described above, in this embodiment, the head of the virtual character is moved independently of the movement of the user's neck based on keywords, voice characteristics, and video characteristics or randomly. It is considered that the movement of the virtual character is more surprising and interesting than the reflection of the movement of the user's neck as it is on the movement of the virtual character. Therefore, it is possible to enhance the entertainment property of the communication using the virtual character.
[0042]
The video processing unit 103, the audio processing unit 109, the virtual character generation unit 111, and the central processing unit 117 included in the videophone terminal according to each of the embodiments described above may operate by executing a program.
[0043]
【The invention's effect】
As described above, according to the virtual character generation device and the virtual character operation control device according to the present invention, it is possible to utilize virtual characters having high entertainment characteristics. Further, according to the videophone terminal of the present invention, it is possible to enhance the entertainment property of communication using virtual characters.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram showing a face image and each feature point. FIG. 2 is a block diagram showing a configuration of a videophone terminal according to an embodiment of the present invention. FIG. And (b) an explanatory diagram showing an example of a virtual character subtly or moderately similar to the user's face. [FIG. 4] an explanatory diagram showing a virtual character similar to the user's face.
101 Camera 103 Video processing unit 105 Microphone 107 Speaker 109 Audio processing unit 111 Virtual character generation unit 113 Display unit 115 Keyboard 117 Central processing unit 119 Radio unit 121 Antenna

Claims (4)

A virtual character generation device that generates a virtual character (hereinafter, referred to as a “virtual character”) based on a human face,
A feature point extracting means for extracting feature points of each part from a video image of the user's face; a feature point of each part extracted by the feature point extracting means; and an average face in which the features of each part are averaged. Feature point difference calculation means for calculating a difference from the feature point of the character,
Virtual character generating means for generating the virtual character of the user by reflecting the difference calculated by the feature point difference calculating means on the character of the average face,
The virtual character generation means, for a predetermined part, changes a difference calculated by the feature point difference calculation means so as to emphasize characteristics of the predetermined part, and reflects the difference on the character of the average face. A virtual character generation device characterized by the following. A virtual character operation control device that gives a virtual character (hereinafter, referred to as a “virtual character”) generated based on a human face a unique character that is not related to the actual expression or movement of the user,
By downloading a desired program from a program storage device storing at least one program for realizing a unique personality and executing the downloaded program, the virtual character can be converted to a predetermined character corresponding to the personality corresponding to the program. A virtual character operation control device for performing an operation. A virtual character operation control device for controlling the movement of the neck of a virtual character (hereinafter, referred to as a “virtual character”) generated based on a human face, comprising: A virtual character operation control device, characterized in that the virtual character performs a predetermined neck movement in accordance with a keyword or a feature of audio or video, or at random, separately from movement. A virtual character generation device according to claim 1 or a virtual character operation control device according to claim 2 or 3, wherein the terminal communicates with another terminal via a network by video and audio of the virtual character. Video phone terminal to do.

Images