JP2005057431A - Video phone terminal apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a video phone terminal apparatus which enhances communication effect while taking speaker's privacy into consideration by using a portrait of a speaker himself or herself as a substitute image. <P>SOLUTION: A voice signal inputted from a voice input part 1 is encoded by a voice encoding part 2 and transmitted to an opposite-side part 2 through a communication I/F part 4. The inputted voice signal is sent to an expression data generation part 3 as well at the same time to generate a signal for adding expression to a face image. A basic face data generation part 6, on the other hand, generates basic face data showing sizes, positions, etc., of respective parts of the face, such as the outline, eyes, and mouth, according to operation at a user operation input part 5. A portrait generation part 7 combines the basic face data and expression data together to generate a portrait image of the speaker in the form of a moving picture. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a videophone terminal device that generates a portrait of a speaker in accordance with the content of a speaker's voice and changes in emotion and transmits it to the other party.

  Many so-called videophone devices, which are equipped with a camera and a display on a telephone and can communicate by image, have already been commercialized. However, there is a persistent resistance from the viewpoint of privacy protection about sending the video of the speaker to an unspecified destination, so that a proxy image stored in advance is sent instead of the actual video of the speaker. A videophone device has been proposed (see, for example, Patent Document 1).

On the other hand, the use of incarnations (avatars) is considered as a communication device for entertainment rather than a device for communication between speakers like a normal videophone. There has been proposed a device in which an imaginary character is interposed instead of the face image (see, for example, Patent Document 2).
JP 2003-37826 A JP 2003-16475 A

  However, although the above-mentioned prior art can reflect the speaker's preference by selecting a proxy image or a fictitious character from a plurality of options, it is not related to the original speaker image at all. Although protection and entertainment properties can be improved, it has been difficult to obtain the same effect as communication using original images.

  Also, when transmitting character information, etc., the character information selected from the stored data and facial expression data are transmitted separately and decoded at the receiving terminal, so both the transmitting side and the receiving side transmit this information. The terminal must be compatible with the system, and connectivity with a normal videophone device has not been ensured.

  The present invention was devised to solve the above-described problems, and provides a videophone terminal device that enhances the communication effect while considering the privacy of the speaker by using the person's portrait as a proxy image. The purpose is to do.

  In order to achieve the above object, the videophone terminal device according to the present invention generates basic face data indicating a facial feature that is a source of a portrait of a reader based on the position and size of each part of a speaker's face. Face data generation means; voice input means for inputting a voice of a speaker; a first average frequency indicating an average of fundamental frequencies from past to present in a voice signal input to the voice input means; Expression data generation means for determining a change in emotion based on a combination with a second average frequency indicating an average of the fundamental frequency in a period shorter than the average frequency, and generating expression data of the speaker's face, and the basic face Caricature generating means for generating caricature image information by changing the angle of a specific part of the face of the data corresponding to the emotion information of the expression data, and sending voice information and video information to the communication line A communication interface unit, is characterized by sending the portrait image in place of the real image of the speaker during a call.

  According to the present invention, a portrait picture created by a speaker can be used as a substitute image for a videophone terminal instead of the actual video of the speaker. Therefore, communication that fully utilizes the video while protecting privacy. You can make calls with unspecified parties without worry.

  Further, since a portrait with a certain expression close to that of the actual video can be transmitted as an image, it is possible to provide a terminal in which actual video transmission means such as camera input are completely omitted.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a videophone terminal device to which the present invention is applied.
Although not shown, the voice input unit 1 is connected to a microphone or the like as in a normal videophone terminal, and performs voice input using the microphone. Here, the input audio signal is encoded by the audio encoding unit 2 and transmitted to the counterpart terminal via the communication I / F (interface) unit 4. Also, the input audio signal is simultaneously sent to the facial expression data generation unit 3, where a signal for adding a facial expression to the facial image is generated. This facial expression data creates mouth movements that reflect the magnitude of the audio signal, and analyzes the movement of the fundamental frequency of the audio signal as described below to quantify the degree of emotions that express emotions. Instruct the mouth angle.

  On the other hand, the basic face data generation unit 6 generates data serving as a base of the face image to be transmitted by operating the user operation input unit 5. This data is numerical data indicating the size and position of each part of the face such as contour, eyes, mouth, etc., and the user creates desired caricature data by operating the operation input unit 5 prior to calling. be able to.

  The portrait generation unit 7 combines the basic face data and the expression data to generate a portrait image of the speaker as a moving image. The generated moving image information is encoded by the image encoding unit 8, and the communication I / F unit 4 Is sent to the other terminal via.

  The basic face data generation unit 6 uses a numerical value parameter that represents facial features as data for generating a portrait. FIG. 2 shows a numerical configuration example of the basic face data. In FIG. 2, the parts constituting the face are contours, hair, eyebrows, eyes, mouth, and numerically expressed as the size (width, height or thickness) and position (X, Y coordinates from the face center) of each part. Yes. For example, the mouth is located below the center of the face by an amount indicated by my, the thickness is mh, and the width is mw. Similarly, the positions and sizes of the eyes and eyebrows are indicated by numerical values. The cheek line is splined and the parameter fr indicates where the relay point is.

Although not shown in FIG. 2, the hairstyle has a variety of shapes so that it is difficult to make a simple numerical value. Therefore, a method of selecting from a plurality of hairstyle candidates determined in advance, or a plurality of hairstyles The hairstyle is determined by a combination of selection from basic hairstyle candidates and numerical data such as the length of each part, and conversion of the hairstyle outline itself into vector data.

  Also, color data such as face color and hair color used for drawing can be used as additional numerical data.

  As described above, in order to create this basic face data, the speaker adjusts each of the above parameters by operating the operation input unit 5 while monitoring the generated face image, and creates his own portrait. Can do. For example, each numerical parameter may be created while monitoring the output portrait in real time by operating a scroll bar on the GUI.

  By making such data, various faces can be expressed with a very small amount of data, and synthesis with facial expression data becomes very simple. For example, if the amplitude of the audio signal corresponds to how to open the mouth, if the emotional expression is “pleasure”, both ends of the mouth are curved upward, and if “angry”, both ends of the eyebrows are raised It is possible to generate a face image with a facial expression by simply changing coordinate information such as.

  Next, the operation of the expression data generation unit 3 and the reflection of the expression data on the face image will be described. The expression data includes the reflection of the facial image on the shape of the mouth portion of the face image and the reflection of the emotional information contained in the sound signal on the shape of not only the mouth but also the eyes and eyebrows. .

  In the simplest method, the voice signal itself is reflected in the mouth shape by linking the amplitude of the signal with the way the mouth is opened. For more precise control, pay attention to the vowels of the audio signal, analyze the overtone structure (formant), and apply a mouth shape that is predefined for the current formant. You can also coordinate movements more naturally.

  The creation of a face image based on emotion determination is performed by analyzing changes in the fundamental frequency component of an input audio signal. FIG. 3 shows an example of a change in the fundamental frequency of the input audio signal. First, the long-term average M of the fundamental frequency is calculated, and the utterance content is divided for each sentence, and the average fundamental frequency S of each sentence is calculated. It is determined whether the average frequency S of each sentence is changing at a high frequency or a low frequency with respect to the long-term average M. The long-term average M is obtained by integrating not only information on the current call but also data on the past call.

  Further, each sentence is further divided into front and rear, the average basic frequency of the first half is H1, the average basic frequency of the second half is H2, and H1 and H2 are compared. As a result, information on whether the sentence average S is higher or lower than the long-term average M and information on whether the average frequency of the sentence is upward or downward are obtained. From this information, the application is applied to each state of “feeling crying” (emotion information).

This method is illustrated in FIG. If we use the symbol for the fundamental frequency mentioned above,
X = SM
Y = H2-H1
The values of X and Y are calculated by the following equation, and the X and Y are applied to the (X, Y) coordinates in FIG. Further, in order to express the degree of emotions of joy and crying, for example, “joy”, each region of joy and crying in FIG. It has a distribution (in the Z-axis direction), and can output the concentration at the point indicated by the coordinates (X, Y) as the degree of emotion (feeling of joy and crying).

  However, this method is not always accurate, and fitting may be performed in consideration of fine movement of the frequency and changes in power in each frequency band in order to improve accuracy.

  Next, the emotion information of “sheerful crying” for each sentence obtained in FIG. 4 is sent from the facial expression data generation unit 3 to the portrait generation unit 7 as facial expression data together with mouth shape data based on the voice signal itself. Based on this emotion information, the portrait generator 7 corrects each part of the face of the basic face data supplied from the basic face data generator 6. Further, the shape of how to open the mouth of the basic face data is determined based on the mouth shape data based on the audio signal itself.

FIG. 5 shows an example in which the angle between the eyebrows and the mouth is changed in accordance with the emotion information of “happy emotions”.

  The expression of “Raku” is in a state where both the eyebrows and the mouth are horizontal, and there is no angle. Based on the emotion information in the facial expression data sent from the facial expression data generation unit 3 based on the emotion of “Easy”, the caricature generation unit 7 selects “feeling”, “anger”, “sorrow”. ", The expression is changed to" crying ". If the emotion information sent from the facial expression data generation unit 3 is “Easy”, the angle of the eyebrows and the mouth is not changed, and the state is maintained as it is. The change of the facial expression will be described with reference to FIG. 5. In the case of “joy”, the center of the eyebrows is raised to make a letter-shaped, and the mouth is raised at both ends. In the case of “anger”, raise the ends of the eyebrows and lower the ends of the mouth. In the case of “sorrow”, the ends of the eyebrows are lowered and both ends of the mouth are also lowered. In the case of “crying”, more tears are added to the expression of “sorrow”.

  The angle of the eyebrows and mouth is set based on the above-described data indicating the degree of emotion (density distribution data in FIG. 4).

  In this way, the portrait generator 7 combines the basic face data created in advance, the emotion information sent for each sentence, and the mouth shape data based on the voice signal itself, thereby providing the emotion and pronunciation of the speaker. Since the caricature corresponding to can be continuously created, the videophone terminal can be used using the moving picture of the caricature.

It is a figure which shows the structure of the video telephone terminal device of this invention. It is a figure which shows the structure of basic face data. It is a figure which shows the change of the fundamental frequency of an audio | voice signal. It is a figure which shows a response | compatibility with the change of the fundamental frequency of an audio | voice signal, and emotion information. It is a figure which shows correction to each part of the face by expression data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Voice input part 2 Voice encoding part 3 Expression data generation part 4 Communication I / F part 5 User operation input part 6 Basic face data generation part 7 Caricature generation part 8 Image encoding part

Claims (1)

Basic face data generating means for generating basic face data indicating the characteristics of the face that is the basis of the caricature of the reader based on the position and size of each part of the speaker's face;
Voice input means for inputting the voice of the speaker;
A first average frequency indicating an average of fundamental frequencies from the past to the present in an audio signal input to the audio input means, and a second average frequency indicating an average of fundamental frequencies in a period shorter than the first average frequency A facial expression data generating means for determining a change in emotion based on the combination of and generating facial expression data of the speaker;
Caricature generating means for generating image information of a caricature by changing an angle of a specific part of the face of the basic face data in correspondence with emotion information of the facial expression data;
Communication interface means for sending audio information and video information to a communication line,
A videophone terminal device that transmits the portrait image instead of an actual video of a speaker during a call.

Images