JP2001034785A - Virtual transformation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reproduce a three-dimensional model which resembles a subject in virtual environment. SOLUTION: A color camera 16 photographs a full-length picture of the subject and an image processing circuit 18 generates body information on the subject from the photographed full-length picture. A microphone 20, on the other hand, picks up voice of the subject and a voice processing circuit 22 generates property information on the subject from the picked-up voice and body information. Those generated pieces of information are transmitted to a three-dimensional model selecting circuit 26 through a communication line 24. The three-dimensional model selecting circuit 26 selects a three-dimensional model matching the body information and property information out of three- dimensional models recorded in a memory 26a. The selected three-dimensional model is displayed on an image display device 30.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a virtual transformation device,
In particular, for example, the present invention relates to a virtual transformation device that reproduces a person in a virtual environment using a three-dimensional CG (computer graphics) model in real time.

[0002]

2. Description of the Related Art Conventionally, three-dimensional C reproduced in a virtual environment
The G model (avatar) has several types of 3
It was arbitrarily selected from the dimensional CG model.

[0003]

However, in the prior art, for example, while it is possible to make an interesting use such as moving a female character by a male, there is a problem that when a person and a character are mismatched, the sense of immersion is reduced. Was.

Further, it is difficult to reproduce emotions of a person, such as anger, sadness, joy, surprise, fear, etc., in a virtual environment. In particular, sensuous meanings and subtle nuance expressions are expressed in a virtual environment. It could not be reproduced.

[0005] Therefore, a main object of the present invention is to provide a virtual transformation device capable of reproducing a three-dimensional model resembling a figure of a person in a virtual environment.

Another object of the present invention is to provide a virtual transformation device capable of reproducing emotions of a person in a virtual environment.

[0007]

According to the present invention, voice input means for inputting voice of a person, voice feature information generating means for generating voice feature information based on voice, and a plurality of three-dimensional models are stored in advance. A virtual make-up comprising three-dimensional model storage means, model selection means for selecting any one of the plurality of three-dimensional models based on the voice feature information, and reproduction means for reproducing the selected three-dimensional model in a virtual environment. Device.

[0008]

When a voice of a person is input, voice feature information generating means generates voice feature information based on the voice.
On the other hand, the three-dimensional model storage means stores a plurality of three-dimensional models in advance, and the model selection means selects one of the plurality of three-dimensional models based on the voice feature information.
The selected three-dimensional model is reproduced in the virtual environment by the reproduction means. The voice feature information includes, for example, a pitch frequency.

In one aspect of the present invention, when a full-body image of a person is input, physical information creating means creates physical information of the person based on the full-body image. The attribute information creating means creates attribute information of the person based on the physical information and the voice feature information. The model selecting means performs 3 based on the physical information and the attribute information thus created.
Select a dimensional model.

In one embodiment of the present invention, the attribute information indicates whether the person is a man, a woman, or a child, and the plurality of three-dimensional models include a plurality of men having different body shapes, a plurality of women having different body shapes, And children of different body types. The model selecting means selects a three-dimensional model whose body type and attributes match those of the person based on the physical information and the attribute information.

[0011] In another aspect of the present invention, the emotion information creating means creates emotion information of the person based on the voice feature information, and the model deforming means converts the expression of the selected three-dimensional model based on the emotion information. To deform.

In one embodiment of the present invention, the speech feature information includes at least one of a pitch frequency, a dynamic range, a spectral envelope component, power, and a speech rate.
Therefore, the emotion information creating means creates the emotion information based on at least one of these parameters.

In another embodiment of the present invention, when a face image of a person is input, the face information creating means creates face information based on the face image. The model deformation means deforms the expression based on at least one of the emotion information and the face information.

In another embodiment of the present invention, the model deforming means changes the expression by rendering processing such as enlargement / reduction of a three-dimensional model, color change, and partial deformation.

[0015]

According to the present invention, an avatar model to be reproduced in a virtual environment is automatically selected based on a voice of a person or a whole body image of the person. A dimensional model can be reproduced in a virtual environment. Also, since emotion information is created from the voice of the person and the facial expression of the three-dimensional model is deformed based on the emotion information, the emotion of the subject can be reproduced in the virtual environment.

The above objects, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

[0017]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A virtual transformation device 10 of this embodiment shown in FIG.
Is a color camera (hereinafter simply referred to as “camera”) 1
2 and 16. The camera 12 photographs the face of a person (subject) from the front, and processes the photographed face image into an image processing device 14.
To enter. On the other hand, the camera 16 photographs the whole body of the person. Then, the photographed whole body image is input to the image processing device 18.

Both of the image processing devices 14 and 18
It is a computer having the processing capability of a personal computer.
The image processing circuit 14 generates face information including face posture data such as inclination of the face axis and rotation about the face axis and face expression data such as joy and sadness based on the input face image. . On the other hand, based on the input whole-body image, the image processing device 18 generates body type data of the person such as the height (height) of the person, the length of the arms, and the width of the torso, and the body such as the position of the limbs. Generate physical information including posture data. In order to more accurately estimate the posture of the person, it is better to prepare a plurality of color cameras for photographing the whole body.

The virtual transformation device 10 also includes a microphone 20 for capturing the voice of a person. The audio processing circuit 22 includes the microphone 2
0 as well as body information from the image processing circuit 18, and based on this, emotion information (emotions and sorrows) of the person and attribute information of the person (male,
Woman, child).

The outputs of the image processing devices 14 and 18 and the audio processing circuit 22 are supplied to a three-dimensional model selecting device 26 and a three-dimensional model deforming device 28 through a communication line 24. Specifically, the attribute information output from the audio processing device 22 and the physical information output from the image processing device 18 are given to the three-dimensional model selecting device 26, and the face information, image The physical information output from the processing device 18 and the emotion information output from the voice processing device 22 are provided to a three-dimensional model transformation device 26.

The three-dimensional model selecting devices 26 and 3
The dimensional model deforming device 28 is also constituted by a computer having a processing capability of a personal computer. The three-dimensional model selecting device 26 and the three-dimensional model deforming device 28 may be integrated in one computer.

A plurality of three-dimensional models are stored in a memory 26a provided in the three-dimensional model selecting device 26 in advance. Furthermore, these three-dimensional models are composed of a plurality of adult men with different body types, a plurality of adult women with different body types,
And multiple children of different body types. The three-dimensional model selection device 26 specifies a three-dimensional model that matches the input attribute information and physical information (particularly, body type data) from among the plurality of three-dimensional models described above, and converts the character data of the specified model into three-dimensional data. Output to the model deformation device 28.

The three-dimensional model deformation device 28 corrects the character data input from the three-dimensional model selection device 26 based on face information and emotion information. Specifically, the emotion of the person is estimated based on the face information (especially facial expression data) and the emotion information, and the character data is corrected so that if the person laughs, the three-dimensional model also laughs. Further, the posture of the person is estimated based on the face information (particularly, the posture data of the face) and the body information (particularly, the posture data of the body).
The character data is corrected so that the dimensional model also faces sideways. Then, the corrected character data is output to the image display device 30.

The image display device 30 combines a three-dimensional model with a virtual environment based on the input character data. Thus, a three-dimensional model similar to a person is reproduced in the virtual environment, and the posture and expression of the three-dimensional model change according to the posture and expression of the person. The alter ego image of the subject reproduced in the virtual environment in this way is called “avatar”. The image display device 30 is, for example, 3
It is configured by a computer equipped with a D graphics accelerator.

Color camera 1 for photographing a person's face
2 operates specifically as shown in FIG. That is, first, in step S1, the face image of the subject is fetched, and then in step S3, the skin color area of this face image is set to "0", and the areas other than the skin color area are set to "1".
Value. As a result, the input color face image is converted to a monochrome binary image. In step S5, the eye area is tracked. That is, a template that passes through the center of the pupil is created in the eye region obtained by the preprocessing, and the position of the eye is tracked by so-called template matching. Thereafter, in step S7, the position of the mouth is estimated based on the relative relationship between the eyes and the mouth obtained by the preprocessing.

In step S9, the facial expression is recognized. Recognition of facial expressions is performed by the inventor of the present invention in the IEICE Transactions
D-II, Vol.J80-D-II, No.6, pp.1547-1554 (June 1997
Monday) ”. This is done in 2
The face expression is recognized by converting an image into a spatial frequency domain by a dimensional discrete cosine transform, and capturing a power change in each frequency band corresponding to a change in a face part. In other words, by detecting changes in the shape of the eyes and mouth,
Recognize facial expressions. As a result, it is possible to perform high-speed detection, which is strong against displacement of a region and deformation of a part.

In step S11, the inclination of the face axis and the rotation about the face axis are detected. The angle formed by the line connecting both eyes with respect to the horizontal axis is defined as the inclination of the face. Further, the rotation angle around the face axis is determined from the size of the interval between the right eye region and the left eye region. The smaller the interval, the more the face is considered to be rotating.

In step S13, the thus obtained face information, that is, face information including face expression data and face posture data, is transmitted to the three-dimensional model deformation device 28 through the communication line 24.

On the other hand, an image processing device 18 connected to a color camera 16 for taking a full-body image of a person operates as shown in FIG. That is, first, in step S21, a color image including the whole body image of the subject is captured from the camera 16,
In step S23, a whole body image is extracted by the background subtraction method. The background subtraction method is a method of extracting only a person image by subtracting an image (background image) other than a person from an image including a person. Note that the background image has been obtained in advance by preprocessing.

In step S25, the person image (whole body image) is set to "1", the background image is set to "0", and the input image is set to 2
Value. That is, a binary image serving as a silhouette image of a person is extracted. Subsequently, in step S27, the outline of the person is detected by performing a morphological process on the binary image, and in step S29, feature points of the outline are determined. In step S31, the posture of the body is estimated based on the determined characteristic points, and in steps S33 to S37, the height of the back, the length of the arms, and the width of the body are calculated based on the same characteristic points. .

When the body information including the posture data and the body type data of the body is obtained in this way, step S39
Transmits this physical information to the three-dimensional model selection device 26.

The voice processing device 22 operates as shown in FIGS. First, in step S41, a speech voice of a person is input from the microphone 20. Next, in step S43, the uttered voice is subjected to frame division, that is, the uttered voice is divided into a fixed time length, and an FFT (Fast Fourier Transform)
Analyze the frequency by In step S45, the dynamic range of the uttered voice is calculated based on the analysis result in step S43. Specifically, the audio waveform is arranged, and the average value is set as the intensity of the sound, that is, the volume.

In step S47, the calculated dynamic range is compared with a predetermined threshold to determine whether or not speech is being made. If the speech is not being made, step S
At 51, the data of "No utterance" is converted to the three-dimensional model deformation device 2
8 On the other hand, if the user is speaking, the emotion of the person is estimated from the uttered voice in step S49, and the attribute of the person is estimated based on the uttered voice and the height information fetched from the image processing circuit 18 in step S53. I do. Then, in step S51, the emotion information of the person is transmitted to the three-dimensional model transformation device 28, and the attribute information of the person is transmitted to the three-dimensional model selection device 26.

The estimation of the emotion in step S49 is performed in the manner shown in FIG. First, in step S61, a linear prediction coefficient (LPC; Linear Prediction Coefficien) is used.
Calculate t). Since the linear prediction coefficient efficiently represents the properties of the speech waveform and spectrum, it is used as a feature amount for phoneme identification and speaker identification. The linear prediction method is a method for minimizing the mean square of a prediction error represented by linear weighting, and is widely used in audio signal processing because a spectrum can be estimated from short-time data. In step S63, a predicted waveform is generated based on the calculated linear prediction coefficient. From the predicted waveform, the vocal tract resonance characteristics are determined.

In the following step S65, a spectral envelope component and a fundamental frequency component are separately extracted from the predicted waveform of the audio signal by cepstrum calculation. The cepstrum method is a method that uses a parameter (cepstrum) obtained by performing a Fourier transform on an audio signal and performing an inverse Fourier transform on the logarithm of the absolute value of the obtained amplitude spectrum. The variable is called quefrency and has a time dimension. In the cepstrum analysis, a spectral envelope component and a fundamental frequency component can be separately extracted by setting a threshold value for the quefrency.

In step S67, a residual waveform is calculated from the original waveform of the audio signal input from the microphone 20 and the predicted waveform generated in step S63. This residual waveform is
This is considered to represent a sound source component obtained by removing a frequency component expressing a phoneme from the original waveform. In step S69,
The pitch frequency is calculated based on such a residual waveform. That is, the correlation coefficient is obtained from the residual waveform, the maximum value of the correlation coefficient is calculated, and the reciprocal of the coefficient that gives the maximum value is obtained,
Output as the pitch frequency of the target frame.

In step S71, an average power is calculated for each audio signal divided into frames. In step S73, the length of uttering one phoneme, that is, the utterance speed is measured. Specifically, the time length of a portion (segmentation) determined as one phoneme is measured. It can be determined that the shorter the time length, the faster the utterance speed.

In step S71, the dynamic range calculated in step S45, the spectral envelope component obtained in step S65, the pitch frequency obtained in step S69, the power obtained in step S71, and the utterance measured in step S73. Using the speed, the emotion of the subject is estimated by a neural network. The dynamic range, the spectral envelope component, the pitch frequency, the power, and the utterance speed are defined as voice feature information.

In step S53, the processing shown in FIG. 6 is specifically performed. First, in step S81, the pitch frequency is compared with a predetermined threshold. If the pitch frequency is lower than the threshold, the subject is estimated to be an adult male in step S85. On the other hand, if the pitch frequency is equal to or higher than the threshold, the process proceeds from step S81 to step S83, and the height data included in the physical information acquired from the image processing device 18 is compared with a predetermined threshold. If the height of the subject is lower than the threshold, the subject is estimated to be a child in step S89. On the other hand, if the height of the subject is equal to or greater than the threshold, the subject is estimated to be an adult woman in step S87.

The three-dimensional model selecting device 26 and the three-dimensional model deforming device 28 operate as shown in FIG. First, in step S91, attribute information and physical information are provided to the three-dimensional model selecting device 26, and emotion information and face information are provided to the three-dimensional model deforming device 26. Step S
At 93, the three-dimensional model selection device 26 reads out from the memory 26a character data of a three-dimensional model (avatar) matching the attribute information and the body information (body type data). The read avatar character data is provided to the three-dimensional model transformation device 28.

In step S95, the three-dimensional model transformation device 28 specifies the subject's emotion based on the emotion information and the face information (face expression data), and performs rendering processing on the avatar. As a result, the facial expression of the avatar changes. Specifically, if the identified emotion indicates "anger,
Give the avatar's face a reddish shade. If the emotion shows "sadness," weep in the avatar's eyes.
For “joy”, the avatar's eyes are shaped like “he”, and for “surprise”, a surprise mark is placed above the avatar's head. In addition, “fear” makes the avatar's face bluish and slender. If the subject does not speak, the emotion estimation processing is not performed. At that time, the rendering processing is performed based only on the face information (face expression data).

In step S95, the posture of the person is estimated based on the face information (face posture data) and the body information (body posture data), and the avatar posture is changed according to the estimation result.

In step S97, the avatar character data that has been subjected to the rendering process and the posture changing process is output to the image display device 30. As a result, an avatar similar to the subject is reproduced in three dimensions in the virtual environment, and the expression and posture of the avatar change in real time in response to the image and voice of the subject.

According to this embodiment, the characteristics of the subject are detected from the voice characteristic information of the subject obtained by the voice processing or the physical information of the subject obtained by the image processing.
Since a three-dimensional model having the same feature is selected from a plurality of three-dimensional models, the three-dimensional model reproduced in the virtual environment is selected.
A dimensional model or avatar can be made to resemble a subject. In addition, since the subject's emotion is specified from the subject's face information obtained by the above-described voice feature information and image processing and the avatar is deformed based on the subject's emotion, the subject's emotion can be reproduced in the virtual environment. .

In this embodiment, only one color camera is used to capture the whole body image of the subject. However, in order to deform the avatar according to the movement of the whole body of the subject, a plurality of color cameras are prepared. The whole body of the subject may be photographed from different directions. In this case, each captured color image is processed as follows. First, a color image is converted into a binary image, and feature points are detected from the binary image. The three-dimensional model deformation device deforms the shape of the avatar based on the feature points detected in this way.

When the feature point cannot be detected or predicted from the binary image because the body and hand of the subject overlap the camera, a two-dimensional skin color region is extracted from the color image, and the extracted skin color region is extracted. Is three-dimensionally restored. Then, a portion having the largest restored volume is set as a hand region, and its center of gravity is set as a feature point.

Further, if each color camera is mounted on an electric pan head and the pan head is controlled based on the image of the subject, the color camera can follow the movement of the subject.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a virtual transformation device according to an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of the audio processing device.

FIG. 3 is a flowchart illustrating a method for estimating emotions. .

FIG. 4 is a flowchart illustrating a method of detecting a human character.

FIG. 5 is a flowchart illustrating an operation of the image processing apparatus.

FIG. 6 is a flowchart for explaining selection and deformation of a three-dimensional model.

FIG. 7 is an illustrative view showing a neural network;

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 ... Virtual transformation apparatus 12 ... Camera 14 ... Image processing apparatus 16 ... Microphone 18 ... Audio processing apparatus 20 ... 3D model selection apparatus 22 ... 3D model transformation apparatus 24 ... Image display apparatus

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Atsushi Atsushi 5 Osanaya, 5 Seira-cho, Seika-cho, Soraku-gun, Kyoto F-term (reference) 5B050 BA07 BA08 BA09 CA07 EA19 FA02 5D015 AA01 KK01 5L096 AA02 AA03 AA07 CA02 FA02 FA67 GA38 9A001 DD12 GG07 HH16 HH29 KK60

Claims (8)

[Claims] A voice input unit configured to input voice of a person; a voice feature information generating unit configured to generate voice feature information based on the voice; a three-dimensional model storage unit storing a plurality of three-dimensional models in advance; A virtual transformation comprising: a model selecting means for selecting any one of the plurality of three-dimensional models based on the voice feature information; and a reproducing means for reproducing the three-dimensional model selected by the model selecting means in a virtual environment. apparatus. 2. A whole-body image inputting means for inputting a whole-body image of the person, a physical-information creating means for creating physical information of the person based on the whole-body image, and a physical information creating means for creating physical information of the person based on the whole body image. The virtual transformation apparatus according to claim 1, further comprising attribute information creating means for creating attribute information of a person, wherein said model selecting means selects a three-dimensional model based on said physical information and said attribute information. 3. The attribute information indicates whether the person is a man, a woman, or a child, and the plurality of three-dimensional models include a plurality of men having different body types, a plurality of women having different body types, and a plurality of body types. The virtual transformation device according to claim 2, comprising a plurality of different children, wherein the model selecting means selects a three-dimensional model whose body type and attribute match the person based on the physical information and the attribute information. 4. An emotion information creating means for creating emotion information of the person based on the voice feature information, and a model deformation for deforming an expression of a three-dimensional model selected by the model selection means based on the emotion information. The virtual makeover device of claim 1, further comprising means. 5. The virtual transformation device according to claim 4, wherein said voice feature information includes at least one of a pitch frequency, a dynamic range, a spectral envelope component, power, and a speech rate. 6. A face image input means for inputting a face image of the person, and a face information generating means for generating face information based on the face image, wherein the model deformation means includes the emotion information and the face information. The virtual transformation device according to claim 5, wherein the expression is deformed based on at least one of the following. 7. The virtual transformation apparatus according to claim 4, wherein said model transformation means changes said expression by a rendering process. 8. The rendering process includes enlarging / reducing the color of the three-dimensional model, changing a color, and partially deforming the three-dimensional model.
Virtual transformation device as described.