JP2670663B2 - Real-time image recognition and synthesis device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-time recognition / synthesis apparatus for the movement and posture of an image of a person or the like, and more particularly, to an image model on the transmission side formed by color texture mapping on a three-dimensional wire frame model. This image model is placed in the generated virtual space using computer graphics technology, and is displayed on the stereoscopic display on the receiving side.
The present invention relates to a real-time recognition / synthesis apparatus for an image which reproduces the movement of an image and stereoscopically displays it.

[0002]

2. Description of the Related Art A conventional real-time image recognition / synthesis apparatus is disclosed in, for example, IEICE Technical Report p. 23-28, "3 in the presence communication conference"
Real-Time Display of Dimensional Images ". Hereinafter, the apparatus will be described.

FIG. 8 is a block diagram of a conventional real-time recognition synthesizing apparatus, and FIG. 9 is a concrete example of the facial expression detector 103 of FIG. The arrows in the figure indicate the direction in which data is sent.

[0004] The real-time recognition / synthesis apparatus is mainly composed of a system 11 on the transmitting side and a system 13 on the receiving side. The transmitting-side system 11 includes a body motion detector 101 that detects a body motion of the transmitting-side person 3 in the real space 1;
And a facial expression detector 103 for detecting facial expressions. As the body movement detector 101, for example, a data glove of VPL, a magnetic sensor of Polhemus, or the like is used. The data obtained by the body motion detector 101 is used as the body motion parameter 21 as the system 1 on the receiving side.
3 is sent to the image synthesis unit 115.

Further, the facial expression detector 103 is provided with a marker 20 on the skin surface above the facial muscles of the face 209 as shown in FIG.
3 is an apparatus which captures an image 205 with the camera 201 fixed to the head and fixed to the head. The movement amount of the marker 203 in the image 205 is tracked and sent to the image synthesizing means 115 in the receiving system 13 as the motion parameter 23 of the facial expression.

On the other hand, the receiving side system 13 is provided with a person model database 111 in which the person 3 on the transmitting side is modeled by a wire frame model and a color texture in advance. The three-dimensional wire frame model is obtained by approximating the surface of a human body with triangular patches, and the color texture is color information of the surface of the human body.

[0007] In the image synthesizing section 115, the person model data composed of the three-dimensional wire frame model and the color texture in the person model database 111,
The transmitted body motion parameters 21 and facial expression motion parameters 23 are combined. In the synthesis, the coordinates of each vertex of the three-dimensional wireframe model of the human model data are
This is performed by moving the body according to the body movement parameter 21 and the facial expression movement parameter 23.

[0008] The synthesized human image 5 is arranged in a virtual space 7 generated using computer graphics technology. The virtual space 7 including the human image 5 is displayed on the stereoscopic display 9 on the receiving side.

[0009]

However, in such a real-time image recognition / synthesis apparatus, since all the movements of the image are transmitted as motion parameters, there is a problem that a subtle change in the image cannot be recognized and synthesized. . In particular, when transmitting facial expressions of people, it is difficult to recognize and synthesize subtle changes in facial expressions around the eyes, which play an important role in communication, only by measuring the displacement of markers attached to the face. is there.

To solve this problem, it is conceivable to use an image processing technique such as edge detection. This makes it possible to increase the data to be measured and to recognize and combine subtle changes in the image to some extent, but the amount of image processing becomes enormous, requiring an expensive image processing device and processing information. Increases the processing speed,
Real-time recognition may not be possible.

[0011] Therefore, a main object of the present invention is to reproduce a subtle change in an image, and to process the amount of communication data and the amount of processing information necessary for measurement on the transmitting side and the amount of processing information required for image synthesis on the receiving side. An object of the present invention is to provide a real-time image recognition / synthesizing apparatus having a sufficiently small amount and suitable for practical use.

[0012]

In order to achieve the above-mentioned object, an image real-time recognition / synthesis apparatus according to the present invention provides an image model data generating apparatus which performs color texture mapping of an image model on a transmitting side to a three-dimensional wire frame model. A real-time image recognition / synthesis device that arranges image model data in a virtual space generated by using computer graphics technology, and reproduces the motion of the image on a receiving-side three-dimensional display. And storage means for storing image model data, and parameter detection means for detecting a motion parameter based on the operation of the image model on the transmission side,
Image data detection means for detecting at least a part of the image model on the transmission side to detect the image data, the stored image model data, and the detected motion parameter are combined to generate the first image data. First synthesizing means for generating, the detected image data, and partial image model data corresponding to a captured image model portion of the stored image model data to synthesize first intermediate image data. First to adjust
Adjusting means, a proximity parameter that is close to the outer edge of the captured image model portion of the detected motion parameters, and a peripheral image model corresponding to at least the periphery of the captured image model portion of the image model data. Second adjusting means for synthesizing the data and adjusting the second intermediate image data, and synthesizing the adjusted first intermediate image data and the second intermediate image data to generate the second image data. A second combining unit that combines the generated first image data and the generated second image data to generate third image data; and a third combining unit that generates the third image data. Image reproduction means for reproducing the movement of the image model on the transmission side based on the third image data.

[0013]

The image real-time recognition / synthesis apparatus according to the present invention has the following functions.

First, the motion parameters of the image sent from the transmitting side and the image model data stored in advance on the receiving side are combined to form the first image data. Particularly, a portion that requires subtle image reproduction is imaged and separately transmitted as image data from the transmission side. The captured image data and the image model data corresponding to the captured portion are combined to form the first intermediate image data. Of the detected motion parameters, the parameter close to the outer edge of the imaged portion and the image model data corresponding to the periphery of the imaged portion are combined to form second intermediate image data. The first intermediate image data and the second intermediate image data are combined into the second image data. Then, the first image data and the second image data are combined into the third image data. The third image data is reproduced on the display on the receiving side in real time.

[0015]

1 is a block diagram showing a real-time recognition / synthesizing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram showing an expression detector 103 shown in FIG.
FIG. 3 is a diagram showing a wireframe model of a person's head included in the person model database 111 of FIG. 1, and FIG. 4 is a diagram for explaining a process performed by the model deformation adjusting unit of FIG. FIG.

The configuration and operation of the real-time recognition and synthesis apparatus of one embodiment of the present invention will be described with reference to FIGS. 1 to 4.

The arrow in FIG. 1 indicates the direction in which data is sent. The real-time recognition device mainly includes a transmitting system 11 and a receiving system 13.

The system 11 on the transmission side is a body motion detector 10 for detecting the movement of the body of the person 3 on the transmission side in the real space 1.
1 and a facial expression detector 103 that detects a facial expression. As the body motion detector 101, for example, VPL
The company's data gloves and Polhemus magnetic sensors are used. The data obtained by the body motion detector 101 is
The body movement parameter 21 is transmitted to the receiving system 13.

The display detector 103, as shown in FIG.
This is an apparatus for photographing facial expressions with a camera 201 fixed to the head. The camera 201 images the face 209 and the marker 203 fixed on the skin surface above the facial expression muscle of the face 209. Marker 2 in the captured image data 205
03, the moving direction and the moving distance of the face marker are tracked.
The data around the eye that is sent to the receiving system 13 is sent to the receiving system 13 as image data 25.

The receiving side system 13 is provided with a person model database 111 in which the person 3 on the transmitting side is modeled in advance by a three-dimensional wire frame model and a color texture.

The three-dimensional wire frame model is an approximation of the surface of the human body with triangular patches, and the color texture is color information of the human body surface.

The body motion parameter 21 and the face marker motion parameter 23 are sent to the first image synthesizing unit 105,
Therefore, 3 extracted from the person model database 111
Dimensional wireframe model and color texture 27
Is synthesized with Specifically, the synthesis is performed by deforming the vertex coordinates of the three-dimensional wire frame model in accordance with the body movement parameter 21 and the face marker movement parameter 23. As a result, the shape change due to the movement of the human body or the movement of the face marker is reproduced.

The image data 25, the movement parameter 23 of the face marker, and the three-dimensional wire frame model 29 in the human model database 111 are used as the model deformation adjusting unit 11.
It is sent to 7. The processing performed by the model deformation adjusting unit 117 will be described below.

In the three-dimensional wireframe model 29 in the person model database 111, the eye portion 45 and the portion 46 around the eye have a higher node density than the other portions 47 as shown in FIG. The patch is set to be fine. This is because the eye portion 45 and the portion 46 around the eye need to reproduce a particularly subtle change. The image data 25 is combined into an eye portion 45 and a portion 46 around the eyes. Model deformation adjustment unit 117
In, the vertex coordinates of the wire frame of the portion 46 around the eyes are moved based on the information from the motion parameter 23 of the face marker transmitted from the expression detector, and the shape change is reproduced.

This change in shape is caused by the fact that the markers 203a to 203d fixed to the face are moved as shown by arrows 204a to 204d as shown in FIG. The movement of each vertex coordinate of the model is represented by calculation by being performed as indicated by arrows 213a to 213d.

In the second image synthesizing unit 107, the wire frame model 59 of the part around the eye where the shape change is reproduced, and the image data 57 synthesized with the eye part 45 and the part 46 around the eye. Synthesized. Image synthesis is done by ONYX workstation of Silicon Graphics
It is performed by The image data 31 and 33 obtained by the first image synthesizing unit 105 and the second image synthesizing unit 107 are sent to the third image synthesizing unit 109 to be synthesized. The synthesized human image 5 is arranged in a virtual space 7 generated using computer graphics technology. The virtual space 7 including the human image 5 is displayed on the stereoscopic display 9 on the receiving side.

The color tone data 41 of the image synthesized by the first image synthesizing unit and the color tone data 43 of the image synthesized by the second image synthesizing unit are sent to a color tone inspection unit 113 and compared there. When the difference is larger than the predetermined value, the difference information 39 is fed back to the person model database 111, and the color texture in the person model database 111 is corrected so that the difference becomes small.

In a conference system or the like, the lighting conditions and the like do not change very often, so this correction process may be performed once every several seconds to several minutes. Since several to several tens of image data 25 are sent per second, correcting the color texture in the human model database 111 is more effective than correcting the image data 25 in reducing the amount of processing information. It is advantageous.

FIGS. 5 to 7 show the change in color tone at the boundary between the image synthesized by the first image synthesizing unit 105 and the image synthesized by the second image synthesizing unit 107 in the embodiment of FIG. It is a figure for demonstrating the process performed in order to make it inconspicuous.

In FIG. 5, the three-dimensional wire frame model 29 around the eyes extracted from the human model data 207 in the human model database, the movement parameter 23 of the face marker, and the image data 25 are sent to the model deformation adjusting unit 117. Sent. As described above, the model deformation adjustment unit 1
At 17, the first intermediate image data 59 in which the three-dimensional wireframe model of the part around the eyes and the motion parameter 23 of the face marker are combined, the three-dimensional wireframe model of the eyes and the part around the eyes Image data 25
The second intermediate image data 57 in which is combined is generated.
The generated intermediate image data 57 and 59 are sent to the second image synthesizing unit 107 to be synthesized and become the second image data 33. On the other hand, the three-dimensional wireframe model and the color texture 27 of the part excluding the area around the eyes extracted from the person model data 207 are sent to the first image synthesis unit 105, and the body motion parameter 21 and the face marker motion parameter 23 are sent. And is combined into the first image data 31. The first image data 31 and the second image data 33
Is sent to the third image synthesizing unit 109, and the person image 5 is synthesized.

At this time, in order to make the boundary portion between the first image data and the second image data inconspicuous, each of them includes data corresponding to the color tone adjusting portion 504. The color tone adjustment portion 504 is an area where the area of the first image data 31 and the area of the second image data 33 overlap each other as shown in FIG. The tone adjustment portion 504 may be the same as or different from the portion 46 around the eyes of FIG.

In FIG. 6, the color tone data 600 at the boundary point A with the color tone adjusting portion 504 of the first image data 31,
Assuming that the color tone data 602 at the boundary point B with the color tone adjustment portion 504 of the second image data 33 has a color tone level as shown in FIG.
The color tone data 600 of the image data 31 is linearly reduced in the area of the color tone adjustment portion 504 as the color data 600 progresses toward the region of the second image data 33.
Is set to 0 at the position of the boundary point B on the area side of the second image data 33 in the area. This linear decrease in tone data is 606
Shown in

Similarly, the color tone data 602 of the second image data 33 is linearly reduced in the area of the color tone adjusting portion 504 as it advances toward the area of the first image data 31, and the color tone adjustment is performed. First image data 3 in the area of the portion 504
It is set to 0 at the position of the boundary point A on the side of the area of 1. This reduction in linear tone data is shown at 608.

The color tone data in the color tone adjusting portion 504 is obtained by adding the color tone data 606 of the first image data and the color tone data 608 of the second image data. The data is assumed to be 604.
With this processing, the boundary between the first image data 31 and the second image data 33 can be made inconspicuous.

As shown in FIG. 7, the color tone of the first image data 31 and the second image data 33 may differ depending on the location. In that case, the color tone adjustment process shown in FIG. 6 may be performed for each part.

In this embodiment, the part to be transmitted as image data is set to the eye part, but it may be another part such as the mouth.

In this embodiment, a human image is used as an example, but an animal or the like can be used.

Further, in this embodiment, the shape of the wire frame model around the eyes is changed by the model deformation adjusting unit, but the shape of the wire frame model around the eyes may be changed.

[0039]

As described above, according to the present invention, a simple part of an image change is transmitted as a motion parameter, and a complicated part of an image change is transmitted as image data. Processing can be performed at high speed. Further, it is possible to accurately reproduce a change in a portion where a complicated change occurs in the image.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a real-time recognition / synthesis apparatus according to an embodiment of the present invention and a direction in which data is sent.

FIG. 2 is a diagram for explaining a specific example of the expression detector of FIG. 1;

3 is a diagram showing a wireframe model of a person's head included in the person model database shown in FIG. 1;

FIG. 4 is a diagram for explaining a process performed by a model deformation adjusting unit in FIG.

5 is a diagram for explaining an image synthesizing method of the receiving side system in FIG. 1. FIG.

6 is a first image data 3 in the third image composition unit of FIG.
FIG. 9 is a diagram for explaining a color tone adjustment process at a boundary between first and second image data 33;

FIG. 7 is a diagram specifically showing the image synthesis of FIG. 3 using the processing method of FIG.

FIG. 8 is a diagram showing a configuration of a conventional real-time recognition synthesizing apparatus and a data sending direction.

9 is a diagram for explaining a specific example of the facial expression detector of FIG.

[Explanation of symbols]

 3 person on the sending side 5 composite image 7 virtual space 9 display 21 body motion parameter 23 face marker motion parameter 25 image data 111 person model database 117 model deformation adjustment unit

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Taiichi Kitamura Taiichi Kitamura, Seika-cho, Kyoto Prefecture, Osamu Osamu, Osamu Osamu, No. 5 Mihiratani, ATR Communication Systems Research Institute Co., Ltd. (56) Reference JP-A-7- 325936 (JP, A)

Claims (1)

(57) [Claims] 1. An image model on a transmission side is configured as image model data by color texture mapping on a three-dimensional wire frame model, and the image model data is stored in a virtual space generated by using computer graphics technology. A real-time image recognition / synthesis device for arranging and stereoscopically displaying the movement of an image on a stereoscopic display on the receiving side, the storage unit storing the image model data; A parameter detection unit that detects a motion parameter based on motion, an image data detection unit that captures at least a part of the image model on the transmission side and detects image data, and the stored image model data, First synthesizing means for synthesizing the detected motion parameter to generate first image data, and the detected image data. Data, and first adjusting means for adjusting the first intermediate image data by synthesizing the partial image model data corresponding to the imaged image model portion of the stored image model data; Among the captured motion parameters, a proximity parameter that is close to the outer edge of the captured image model portion and peripheral image model data that corresponds to at least the periphery of the captured image model portion of the image model data. Second adjusting means for combining and adjusting the second intermediate image data; and a second adjusting means for combining the adjusted first intermediate image data and second adjusted intermediate image data to generate second image data. A second combining unit; a third combining unit that combines the generated first image data with the generated second image data to generate third image data; Third Based on the image data, and a image reproduction means for reproducing the motion image model of the transmission side,
Real-time image recognition and synthesis device.