JP2627487B2 - Real-time image recognition and synthesis device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-time recognition / synthesis apparatus for the movement and posture of an image of a person or the like, and more particularly, to an image model on the transmission side formed by color texture mapping on a three-dimensional wire frame model. This image model is placed in the generated virtual space using computer graphics technology, and is displayed on the stereoscopic display on the receiving side.
The present invention relates to a real-time recognition / synthesis apparatus for an image which reproduces the movement of an image and stereoscopically displays it.

[0002]

2. Description of the Related Art A conventional real-time image recognition / synthesis apparatus is disclosed in, for example, IEICE Technical Report p. 23-28, "3 in the presence communication conference"
Real-Time Display of Dimensional Images ". Hereinafter, the apparatus will be described.

FIG. 6 is a block diagram of a conventional real-time recognition / synthesis apparatus, and FIG. 7 is a specific example of the expression detector 103 of FIG. The arrows in the figure indicate the direction in which data is sent.

[0004] The real-time recognition / synthesis apparatus is mainly composed of a system 11 on the transmitting side and a system 13 on the receiving side. The transmitting-side system 11 includes a body motion detector 101 that detects a body motion of the transmitting-side person 3 in the real space 1;
And a facial expression detector 103 for detecting facial expressions. As the body movement detector 101, for example, a data glove of VPL, a magnetic sensor of Polhemus, or the like is used. The data obtained by the body motion detector 101 is used as the body motion parameter 21 as the system 1 on the receiving side.
3 is sent to the image synthesis unit 115.

[0005] Further, as shown in FIG.
3 is an apparatus which captures an image 205 with the camera 201 fixed to the head and fixed to the head. The movement amount of the marker 203 in the image 205 is tracked and sent to the image synthesizing means 115 in the receiving system 13 as the motion parameter 23 of the facial expression.

On the other hand, the receiving side system 13 is provided with a person model database 111 in which the person 3 on the transmitting side is modeled by a wire frame model and a color texture in advance. The three-dimensional wire frame model is obtained by approximating the surface of a human body with triangular patches, and the color texture is color information of the surface of the human body.

[0007] In the image synthesizing section 115, the person model data composed of the three-dimensional wire frame model and the color texture in the person model database 111,
The transmitted body motion parameters 21 and facial expression motion parameters 23 are combined. In the synthesis, the coordinates of each vertex of the three-dimensional wireframe model of the human model data are
This is performed by moving the body according to the body movement parameter 21 and the facial expression movement parameter 23.

[0008] The synthesized human image 5 is arranged in a virtual space 7 generated using computer graphics technology. The virtual space 7 including the human image 5 is displayed on the stereoscopic display 9 on the receiving side.

[0009]

However, in such a real-time image recognition / synthesis apparatus, since all the movements of the image are transmitted as motion parameters, there is a problem that a subtle change in the image cannot be recognized and synthesized. . In particular, when transmitting the facial expressions of a person, it is difficult to recognize and combine subtle changes in facial expressions around the eyes, which play an important role in communication, only by measuring the displacement of the marker anchored to the face. is there.

To solve this problem, it is conceivable to use an image processing technique such as edge detection. This makes it possible to increase the data to be measured and to recognize and combine subtle changes in the image to some extent, but the amount of image processing becomes enormous, requiring an expensive image processing device and processing information. Increases the processing speed,
Real-time recognition may not be possible.

[0011] Therefore, a main object of the present invention is to reproduce a subtle change in an image, and to process the amount of communication data and the amount of processing information necessary for measurement on the transmitting side and the amount of processing information required for image synthesis on the receiving side. An object of the present invention is to provide a real-time image recognition / synthesizing apparatus having a sufficiently small amount and suitable for practical use.

[0012]

In order to achieve the above-mentioned object, an image real-time recognition / synthesis apparatus according to the present invention provides an image model data generating apparatus which performs color texture mapping of an image model on a transmitting side to a three-dimensional wire frame model. A real-time image recognition / synthesis device that arranges image model data in a virtual space generated by using computer graphics technology, and reproduces the motion of the image on a receiving-side three-dimensional display. And storage means for storing image model data, and parameter detection means for detecting a motion parameter based on the operation of the image model on the transmission side,
Image data detecting means for capturing at least a part of the image model on the transmitting side and detecting image data, synthesizing the stored image model data and the detected motion parameter, and converting the first image data First synthesizing means for generating, second image synthesizing means for synthesizing the stored image model data and the detected image data to generate second image data, and the generated first image Third combining means for combining the data with the generated second image data to generate third image data; and a transmitting-side image model based on the generated third image data. Image reproducing means for reproducing motion.

[0013]

The image real-time recognition / synthesis apparatus according to the present invention has the following functions.

First, the motion parameters of the image sent from the transmitting side are combined with the image model data stored in advance on the receiving side to obtain first image data. Particularly, a portion requiring a delicate image reproduction is separately transmitted as image data from the transmission side, and the image data is combined with the image model data on the reception side to obtain second image data. Then, a combination of the first image data and the second image data is defined as third image data. The third image data is reproduced on the display on the receiving side in real time.

[0015]

1 is a block diagram showing a real-time recognition / synthesizing apparatus according to an embodiment of the present invention. FIG. 2 is a diagram showing an expression detector 103 shown in FIG.
Is a specific example.

The arrows in the figure indicate the direction in which data is sent. The real-time recognition device mainly includes a transmitting system 11 and a receiving system 13.

The system 11 on the transmitting side includes a body motion detector 10 for detecting the body motion of the person 3 on the transmitting side in the real space 1.
1 and an expression detector 103 for detecting an expression. As the body motion detector 101, for example, a data glove manufactured by VPL or a magnetic sensor manufactured by Polhemus is used. The data obtained by the body motion detector 101 is transmitted to the receiving system 13 as the body motion parameter 21.

The display detector 103, as shown in FIG.
This is an apparatus for photographing facial expressions with a camera 201 fixed to the head. The camera 201 images the face 209 and the marker 203 fixed on the skin surface above the facial expression muscle of the face 209. Marker 2 in the captured image data 205
03, the moving direction and the moving distance of the face marker are tracked.
The data around the eye, which undergoes particularly complicated changes, is sent as image data 25 to the receiving system 13.

The receiving side system 13 is provided with a person model database 111 in which the person 3 on the transmitting side is modeled in advance using a three-dimensional wire frame model and a color texture.

The three-dimensional wire frame model is obtained by approximating the surface of the human body with triangular patches, and the color texture is color information on the surface of the human body. The image data is sent to the one-image combining unit 105, where it is combined with the three-dimensional wireframe model and the color texture 27 extracted from the person model database 111. Specifically, the synthesis is performed by deforming the vertex coordinates of the three-dimensional wire frame model in accordance with the body movement parameter 21 and the face marker movement parameter 23. Thereby, the shape change due to the movement of the human body or the movement of the face marker is reproduced.

On the other hand, the image data 25 is stored in the second image
7 is mapped to a desired position of the three-dimensional wire frame model 29 extracted from the person model database 111. Mapping is for example Silicon Graphics
It is performed by the company's workstation ONYX. The image data 31 and 33 obtained by the first image combining unit 105 and the second image combining unit 107 are sent to the third image combining unit and combined. The synthesized human image 5 is arranged in a virtual space 7 generated using computer graphics technology. The virtual space 7 including the human image 5 is displayed on the stereoscopic display 9 on the receiving side.

The color tone data 41 of the image synthesized by the first image synthesizing unit and the color tone data 43 of the image synthesized by the second image synthesizing unit are sent to a color tone inspection unit 113 and compared there. When the difference is larger than the predetermined value, the information 39 of the difference is fed back to the person model database 111, and the color texture in the person model database 111 is corrected so that the difference becomes smaller.

In a conference system or the like, since the lighting conditions and the like do not change very frequently, this correction process may be performed once every several seconds to several minutes. Since several to several tens of image data 25 are sent per second, correcting the color texture in the human model database 111 is more effective than correcting the image data 25 in reducing the amount of processing information. It is advantageous.

FIGS. 3 to 5 show the change in color tone at the boundary between the image synthesized by the first image synthesizing unit 105 and the image synthesized by the second image synthesizing unit 107 in the embodiment of FIG. It is a figure for explaining processing performed in order to make it inconspicuous.

In FIG. 3, the three-dimensional wire frame model 29 around the eyes extracted from the human model data 207 in the human model database is sent to the second image synthesizing means 107, where it is synthesized with the image data 25. 2 image data 33. On the other hand, the three-dimensional wire frame model and the color texture 27 of the part except for the area around the eyes extracted from the human model data 207 are sent to the first image synthesizing means 105, and the body motion parameters 21 and the face marker motion parameters 23 Are combined to form the first image data 31. The first image data 31 and the second
Is sent to the third image synthesizing means 109,
The human image 5 is synthesized.

At this time, in order to make the boundary between the first image data and the second image data inconspicuous, each includes data corresponding to the color tone adjustment portion 504. The color tone adjustment part 504 is an area where the area of the first image data 31 and the area of the second image data 33 overlap as shown in FIG.

In FIG. 4, the color tone data 600 at the boundary point A between the first image data 31 and the color tone adjustment portion 504,
Assuming that the color tone data 602 at the boundary point B with the color tone adjustment portion 504 of the second image data 33 has a color tone level as shown in FIG.
The color tone data 600 of the image data 31 is linearly reduced in the area of the color tone adjustment portion 504 as the color data 600 progresses toward the region of the second image data 33.
Is set to 0 at the position of the boundary point B on the area side of the second image data 33 in the area. This linear decrease in tone data is 606
Shown in

Similarly, the color tone data 602 of the second image data 33 is linearly decreased in the area of the color tone adjustment portion 504 as the area advances toward the area of the first image data 31, and the color tone adjustment is performed. First image data 3 in the area of portion 504
It is set to 0 at the position of the boundary point A on the side of the area of 1. This reduction in linear tone data is shown at 608.

The color tone data in the color tone adjustment portion 504 is obtained by adding the color tone data 606 of the first image data and the color tone data 608 of the second image data. The data is assumed to be 604.
With this processing, the boundary between the first image data 31 and the second image data 33 can be made inconspicuous.

As shown in FIG. 5, the color tone may be different depending on the location in the first image data 31 and the second image data 33. In that case, the color tone adjustment processing shown in FIG. 4 may be performed for each portion.

In this embodiment, the part to be transmitted as image data is set at the eye, but may be at another part such as the mouth.

In this embodiment, a human figure is taken as an example, but the present invention can be applied to an animal or the like.

[0033]

As described above, according to the present invention, a simple part of an image change is transmitted as a motion parameter, and a complicated part of an image change is transmitted as image data. Processing can be performed at high speed. Further, it is possible to accurately reproduce a change in a portion where a complicated change occurs in the image.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of a real-time recognition / synthesis apparatus according to an embodiment of the present invention and a direction in which data is sent.

FIG. 2 is a diagram for explaining a specific example of the expression detector of FIG. 1;

FIG. 3 is a diagram for explaining an image synthesizing method of the receiving side system in FIG. 1;

FIG. 4 is a diagram illustrating first image data 3 in a third image synthesizing unit in FIG. 1;
FIG. 9 is a diagram for explaining a color tone adjustment process at a boundary between first and second image data 33;

FIG. 5 is a diagram specifically illustrating the image synthesis of FIG. 3 using the processing method of FIG. 4;

FIG. 6 is a diagram showing a configuration of a conventional real-time recognition / synthesis apparatus and a direction in which data is sent.

FIG. 7 is a diagram for explaining a specific example of the expression detector of FIG. 6;

[Explanation of symbols]

 Reference Signs List 3 Sending person 5 Synthesized person image 7 Virtual space 9 Display 21 Body movement parameter 23 Face marker movement parameter 25 Image data 111 Person model database

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Taiichi Kitamura 5 Sanriya, Seiya-cho, Sorika-cho, Kyoto, Japan Research Report VOL. 92, NO. 443 p. 23-28 "Real-time display of 3D face images in a telecommunications conference" (1993.1.25)

Claims (1)

(57) [Claims] 1. An image model on a transmission side is configured as image model data by color texture mapping on a three-dimensional wire frame model, and the image model data is stored in a virtual space generated by using computer graphics technology. A real-time image recognition / synthesis device that arranges and reproduces the motion of an image on a stereoscopic display on the receiving side, and that stores the image model data. Based on an operation or the like, a parameter detection unit that detects a motion parameter, an image of at least a part of the image model on the transmission side, an image data detection unit that detects image data, and the stored image model data. First synthesizing means for synthesizing the detected motion parameters to generate first image data; Data, by combining said detected image data, the second for generating the second image data
Combining means for combining the generated first image data and the generated second image data to generate third image data; and Image reproducing means for reproducing the movement of the image model on the transmission side based on third image data.
Real-time image recognition and synthesis device.