JP2915846B2 - 3D video creation device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a three-dimensional image creating apparatus for creating a three-dimensional image in a virtual space using computer graphics technology.

[0002]

2. Description of the Related Art In recent years, technologies such as virtual reality and virtual environment have been attracting attention as advanced information transmission means between people and computers, or between people, and have been used in fields such as human interfaces, multimedia, and video communication. Research is progressing rapidly.

[0003] Here, the development of a video communication system (hereinafter, also referred to as a "realistic communication conference system") capable of performing a conference or a cooperative operation with a feeling that people in remote places are as if they are together. A technique for detecting three-dimensional information of the movement of a human body with high accuracy, estimating a posture of a person, and generating a three-dimensional person CG (Computer Graphics) model in real time is being developed.

FIG. 10 shows a three-dimensional image for generating a three-dimensional person image in a realistic communication conference system that has already been devised.
FIG. 1 is a diagram illustrating an overall configuration of a two-dimensional image creation device.

[0005] As shown in FIG. 10, the three-dimensional image creating apparatus includes a modeling unit 10 for generating a three-dimensional model corresponding to each part 11 of a human body, and a real-time movement of a person 2 in a real space 1. It comprises a detecting section 3 for detecting and a reproducing section 20 for synthesizing a human image in real time.

Here, the modeling unit 10 includes a shape input unit 13 for storing the shape of each part 11 of the human body, and a three-dimensional model for each part based on the shape of each part of the human body stored in the shape input unit 13. A model generation unit 15 for creating a wire frame model; a video input unit 17 for inputting an image of each part of the human body; and a color texture storage for acquiring color information on the human body surface based on the video input to the video input unit 17 Unit 19.

The detecting section 3 includes a torso rotational movement detecting section 4 for detecting the rotational movement of the torso of the person 2, a head rotational movement detecting section 5 for detecting the rotational movement of the head of the person 2, 1 includes a finger movement detection unit 7 that detects the movement of the finger of the person 2 and an arm rotation movement detection unit 9 that detects the rotation movement of the arm of the person 2.

The reproducing unit 20 includes a model deforming unit 21 that deforms the three-dimensional wire frame model generated by the model generating unit 15 so as to correspond to the movement of the person 2 based on the information obtained by the detecting unit 3. Supplies the deformed three-dimensional wireframe model with a texture mapping unit 23 for applying a color based on the color information obtained by the color texture storage unit 19, and data for generating a three-dimensional virtual space 31 to a display 29. Virtual space generation unit 25 and 3
It includes a combining unit 27 that combines a human image 33 composed of a three-dimensional wireframe model with the virtual space 31 and displays a three-dimensional video on a display 29.

As described above, the three-dimensional model of the conference participant is transmitted to the receiving side prior to the communication conference, and during the conference, the motion information of the transmitting participant is detected in real time and transmitted to the receiving side. At the sending and receiving sides, the three-dimensional person model is transformed according to the detected movement, color texture is mapped, and a person image 33 is synthesized in the virtual conference room scene according to the viewpoint of the receiver participant 35. Then, it is displayed on the display 29.

[0010]

However, the creation of a three-dimensional person image in the above-mentioned immersive communication conference system corresponds only to the upper body movement of the person attending the conference, and the whole body movement including the movement of the lower limbs. Has not been created.

In addition, since a three-dimensional model of a human torso is used as a rigid body for modeling, a problem has arisen in that a three-dimensional image reconstructing a motion involving twisting and bending of a human torso becomes unnatural.

The present invention has been made in order to solve these problems, and provides a three-dimensional image creating apparatus which creates a three-dimensional whole-body image closer to the actual posture of a person and imitating the whole body. With the goal.

[0013]

[0014]

A three-dimensional image creating apparatus according to the present invention is a three-dimensional image creating apparatus for creating a person image in a virtual space in a real-life communication conference system. Model generation means for generating a model, mounted on a person, a sensor for detecting the position and rotation angle of the part of the mounted person, a torso image determination means, an upper limb image determination means, a lower limb image determination means ,
Image combining means. Here, the torso image determining means divides the torso portion of the three-dimensional wire frame model generated by the model generating means into a plurality of regions,
The image of the torso portion is determined by calculating the coordinates of each point constituting the joint surface between the regions by using a freeform deformation method based on the data obtained by the sensor. The upper limb image determining means defines the kinetic energy of the upper limb of the person and determines an image of the upper limb portion of the three-dimensional wireframe model based on data obtained by the sensor so as to minimize the kinetic energy. The lower limb image determining means determines an image of the lower limb portion of the three-dimensional wire frame model corresponding to the motion of the lower limb from the sitting to the standing of the person based on the data obtained by the sensor and a predetermined constraint condition. I do. The image combining means combines the images determined by the torso image determining means, the upper limb image determining means, and the lower limb image determining means to create a human image in the virtual space.

[0015]

Embodiments of the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

FIG. 1 is a block diagram showing a configuration of a three-dimensional image creating apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the three-dimensional image creating apparatus includes four sensors 37 mounted on the human chest, head, and both hands, three-dimensional coordinates indicating the position of each sensor 37, and A data input unit 39 for taking in and storing data of rotation angles around three axes of X-axis, Y-axis, and Z-axis;
A shape input unit 13 for inputting a shape of each part of a human body such as a head and an arm, a model generation unit 15 for generating a three-dimensional wireframe model based on data input to the shape input unit 13, and a model generation unit 15 And a torso image determining unit 41 for transforming the three-dimensional wireframe model of the torso part generated based on the data input to the data input unit 39, and the three-dimensional wireframe model of the upper limb part generated by the model generating unit 15. Upper limb image determining unit 47 that deforms the image based on the data input to the data input unit 39, and the model generating unit 15
The lower limb image determining unit 53 that deforms the three-dimensional wireframe model of the lower limb portion generated in the above based on the data input to the data input unit 39, the torso image determining unit 41, the upper limb image determining unit 47, and the lower limb image determining unit An image synthesizing section 59 is provided for synthesizing the three-dimensional images determined in 53 and generating a three-dimensional human image imitating a real person in a virtual space projected on a display.

Hereinafter, each of the above parts will be described in detail. The data input unit 39, the model generation unit 15,
Torso image determination unit 41, upper limb image determination unit 47, lower limb image determination unit 5
3 and the image synthesizing unit 59 are, for example, a graphics workstation (Onyx, Reality Engine 2) manufactured by SGI.
).

A three-dimensional model of a person is created by creating a three-dimensional wire frame model in the model creating unit 15 for each part 11 of the human body such as a head, a torso, and a hand, and connecting these based on the structure of the human body. I do. Here, in the acquisition of the three-dimensional information of each part 11 of the human body in the shape input unit 13, a linear laser light is irradiated while rotating around the measurement object using a color three-dimensional digitizer manufactured by Cyberware. The three-dimensional shape information of the measurement target is input by measuring the deformation, and the color information (color texture) of the measurement target surface is also acquired. Then, the obtained three-dimensional information is stored in the model generation unit 15.
Are converted into large and small triangles (triangular patches) according to the accuracy, and a three-dimensional wireframe model is created. Further, the model generation unit 15 creates a three-dimensional model of each part 11 of the human body by mapping the color texture to a triangular patch at a corresponding location.

Next, the determination of the torso image performed by the torso image determining unit 41 will be described. As shown in FIG. 2, the three-dimensional wireframe model of the torso created by the model generation unit 15 is vertically divided into two parts in the displacement amount calculation unit 43 according to the region of the chest and the waist, as shown in FIG. By dividing each region into three, each region is divided into a total of six segments.

Here, in order to reconstruct the natural torso bending motion of a person, continuity at the joining surface of the adjacent segments is required. Therefore, in order to satisfy this demand, Free Form Defo
rmation) method (hereinafter, also referred to as “FFD method”) is applied to the deformation of the three-dimensional wireframe model of the body generated by the model generation unit 15. That is, the displacement amount calculation unit 43 performs the three-dimensional wireframe model of the torso created by the model generation unit 15 in accordance with the displacement amount (coordinates and rotation angles) obtained by the sensor 37 mounted on the chest of the person. Is automatically calculated, and the torso image deformation unit 45 drives the vertices based on the result of the above calculation in the displacement amount calculation unit 43, thereby deforming the three-dimensional wireframe model of the torso. Perform

Here, in the deformation of the three-dimensional wireframe model of the body, the vertices of the wireframe model located at the joint surface with the adjacent segment are FFD.
By calculating the displacement amount by using the technique, it is possible to smoothly deform the three-dimensional shape while maintaining the continuity with the adjacent segments.

More specifically, 3 which constitutes the chest and waist
As shown in FIG. 3, the control point Cn (Cnx, Cnx
ny, Cnz) and (n = 0 to 3). Assuming that the weight for deformation of the vertex P (p _x , p _y , p _z ) of the three-dimensional wire frame model is t (t = 0 to 1), the three-dimensional coordinate P ′ after the change by the FFD method is

[0024]

(Equation 1)

## EQU2 ## By multiplying the obtained coordinate value P 'by the homogeneous transformation matrix, the coordinates of the vertex of the wire frame model in the three-dimensional space are obtained.

Next, a method for describing the structure of the whole body of a person will be described. The skeletal structure of the whole body of the person has three-dimensional coordinate values and 26 revolute joints, as shown in FIG. 4, by a revolute joint having one or three degrees of freedom and a link having the same length as the actual person. It can be defined as a link mechanism having a total of 29 degrees of freedom. In this link model, one coordinate system is assigned to one link, and the link coordinate system (n
Assuming that the rotation angles of the X, Y, and Z axes from -1) to the link coordinate system n are α, β, γ, and the length of the link is d, the homogeneous transformation matrix Rn representing the coordinate transformation is

[0027]

(Equation 2)

## EQU1 ## Where S and C are the sin function and c
Represents the os function. Further, the position vectors of the link tip in the reference coordinate system and the link coordinate system n are represented by X ₀ , Xn
X ₀ = R ₁ R ₂ ... RnXn, and the geometric structure of the link model can be described by using the matrix Rn.

In this embodiment, in order to estimate the posture of the whole body in real time, as shown in FIG. 4, a magnetic field capable of detecting a total of six degrees of freedom of three-dimensional coordinates and three-axis rotation angles is detected. By mounting the sensors 61, 63, 65 and 67, the whole body operation is detected. At this time, it is desirable to minimize the number of sensors in consideration of the burden of the user wearing the sensors, but it is difficult to accurately detect all joint rotation angles from limited sensor information.

Therefore, the magnetic sensors 61, 63, 65, and 67 are mounted on the chest, head, and both hands, respectively, and the three-dimensional coordinates and the three-axis rotation angles obtained from the four sensors are used to obtain a constraint described later from a total of 24 data. By using the condition, 29 degrees of freedom regarding the posture of the whole body are estimated. In addition, the movement of the finger is detected by wearing the data glove.

Here, more specifically, Polhemu
A magnetic sensor (FASTRAK) manufactured by s Corporation is used.

Next, the determination of the upper limb image performed by the upper limb image determination section 47 will be described. In the link model shown in FIG. 4 described above, the position of the chest is used as the reference coordinates,
The position of the torso and the three-axis rotation angle are uniquely determined from the data of the magnetic sensor 63 mounted on the chest, and the three-axis rotation angle of the head is determined from the data of the magnetic sensor 61 mounted on the head.

On the other hand, in determining the upper limb image, the upper limb joint angle estimating unit 49 uses the shoulder position shown in FIG. 4 as a reference coordinate, the three-axis rotation angles (α2, β2, γ2) of the chest and the link model. By using the shoulder link length l _S of
The three-dimensional coordinates of the shoulder are calculated.

The link model is a model assumed by the upper limb joint angle estimating unit 49 based on the three-dimensional wire frame model created by the model generating unit 15.

However, since the upper limb (here, one arm means) is modeled as a link mechanism having seven degrees of freedom, as shown in FIG. 6-dimensional data obtained (3-dimensional coordinates,
Estimating these seven joint angles from the (three-axis rotation angles) does not have a single degree of freedom, so a solution cannot be uniquely obtained. Therefore, in order to solve this problem, the kinetic energy of the upper limb is defined as a constraint condition, and the joint angle is uniquely determined by a solution that minimizes the energy.

More specifically, in FIG. 5, the hand position vector obtained from the sensor 65 is represented by X = (p _X , p _Y ,
p and _Z, r _X, r _Y, and r _Z), joint angle bell to theta = (.theta.1 representing the attitude of the upper limbs, ..., when .theta.7), to seek from X theta is insufficient one degree of freedom Therefore, it is a so-called bad setting problem in which the solution exists infinitely. Thus, each joint is modeled as a rotary spring that generates a repulsive force in proportion to the rotation angle, and the link is modeled as a cylinder having mass, and the kinetic energy of the upper limb model is defined. The rotation angle θ3 of the redundant joint is changed within a certain range and the kinetic energy is sequentially calculated, and the value of the rotation angle at which the kinetic energy is minimized within the aforementioned range is set as an optimum value, and the joint angle vector Θ is uniquely determined. To decide.

The limb kinetic energy evaluation function is

[0038]

(Equation 3)

Where the first term represents potential due to gravity, the second term represents stress, the third term represents friction, and the fourth term represents acceleration energy.

Since the potential energy P of the first term depends only on the height of the elbow, it is defined by the following equation.

[0041]

(Equation 4)

In this equation, g is the gravitational acceleration, m1,
m2 is the mass of each upper arm and lower arm, the P _EY represents the height of the elbow.

Regarding the stress energy of the second term, 7
Each joint is modeled by a rotary spring, and a threshold (θ′n, θ ″) is determined with respect to the rotational angle θn (n = 1 to 7) of the joint.
By providing n), it is defined by the following equation.

[0044]

(Equation 5)

In this equation, when the rotation angle θn of the joint exceeds the threshold value, / θn becomes the difference value between θn and the threshold value, and otherwise becomes 0. Also, Kn
Represents the spring constant of the rotary spring.

The friction energy of the third term is defined by the following equation.

[0047]

(Equation 6)

In this equation, μn represents a friction coefficient, Nn represents a drag, and Δθn represents an angular displacement per unit time.

The energy due to the acceleration of the fourth term is defined by the movement of the elbow and is expressed by the following equation.

[0050]

(Equation 7)

In this equation, P _E is the position vector of the elbow, ΔP _E is the displacement of the coordinates of the elbow in unit time, (m1
+ M2) / 2 represents the inertial mass.

Then, based on the parameters determined by the upper limb joint angle estimating section 49 as described above, the upper limb image deforming section 51 deforms the three-dimensional wire frame model of the upper limb created by the model generating section 15. .

Next, the determination of the lower limb image performed by the lower limb image determining section 53 will be described. Lower limb joint angle estimation unit 5
5 obtains the three-dimensional coordinates Ph of the waist using the three-axis rotation angle data (α2, β2, γ2) obtained from the chest magnetic sensor 63 shown in FIG. 4 and the link length lt of the trunk. Then, the whole body movement accompanied by the movement of the lower limb is only a transition movement from sitting to standing, and as shown in FIG.
Since the joints are modeled as link mechanisms having degrees of freedom (hips, knees, and ankles), these joint angles are estimated by using the calculated three-dimensional coordinates Ph of the hips and the three constraint conditions described below. You. 1) Both feet are fixed on the floor. 2) The joint rotation angles of the hips, knees and ankles have the same left and right values. 3) The locus of the waist moves on the YZ plane (X = 0).

Magnetic sensor 6 mounted on chest shown in FIG.
YZ coordinates (Yc, Z) indicating the chest position Pc obtained from
c), the waist position P obtained by the above calculation
YZ coordinates (Yh, Zh) indicating h and ankle position P
length L ₁ L ₂ using YZ coordinates (Yf, Zf) indicating f
, And using the link lengths l _{1 to} l ₃ of the model, these joint angles can be calculated geometrically.

The link model is a model assumed by the lower limb joint angle estimating unit 55 based on the three-dimensional wire frame model created by the model generating unit 15.

Based on the parameters determined by the lower limb joint angle estimating section 55 in this way, the lower limb image deforming section 57
Transforms the three-dimensional wireframe model of the lower limb created by the model generating unit 15.

Next, the deformation of the three-dimensional wire frame model in the torso image deforming section 45, the upper limb image deforming section 51, and the lower limb image deforming section 57 will be described.

As described above, the structure of the three-dimensional person model can be described by a homogeneous transformation matrix.
3. Upper limb joint angle estimator 49, lower limb joint angle estimator 55
Using the parameters calculated or estimated respectively in
The above-mentioned homogeneous transformation matrix Rn is calculated for each human body part. Then, by multiplying each vertex of the three-dimensional wireframe model by the homogeneous transformation matrix Rn, the triangular patch of the wireframe model included in the human body part is deformed and translated. Parts including joints such as elbows have "weight" used for shape deformation. As shown in FIG. 7, assuming that the weight of the vertex vector X of the joint part 69 is a and the homogeneous transformation matrices of the adjacent parts 71 and 73 are R ₁ and R ₂ respectively, the vertex vector X ′ after deformation is

[0059]

(Equation 8)

The three-dimensional shape of the joint part 69 is calculated in such a manner that the movement of the joint can be reproduced.

For example, in the displacement calculating section 43,
The three-axis rotation angle obtained from the magnetic sensor 63 of the chest is divided based on the deformation ratio of each segment obtained by the preliminary experiment, and the torso image deformation unit 45 performs the calculation based on the displacement calculated by the displacement calculator 43. To change the shape of each segment.
In addition, the three-dimensional shape of the body part is deformed in such a manner that a smooth bending operation can be performed by applying the FFD method as described above with respect to the vertices located at the joining surface of the adjacent segments.

Then, the image synthesizing unit 59 synthesizes the three-dimensional human model from the three-dimensional image deformed by the above method into the virtual space 31 projected on the display 29, and reproduces the actual motion of the human.

The three-dimensional rotation angle of the body is changed by simulation, the rotation angle data is distributed based on the deformation ratio of each segment obtained in advance in a preliminary experiment, and the three-dimensional shape of the body part is deformed. An experiment in which the body motion was reconstructed and the results are described.

First, as a method of the preliminary experiment, the bending motion of the body is photographed by two cameras installed in front of and directly beside the subject, and recorded on a video disk. Next, the captured images in two directions are taken into a graphic workstation (GWS), and superimposed on a three-dimensional human computer graphic model on the GWS and displayed on a display. Further, the deformation ratio of each segment of the body part is appropriately changed, and the degree of overlap between the actual image on the display and the computer graphic (CG) model is visually evaluated to determine the deformation ratio of the body in each segment. In addition, each rotation axis (X,
The sum of the deformation ratios of the six segments with respect to (Y, Z) is defined as 1.

The experimental results are shown in Table 1 below.

[0066]

[Table 1]

With respect to the segment numbers, numbers 1 to 6 are assigned with the top of the trunk segment shown in FIG. From Table 1, with respect to the X-axis rotation for bending the body in the front-rear direction, each segment is deformed at an equal rate, and the Y-axis rotation indicating the twist of the body and the Z-axis rotation indicating the bending in the left-right direction are shown. Indicates that the deformation ratio at the waist is high and the ratio decreases as approaching the chest.

By applying the deformation ratio at the body part obtained by this preliminary experiment to the FFD shape deformation of the body part, natural movement of the body can be reproduced.

Specifically, FIG. 8A shows a reconstructed image of the torso motion according to the conventional method, whereas FIG. 8B shows a reconstructed image of the torso motion according to the method according to the present invention.
Is shown in

As described above, it has been confirmed that the three-dimensional image creating apparatus according to the present invention can reconstruct a smooth and natural torso bending motion.

Next, a description will be given of a comparison experiment with an actually measured value for evaluating the operation of the upper limb image determining unit 47 and the result thereof. As an experimental method, by attaching a magnetic sensor to the elbow, a comparison was made between the measured value of the three-dimensional coordinates of the elbow and the estimated value calculated by the upper limb joint angle estimating unit 49 according to the present invention. The experimental results are shown in FIG. Here, FIG.
(A), (b), and (c) show the X coordinate, Y coordinate, and Z coordinate, respectively.
Regarding the coordinates, a solid line indicates an actually measured value, and a broken line indicates an estimated value. As shown here, the estimated value followed the movement with the same tendency as the actually measured value with some vibration, and the estimation error was within 3 cm.

Further, based on a total of 24 data obtained from the magnetic sensors 61, 63, 65, and 67 mounted on the chest, head, and both hands, the whole body movement of the person who attended the meeting from sitting to standing up to bowing was re-executed. Although the configuration was performed, it was confirmed that the whole body motion including the natural torso bending motion can be reconstructed at a speed of about 8 frames / second.

Although the above description relates to the motion of a person, the present invention creates a three-dimensional image of an object that moves with some degrees of freedom, such as an arm of an industrial robot. It is also useful in such cases.

[0074]

[0075]

According to the three-dimensional image creating apparatus according to the present invention, in a virtual space teleconferencing system, can 3D image is created to reproduce the smooth and natural fuselage bending motion of the person, Systemic person A three-dimensional image that reproduces the operation can be created.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a three-dimensional image creating apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram illustrating a wireframe model of a trunk.

FIG. 3 is a diagram for explaining application of the FFD method to a body part.

FIG. 4 is a diagram showing a skeleton model of a person.

FIG. 5 is a diagram showing a skeletal model of an upper limb.

FIG. 6 is a diagram showing a skeletal model of a lower limb.

FIG. 7 is a diagram showing a deformation method of a joint part.

FIG. 8 is a diagram showing a result of reconstruction of a torso motion.

FIG. 9 is a graph showing a result of reconstruction of upper limb motion.

FIG. 10 is a block diagram showing an overall configuration of a conventional three-dimensional video image creating apparatus.

[Explanation of symbols]

 Reference Signs List 15 Model generation unit 37 Sensor 41 Torso image determination unit 43 Displacement amount calculation unit 45 Torso image deformation unit 47 Upper limb image determination unit 49 Upper limb joint angle estimation unit 51 Upper limb image deformation unit 53 Lower limb image determination unit 55 Lower limb joint angle estimation unit 57 Lower limb Image deformation unit 59 Image synthesis unit 61, 63, 65, 67 Magnetic sensor

────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Fumio Kishino 5 Sanriya, Seiya-cho, Seika-cho, Soraku-gun, Kyoto ATR Communication Systems Research Institute, Inc. (56) References Information Processing Society of Japan Vol. 96 , No .. 18, 96-CG-79, pp. 31-37, "Broadcast use of real-time motion capture", published February 23, 1996. (58) Fields investigated (Int. Cl. ⁶ , DB name) G06T 15/00-17/50

Claims (1)

(57) [Claims] 1. A three-dimensional video creating apparatus for creating a person image in a virtual space in a sensation communication conference system, comprising: a model generating means for generating a three-dimensional wireframe model as the person image; A sensor for detecting a position and a rotation angle of a part of the person attached to the body, and a body part of the three-dimensional wire frame model generated by the model generating unit is divided into a plurality of regions;
A torso image determining means for determining the image of the torso portion by calculating the coordinates of each point constituting the joint surface between the regions by using a freeform deformation method based on data obtained by the sensor, Upper limb image determining means for defining kinetic energy of the upper limb of the person and determining an image of an upper limb portion of the three-dimensional wireframe model based on data obtained by the sensor so that the kinetic energy is minimized; A lower limb image determination for determining an image of a lower limb portion of the three-dimensional wire frame model corresponding to the lower limb movement from the sitting to the standing of the person based on the data obtained by the sensor and a predetermined constraint condition; Means, the torso image determining means, the upper limb image determining means, and the image respectively determined by the lower limb image determining means Synthesized and provided with an image synthesizing means for creating the figures in the virtual space, three-dimensional image creation device.