JP2813971B2 - State reproduction method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a state reproducing method for reproducing a state when an object moves arbitrarily. In particular, a facial expression of a person is naturally reproduced using a three-dimensional model of a person. It relates to a state reproduction method that can be performed.

[0002]

2. Description of the Related Art Conventional state reproducing methods include, for example,
(1) Atsushi Otani, Taiichi Kitamura, Haruo Takemura, Fumio Kishino "Real-Time Display of 3D Face Images in Realistic Communication Conference" IEICE Technical Report HC92-61, (2) Noriko Suzuki, Atsushi Otani, Fumio Kishino " Examination of Facial Expression Reproduction Method Based on Three-Dimensional Measurement ", IEICE Technical Report, A-155. The literature described above describes a method of reproducing a facial expression of a person as a state reproduction method of reproducing a state when an object moves arbitrarily. Therefore, a conventional facial expression reproducing method will be described below as a conventional state reproducing method.

[0003] Communication is performed for the purpose of providing a visual interaction as if people in remote places are gathering together and providing a place where collaborative work in a virtual space is possible. This is called a “realistic communication conference”. In a system such as this real-life communication conference, an expression detected on the transmission side is converted into a person 3D model on the reception side (in a conventional expression reproduction method as a state reproduction method, a 3D wire frame model (3D Wire Frame model)). Model: WFM), which is hereinafter referred to as “3D WFM”).

That is, in order to detect the expression of a person in real time, a marker (means for capturing the expression of the person) is attached directly to the face of the person, and the movement of the marker in the face image of the face captured by a television camera is determined. Chase. Using the tracking result of this marker, 3D previously applied to the 3D image
The WFM is transformed to reproduce the expression of a person. 3D
WFM is a set of triangular patches, and 3D WFM
Is performed by driving each vertex of the triangular patch.

In the expression reproducing method as the conventional state reproducing method shown in the above-mentioned prior art document (1), 3D W is used for each range corresponding to each marker attached to the face of a person.
The motion rule that determines how to move each vertex of FM is
Created based on human judgment.

In the facial expression reproducing method as the conventional state reproducing method shown in the above-mentioned prior art document (2), three-dimensional measurement is performed in advance when several facial expressions of a person are expressed, and based on the results, Then, the facial expression of the person is reproduced by a method using an estimation matrix for converting the two-dimensional movement vector of the marker in the face image into the three-dimensional movement vector of each vertex of the 3D WFM. In this case, one of the three-dimensional movement vectors closest to the movement of the two-dimensional movement vector of the marker in the face image is selected from the three-dimensional movement vectors obtained by expressing three expressions of the person obtained by the three-dimensional measurement. Uses movement vectors.

[0007]

In the conventional facial expression reproducing method as the state reproducing method shown in the prior art document (1), as described above, the motion rule for determining how to move each vertex of the 3D WFM is a human rule. Therefore, there is a problem that the facial expression may be different by comparing the actual face of the person with the synthesized face image.

In the facial expression reproducing method as the conventional state reproducing method shown in the prior art document (2), as described above, when one vertex of the 3D WFM is moved, it is determined by the three-dimensional measurement previously performed. Since one of the three-dimensional movement vectors of the given person at the time of expression is used, the movement of the marker corresponding to an expression other than the expression whose three-dimensional measurement has been performed in advance is input. In such a case, there is no flexibility to follow the movement of the marker, and there is a problem that a natural expression of a person cannot always be reproduced.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems. Even when a person performs an expression other than the expression for which three-dimensional measurement has been performed in advance, a natural expression can be obtained by using a three-dimensional human model. It is an object of the present invention to provide a facial expression reproducing method capable of reproducing an image.

[0010] That is, an object of the present invention is to provide a state reproducing method capable of improving the reproducibility of a three-dimensional model when reproducing the state of the object after any movement.

[0011]

According to a first aspect of the present invention, there is provided a method for reproducing a state, comprising the steps of: providing a plurality of reference points; A plurality of two-dimensional vectors obtained by projecting a plurality of three-dimensional movement vectors representing the movement of a reference point of an object onto an arbitrary object provided with a plurality of feature points represented by a two-dimensional image; A plurality of grid points including grid points corresponding to a plurality of reference points of the reference target object, based on a two-dimensional movement vector representing the movement of the feature point of the arbitrary target when moving. 3 as the image of the reference object
A state reproduction method that reproduces a state when an arbitrary object makes an arbitrary movement by driving a three-dimensional model, wherein a reference object is set at each of a plurality of feature points corresponding to a plurality of reference points. From a plurality of two-dimensional vectors corresponding to a plurality of three-dimensional movement vectors when the object makes some predetermined movement, the two-dimensional movement vector is close to the two-dimensional movement vector and 2
Extracting two two-dimensional vectors sandwiching the two-dimensional movement vector; and weighting each of the two extracted two-dimensional vectors at each of a plurality of feature points corresponding to the plurality of reference points, The two-dimensional movement vector is
The two extracted two-dimensional vectors are represented by a first sum of the two extracted two-dimensional vectors.
Calculating the respective weights, and based on the weights calculated for each of the plurality of feature points corresponding to the plurality of reference points, based on the weights calculated for each of the plurality of grid points, Obtaining a plurality of vectors for determining the motion, and driving the three-dimensional model by moving the plurality of grid points based on the plurality of vectors for determining the motion of the plurality of grid points.

In the state reproducing method according to the first aspect, a two-dimensional movement vector representing an arbitrary movement of an arbitrary object is represented by:
Lattice points of a three-dimensional model expressed as a first sum of two-dimensional vectors obtained from three-dimensional movement vectors representing a predetermined movement of a reference object, and obtained based on weights calculated at that time The lattice points of the three-dimensional model are moved according to the vector that determines the movement of the three-dimensional model, and the three-dimensional model is driven.

In the state reproducing method according to the second aspect of the present invention, in the state reproducing method according to the first aspect, the step of driving the three-dimensional model includes the steps of: Weighting the two three-dimensional motion vectors calculated with respect to, calculating the second sum of the two weighted three-dimensional motion vectors, and determining the movement of the grid point as the second sum Moving the grid points of the three-dimensional model based on the vector.

According to a second aspect of the present invention, in the state reproducing method of the first aspect, a two-dimensional motion vector representing an arbitrary motion of an arbitrary object is represented by a first sum.
Two three-dimensional motion vectors corresponding to predetermined motions respectively represented by the two two-dimensional vectors are weighted when the two-dimensional motion vector is expressed by the first sum, and the two weighted two-dimensional motion vectors are used. Is obtained, and the three-dimensional model is driven by moving the lattice points of the three-dimensional model based on the vector that determines the movement of the lattice point, which is the second sum.

According to a third aspect of the present invention, there is provided the state reproducing method according to the second aspect, wherein two three-dimensional movement vectors corresponding to some predetermined movements represented by the two extracted two-dimensional vectors, respectively. Are two three-dimensional movement vectors of a reference point corresponding to a feature point indicated by a two-dimensional movement vector represented by a first sum of two extracted two-dimensional vectors.

According to a third aspect of the present invention, in the state reproducing method of the second aspect, two weighted three-dimensional motion vectors at a reference point corresponding to a feature point characterizing an arbitrary movement of an arbitrary object are used. The three-dimensional model is driven based on a vector that determines the movement of the grid point, which is the second sum.

According to a fourth aspect of the present invention, there is provided the state reproducing method according to the second aspect, wherein two three-dimensional motion vectors corresponding to some predetermined motions represented by the two extracted two-dimensional vectors, respectively. Are two three-dimensional movement vectors of a reference point near a reference point corresponding to a feature point indicated by a two-dimensional movement vector represented by a first sum of two extracted two-dimensional vectors.

According to a fourth aspect of the present invention, in the state reproducing method of the second aspect, the weighted two points at the reference point near the reference point corresponding to the characteristic point characterizing the arbitrary movement of the arbitrary object are set. The three-dimensional model is driven based on a vector that determines the movement of the lattice point, which is the second sum of the three three-dimensional movement vectors.

According to a fifth aspect of the present invention, there is provided the state reproducing method according to the first aspect, wherein an arbitrary object makes a predetermined movement with respect to a plurality of reference points and some predetermined movements. Determining the measurement condition of the three-dimensional movement vector so that the sum of squares of the error between the two-dimensional movement vector at the time and the two-dimensional vector when the reference object moves in a predetermined manner is minimized. In addition.

According to a fifth aspect of the present invention, in the state reproducing method of the first aspect, an error between the two-dimensional vector and the two-dimensional movement vector is reduced, that is, a three-dimensional measurement system (three-dimensional moving system). Vector measurement system) and 2
A process is performed so as to reduce an error with a dimensional measurement system (a two-dimensional movement vector measurement system) to drive a three-dimensional model.

In the state reproducing method according to the sixth aspect of the present invention, in the state reproducing method according to the fifth aspect, the conditions for measuring the three-dimensional movement vector are a viewpoint, a magnification, and a rotation.

According to a sixth aspect of the present invention, in the state reproducing method of the fifth aspect, an error between the two-dimensional vector and the two-dimensional movement vector is reduced, that is, a three-dimensional measuring system (three-dimensional moving Vector measurement system) and 2
The viewpoint, magnification and rotation for measuring the three-dimensional movement vector are determined so as to reduce the error from the three-dimensional measurement system (two-dimensional movement vector measurement system), and the three-dimensional model is driven.

According to a seventh aspect of the present invention, the method of the fifth or sixth aspect further includes the step of storing an error for a plurality of reference points and some predetermined movements.

According to a seventh aspect of the present invention, in the state reproducing method of the fifth or sixth aspect, the error between the two-dimensional vector and the two-dimensional movement vector is determined for a plurality of reference points and some predetermined movements. And drive the 3D model.

According to the state reproducing method of claim 8 of the present invention, in the state reproducing method of any one of claims 5 to 7,
The step of driving the three-dimensional model consists of two extracted two
Weighting two calculated three-dimensional motion vectors respectively corresponding to some predetermined motions represented by the respective dimension vectors, and calculating a second sum of the two weighted three-dimensional motion vectors; Determining two three-dimensional error vectors representing the errors of the two three-dimensional motion vectors relative to some predetermined movements respectively represented by the two extracted two-dimensional vectors, based on the errors; Weighting the calculated vector, calculating a third sum of the two weighted three-dimensional error vectors, and calculating a fourth sum of the second and third sums; , Based on the vector that determines the movement of the grid point, which is the fourth sum, 3
Moving the grid points of the three-dimensional model.

In the state reproducing method according to the eighth aspect, in the state reproducing method according to any one of the fifth to seventh aspects, a two-dimensional movement vector representing an arbitrary movement of an arbitrary object is added to the first sum. The two-dimensional motion vector is defined as a two-dimensional motion vector corresponding to a predetermined motion represented by each of the two two-dimensional motion vectors and two three-dimensional error vectors each representing an error between the two three-dimensional motion vectors. , And a second sum of the two weighted three-dimensional motion vectors and a third sum of the two weighted three-dimensional error vectors are obtained. A fourth sum of the sum of the first and third sums is obtained, and the grid points of the three-dimensional model are moved based on a vector that determines the movement of the grid point, which is the fourth sum, to drive the three-dimensional model. .

[0027]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A method for reproducing a facial expression of a person as a state reproducing method according to the present invention will be described below with reference to the drawings.

(First Embodiment) In a facial expression reproducing method as a state reproducing method according to a first embodiment of the present invention, a face is photographed by a television camera on a transmitting side, and human faces are included in the facial image. The marker attached to the face is tracked, and on the receiving side, the facial expression is reproduced by appropriately modifying the three-dimensional model of the person using the tracking result of the marker. In the first embodiment, a three-dimensional wireframe model (hereinafter, referred to as “3D WFM”) is used as a person three-dimensional model.

FIG. 1 is a diagram showing the position of a marker attached to the face of a person when detecting the expression of the person.

In FIG. 1, seven markers 2 are attached to a face 1 of a person schematically shown. Marker 2
Is a means for capturing the expression of a person, and the number and position of the markers are not limited to the case shown in FIG. When the expression of the face 1 of the person shown schematically changes, the marker 2 moves. When the marker 2 is tracked, a small camera is used.
To capture a human face as two-dimensional data from the front,
The movement amount of the marker 2 is represented by a relative two-dimensional movement vector from the time of no expression.

Next, the procedure of the facial expression reproducing method as the state reproducing method according to the first embodiment will be described in detail.

(1) Three-dimensional measurement of a face when various facial expressions of a person are displayed. FIG. 2 is a diagram showing dots drawn on the face of a person in order to perform three-dimensional measurement of the face when expressing various facial expressions of the person.

In FIG. 2, a plurality of dots 4 are drawn on a half surface of a person's face 3 schematically shown. The first
In the embodiment, the dot 4 is, for example, 170
Points.

The three-dimensional measurement of the face at the time of expressing various facial expressions of a person refers to obtaining a three-dimensional movement vector of each dot 4 at the time of expressing various facial expressions from the position of the face when there is no facial expression. . Here, assuming that the number of measured expression types is N,
N three-dimensional movement vectors are determined for one dot 4.

The number of dots 4 is larger than the number of markers 2 for detecting an arbitrary facial expression of a person's face in FIG. Using the dots 4 in FIG. 2, a person three-dimensional model, that is, a 3D WFM is created.

FIG. 3 is a diagram showing a basic three-dimensional wire frame model (3D WFM).

In FIG. 3, the basic 3D WFM represents a half face of a person's face, and is composed of a set of triangular patches 5. As a vertex of the basic 3D WFM (vertex of the triangular patch 5) in FIG. 3, a plurality of dots 4 drawn directly on the half face of the person in FIG. 2 are used. That is, the dots 4 in FIG. 2 correspond to the vertices of the basic 3D WFM in FIG.

FIG. 4 is a diagram showing a 3D WFM obtained by expanding the basic 3D WFM of FIG. The expanded 3D of FIG.
The WFM is obtained assuming that the vertices of the 3D WFM in FIG. 3 are symmetric. In addition, expanded 3D
WFM is 3D other than the vertex for which 3D measurement was performed in advance.
The movement rule for the vertices of the WFM is obtained by using a method of using weights based on distances from three vertices closest to vertices other than vertices in which three-dimensional measurement has been performed in advance.

Thus, the expanded 3D WFM becomes 1
It has 300 vertices. Therefore, the expanded 3D
The WFM is composed of a set of triangular patches 7 finer than the triangular patches 5 of the basic 3D WFM.

(2) The three-dimensional movement vector of dot 4 in FIG. 2 at the time of expressing N types of facial expressions of the person obtained in the description of (1) (dot 4 in FIG. 2 is a basic 3D WFM in FIG. 3) 3D for displaying N types of facial expressions
It can also be called a three-dimensional movement vector of the vertex of WFM. ), A marker 2 attached to the face of the person as shown in FIG.
, A three-dimensional movement vector M _hj (j = 1, 2,..., N) of the vertex h of the basic 3D WFM of FIG.
That is, N three-dimensional movement vectors M _hj are extracted for one marker.

(3) The three-dimensional movement vector M _hj extracted in the description of (2) is projected on a face image of a person photographed by a small camera (not shown), and the two-dimensional vector m _hj (j =
1, 2,..., N).

(4) When detecting an arbitrary facial expression of a person, the movement of the marker accompanying the change in the facial expression is tracked in the facial image of the person photographed by a small camera (not shown), and the two-dimensional marker is detected. Obtain the movement vector u _h .

(5) FIG. 3 obtained in the explanation of (3)
3D motion vector M of vertex h of the basic 3D WFM
N types of two-dimensional vectors m _hj obtained by projecting _hj onto a face image
Of the basic 3 of FIG. 3 obtained in the explanation of (4).
The two-dimensional movement vector u _h of the marker corresponding to the vertex h of the D WFM is sandwiched between the two, and the most two-dimensional movement vector u
Extract two two-dimensional vectors m _ha and m _hb close to _h .

Then, the two extracted two-dimensional vectors m _ha and m _hb are weighted by k _h and l _h respectively, and the two-dimensional movement vector u _h of the marker is converted into two weighted two-dimensional vectors m _h and m _{hb. Expressed} as the sum of _ha and m _hb . That is, the two-dimensional movement vector u _h of the marker is represented by the following equation.

[0045]

(Equation 1)

FIG. 5 shows the two-dimensional vector m _hj obtained in the description of (3) and the marker obtained in the description of (4) in the marker corresponding to the vertex h of the basic 3D WFM in FIG. FIG. 7 is a diagram showing a two-dimensional movement vector u _h of FIG.

In FIG. 5, the two-dimensional movement vector u _h of the marker 2 is sandwiched between two two-dimensional vectors m _ha and m _hb that project the three-dimensional movement vector m _hj onto a face image. Then, the two-dimensional movement vector u _h is a two-dimensional vector m weighted by k _h and l _h , respectively.
It is expressed as the sum of _ha and m _hb . Note that the two-dimensional vectors m _ha and m _hb are selected as the ones closest to the two-dimensional movement vector u _h among the N two-dimensional vectors m _hj .

Here, the weights k _h and l _h are calculated as follows. Marker two-dimensional movement vector u _h
= (X ₀ , y ₀ ), two-dimensional vector m _ha = (x
_a, y _a), 2-dimensional vector m _hb = (x _b, When y _b), the formula [1] is expressed as follows.

[0049]

(Equation 2)

Expression (2) can be expressed as follows with respect to each of the x coordinate and the y coordinate.

[0051]

(Equation 3)

When equations [3] and [4] are solved as simultaneous equations, k _h and l _h are expressed as follows.

[0053]

(Equation 4)

(6) Determined in the description of (1) to (5),
The three-dimensional movement vector of the dot 4 in FIG.
Based on the three-dimensional motion vector of the vertices of the D WFM) and the weights k _h , l _h , the expanded 3D WFM of FIG.
By driving each of the vertices, the facial expression when the person takes an arbitrary facial expression is reproduced.

Hereinafter, (a) the expanded 3D WFM of FIG.
(B) When the vertex of each of the extended 3D WFMs shown in FIG. 4 matches the dot and the marker does not exist on the dot, (b) 4) For three cases where there is no dot corresponding to each vertex of the expanded 3D WFM of FIG. 4, how to obtain a three-dimensional vector that determines the motion of each vertex of the expanded 3D WFM of FIG. 4 will be described in detail. .

(A) The expanded 3D W of FIG. 4 corresponding to the two-dimensional vector m _ha and the two-dimensional vector m _hb of FIG.
When 3-dimensional movement vectors each M _ha and M _hb determined in the description of the FM vertex h (2), the motion of the vertex h of 3D WFM a person extended in Figure 4 upon any expression 3-dimensional vector U _h to determine the is represented by the following equation.

[0057]

(Equation 5)

(B) Consider a region (indicated by a dotted line) represented by each of the markers 2 in FIG. 1, and for example, assume that the vertex i of the expanded 3D WFM in FIG. 4 is in the marker region corresponding to the vertex h. I do. Vertex i of the expanded 3D WFM of FIG.
The vector U _i for determining the movement, the weighting at the vertex h k _h, l _h and, 3-dimensional movement vector M _ha vertex h,
The three-dimensional movement vectors M _ia and M _ib at the vertex i, which represent the facial expressions corresponding to two of the N types of facial expressions represented by M _hb , are represented by the following equations.

[0059]

(Equation 6)

(C) As a movement rule for a vertex of the expanded 3D WFM of FIG. 4 having no vertex corresponding to a dot which has been subjected to three-dimensional measurement in advance, a triangle including the vertex inside (three dots) is used. We use weighting based on the distance from each vertex of the triangle.

FIG. 6 is a diagram for explaining a method of obtaining a three-dimensional vector for determining the motion of the vertex of the 3D WFM without a corresponding dot in the expanded 3D WFM of FIG.

In FIG. 6, 3D with three dots
Triangle ABC formed by connecting vertices A, B, and C of WFM
A vertex P without a dot exists inside. The influence (weight) of the movement that the vertex P receives from the vertices A, B, and C is obtained as follows.

[0063] feet feet drawn vertex P as the side BC from the apex A A _1, which lower the feet put an apex P as the side AC from B ₁ and vertex C from the apex B of the apex P as edge AB Is C ₁ . Weight P of influence of vertex P on vertex A
W _a , the weight PW _b of the influence of the vertex P on the vertex B, and the weight PW _c of the influence of the vertex P on the vertex C are expressed by the following equations, respectively.

[0064]

(Equation 7)

However, there is the following relationship between PW _a , PW _b and PW _c .

[0066]

(Equation 8)

As described above, the three-dimensional vector dP that determines the movement of the vertex P is represented by the following equation using the three-dimensional vectors dA, dB, and dC that determine the movement of the vertices A, B, and C.

[0068]

(Equation 9)

Here, the three-dimensional vector dA for determining the movement of the vertex A is obtained in the above-mentioned (a) and (b).
Extended 3D WFM when a person takes an arbitrary expression
Are the three-dimensional vectors U _h , U _i, etc., which determine the movement of the vertices.

As described above, (1) to (6) (a) (b) (c)
As described above, the facial expression when the person takes an arbitrary facial expression is reproduced by 3D WFM.

FIG. 7 is a diagram showing a flow of processing for reproducing the expression of a person. In FIG. 7, step S1,
S3, S5, S7 and S9 correspond to the descriptions (2), (3), (4), (5) and (6) (b) described above, respectively.
Corresponding to

In FIG. 7, in step S 1, the vertex h of the 3D WFM corresponding to the marker attached to the face of the person
, A three-dimensional movement vector M _hj (j = 1, 2,..., N) is obtained.

In step S3, the vertex h of the 3D WFM
_Is projected on the face image to _obtain a two-dimensional vector m _hj (j = 1, 2,..., N). Steps S1 and S3 are preprocessing steps.

In step S5, when a facial expression of a person is detected,
The marker is tracked in the face image to obtain a two-dimensional movement vector u _h .

In step S7, the three-dimensional movement vector M
_hj is a two-dimensional vector m _hj (1, 2,
..., N) two-dimensional movement vector u of the marker selected from
The two-dimensional movement vector u _h of the marker is represented by a vector sum using two two-dimensional vectors m _ha and m _hb sandwiching _h .

In step S9, the weight used in obtaining the vector sum of S7 is used for the vertex i of the 3D WFM of the face area represented by each marker to determine the motion of the vertex i of the 3D WFM. The vector U _i is calculated, and the 3D WFM is transformed using the vector U _i to reproduce the facial expression of the person.

FIG. 8 is a diagram showing a face image when the facial expression of a person's face is actually reproduced using the facial expression reproducing method as the state reproducing method according to the first embodiment.

The left side of FIG. 8 shows an original image when a person says “A”. The right side of FIG. 8 shows a face image when a facial expression when a person says “A” is reproduced.

As shown in FIG. 8, according to the expression reproducing method as the state reproducing method according to the first embodiment, it is possible to reproduce the natural facial expression of a person.

As described above, in the facial expression reproducing method as the state reproducing method according to the first embodiment, the two-dimensional movement vector u _h of the marker representing an arbitrary facial expression of the person's face is determined by using N types of the human face. _Is expressed as the sum of two two-dimensional vectors m _ha and m _hb obtained from N three-dimensional movement vectors M _hj representing a predetermined expression, and a weight k calculated at that time
_In order to move the vertex of the 3D WFM and drive the 3D WFM in accordance with the vector that determines the motion of the vertex of the 3D WFM obtained based on _h and lh, the facial expressions other than the facial expression previously subjected to three-dimensional measurement are used. Even in the case of input, the expression can be reproduced with good reproducibility, and the expression of a natural person can be reproduced.

(Second Embodiment) A three-dimensional object measuring apparatus (3D scanner) for measuring a three-dimensional movement vector measures around an object to be measured. Therefore, in the three-dimensional object measuring device, since a mechanical operation is involved, a two-dimensional object measuring device without a mechanical driving part (such as a camera)
And an error easily occurs. That is, a result measured by a three-dimensional object measuring device that measures a three-dimensional movement vector,
An error occurs with a result measured by a two-dimensional object measuring device (such as a camera) that measures a two-dimensional movement vector.

In the first embodiment, the expression is reproduced without considering the error. For this reason, there is a possibility that the quality of the expression reproduction quality is reduced.

The facial expression reproducing method as the state reproducing method according to the second embodiment has been made to solve such a problem, and the result of measurement by the three-dimensional object measuring device and the result of the two-dimensional object measuring device The facial expression is reproduced in consideration of the error from the result measured in the step. The cause of such an error will be described in detail.

FIG. 9 is a diagram showing a state in which dots are projected on a two-dimensional image of a human face with a marker by measurement using a three-dimensional object measuring device (hereinafter, referred to as “three-dimensional measurement”). Note that the markers and dots are the same as in the first embodiment.

In FIG. 9, the expression of the human face by measurement by the two-dimensional object measuring device (hereinafter referred to as “two-dimensional measurement”) is the same as the expression of the human face by three-dimensional measurement. That is, as described in the first embodiment, a large number of dots are drawn on the surface of a human face, and a 3D scanner (3
A three-dimensional movement vector of each dot at a predetermined facial expression of a human is obtained by a three-dimensional object measuring device). Then, a CCD camera (two-dimensional object measuring device) is used as an input system at the time of facial expression reproduction, and two-dimensional movement of each marker in the two-dimensional face image at the time of expressing the same predetermined facial expression as at the time of three-dimensional measurement Find a vector.

As described above, when the expression at the time of the two-dimensional measurement and the expression at the time of the three-dimensional measurement are the same, ideally, if the three-dimensional measurement data is projected on the two-dimensional face image, the three-dimensional measurement is performed. The position of the dot should match the position of the marker by two-dimensional measurement. However, as shown in FIG.
The position of the marker 2 obtained by the two-dimensional measurement does not match the position of the dot 4 obtained by projecting the three-dimensionally measured dot on the two-dimensional image.

This is because of the misalignment of the helmet (to which the subject wears) equipped with a CCD camera for two-dimensional measurement, the tilt of the subject's neck during three-dimensional measurement by the 3D scanner, and the subject's neck during three-dimensional measurement. This is based on an error factor such as a difference between a facial expression expressed and a facial expression expressed by the subject at the time of two-dimensional measurement.

Therefore, in the expression reproducing method as the state reproducing method according to the second embodiment, the following processing is performed. First, facial expressions in 3D measurement and 2
The three-dimensional movement vector and the two-dimensional movement vector are measured for a plurality of facial expressions with the same facial expression in the dimension measurement. Note that the three-dimensional movement vector and 2
The dimensional movement vector is the same as that described in the first embodiment.

Here, the number of facial expressions is expressed as N (j = 1,
2,..., N) and the number of dots (the number of markers) is Q (q = 1,
2,..., Q). The two-dimensional position vector obtained by projecting the three-dimensional coordinates (three-dimensional movement vector) of the dot input by the 3D scanner onto the two-dimensional face image is defined as a two-dimensional vector m _qj.
The two-dimensional position vector is defined as a two-dimensional movement vector u _qj . Note that the two-dimensional vector m _qj and the two-dimensional movement vector u _qj
Are the same as the two-dimensional vector and the two-dimensional movement vector described in the first embodiment.

For example, referring to FIG. 5, two-dimensional vector m _hj corresponds to two-dimensional vector m _qj in the second embodiment. Referring to FIG. 5, two-dimensional movement vector u
_h corresponds to the two-dimensional movement vector u _qj of the second embodiment. In FIG. 5, the two-dimensional movement vector u _h is
This is a vector when a person expresses an arbitrary expression.
In the embodiment, the two-dimensional movement vector u _qj is a vector when the same facial expression as the facial expression in the three-dimensional measurement is performed.

Note that the subscript “q” of m _qj representing a two-dimensional vector and u _qj representing a two-dimensional movement vector represents a dot (marker), and the subscript “j” represents an expression. For the number N of expressed expressions and the number Q of dots (the number of markers), the sum of squares of the error between the two-dimensional vector m _qj and the two-dimensional movement vector u _qj is expressed by the following equation.

[0092]

(Equation 10) Measurement result by 3D scanner (3D movement vector)
Is projected onto a two-dimensional face image by the CCD camera (at the time of two-dimensional projection) so that the sum of squares E of the error (hereinafter, simply referred to as “error E”) is minimized. : Hereinafter referred to as “GA”) to optimize coefficients such as “viewpoint”, “magnification”, and “rotation (tilt of subject's neck)” for measuring a three-dimensional movement vector.

In the GA, teacher data is set so that when a generation with a small error E appears, descendants are reflected.
If the error is large, the descendants are set to decrease. Here, the magnification, viewpoint, and rotation during two-dimensional projection will be described in detail.

In the second embodiment, two-dimensional measurement is performed using a general video camera (CCD camera or the like).
The three-dimensional measurement is performed using a 3D scanner. In the two-dimensional measurement and the three-dimensional measurement, the same facial expression is measured. Therefore, when the face obtained by the three-dimensional measurement and the face obtained by the two-dimensional measurement are displayed on the two-dimensional screen of the computer, they are displayed in exactly the same image. .

However, there is a significant difference between two-dimensional measurement and three-dimensional measurement. A face image on which two-dimensional measurement has been performed cannot be viewed from the side or top and bottom, and only an image viewed from the front is displayed. On the other hand, when three-dimensional measurement is performed, the face can be displayed on the screen from an arbitrary viewpoint by using the data. For example, a face image viewed from the side, a face image viewed from above, and a face image viewed from below can be arbitrarily displayed.

[0096] The above is more than a human face.
It is easy to understand when you look at cylinders and half cylinders. This will be described with reference to the drawings.

FIG. 10 shows a “viewpoint” during two-dimensional projection.
FIG. 4 is a diagram for explaining “magnification” and “rotation”.

Referring to FIG. 10, when photographing half cylinder 9 from the front (in the direction of arrow b), it cannot be distinguished from the cylinder as shown in screen B (corresponding to two-dimensional measurement). But,
When three-dimensional measurement is performed, since it is possible to move to an arbitrary "viewpoint", display (screen C1) from right beside (in the direction of arrow c) becomes possible, and discrimination between a cylinder and a semi-cylinder can be made. Is optional. Similarly, display (screen A) from above (in the direction of arrow a) by moving the “viewpoint” becomes possible when three-dimensional measurement is performed.

As described above, the “viewpoint” refers to a three-dimensional object captured by a video camera (regardless of the two-dimensional measurement described above.
This is simply an appearance for explaining the “viewpoint”), and corresponds to the position of the video camera when shooting is performed. As for "rotation", a three-dimensional object is photographed with a video camera (independently of the two-dimensional measurement described above, which is merely used for explaining "rotation") and displayed. The meaning can be understood well when assuming a case where the displayed screen is viewed with the head tilted. That is, if the video camera is tilted instead of tilting the head to the right, an oblique image (screen C2) can be seen. The screen C2 is a case where the video camera is tilted from the “viewpoint” from the side (in the direction of the arrow c). The screen C1 is a “viewpoint” from the side, but the video camera is not tilted.

As described above, "rotation (tilt of the subject's neck)" corresponds to the tilt (degree of rotation) of the video camera when a three-dimensional object is photographed by the video camera.

The term "magnification" means that a three-dimensional object is simply referred to as a "magnification" regardless of the two-dimensional measurement described above.
Is introduced to explain the above). That is, by changing the “magnification”, a distant object can be displayed small and a near object can be displayed large.

In other words, when projecting a three-dimensional movement vector based on three-dimensional measurement onto a two-dimensional face image based on two-dimensional measurement (for example, when a three-dimensional object is
Irrespective of the dimension measurement, this is merely an example for the purpose of explanation) and is equivalent to the same processing as the two-dimensional conversion when the image is taken and displayed on the screen), and the error E is minimized. In the execution of GA, "viewpoint" in three-dimensional measurement,
Determining “magnification” and “rotation” means, for example, that a three-dimensional object can be viewed from where (viewpoint) and at what distance (magnification) by the subject's neck (or a video camera (as described above). Irrespective of the measurement, it is simply introduced for explanation)), how much the camera is tilted (rotated) and photographed (projected on a two-dimensional screen), This corresponds to determining whether or not the measurement is the same as a two-dimensional image captured by a video camera (CCD camera or the like).

That is, in the execution of the GA, the face image of the two-dimensional measurement using the video camera and the face image of the three-dimensional measurement projected on the face image subjected to the two-dimensional measurement are made to look the same (2). Distance (magnification) so that the difference between the position of the marker drawn on the face when performing three-dimensional measurement and the position of the dot drawn on the face when performing three-dimensional measurement is minimized) ) Α, the inclination (rotation) β of the subject's neck (or a video camera (which has nothing to do with the above-described two-dimensional measurement and is merely used for explanation)), and the point / position to be viewed (viewpoint) ) D _X, calculating the D _Y.

[0104] As described above, in the facial expression reproduction method according to the second embodiment, projecting a 3-dimensional motion vector in a two-dimensional image, when obtaining the two-dimensional vector m _qj, two-dimensional vector m _qj and 2 The “viewpoint”, “magnification”, and “rotation” in three-dimensional measurement are determined by the GA so that the error with the dimensional movement vector u _qj is reduced. For this reason,
Expression reproduction quality is improved as compared with the first embodiment in which the error E is not taken into account.

In calculating the parameters (“viewpoint”, “magnification”, “rotation”) for minimizing the error E, a method such as a neural network and a minimum error square method having the same effect as GA is used. Can also be used. Also in this case, the expression reproduction quality is improved as compared with the first embodiment.

As described above, even if the parameters (“viewpoint”, “magnification”, “rotation”) are optimized,
3D object measurement device (3D scanner) that performs 3D measurement
Object measuring device (CCD)
Cameras such as cameras) each include an accuracy error, and it is extremely difficult to set the error E to zero.

Therefore, in the expression reproducing method according to the modified example of the second embodiment, the three-dimensional measuring system and the two-dimensional measuring system are performed as follows based on the input from the CCD camera which performs the two-dimensional measurement with a small accuracy error. Minimize the error of the dimensional measurement system. A marker corresponding to the vertex h of the 3D WFM (a marker of the face image by two-dimensional measurement)
The two-dimensional vector m _hj (FIG. 5)
(Corresponding to the two-dimensional vector m _hj of). The suffix “h” of the two-dimensional vector m _hj indicates that it corresponds to the vertex h of the 3D WFM, and the suffix “j” indicates an expression expression in three-dimensional measurement.

Here, a three-dimensional measured expression expression
(Basic expression) Two-dimensional movement vector of the intermediate expression between a and b
x_hIs a two-dimensional vector m of the basic expression a_ha(Secondary in FIG. 5
Source vector m_haAnd a two-dimensional vector of the basic expression b
m_hb(The two-dimensional vector m in FIG. 5 _hb) And
It is expressed as below.

[0109]

[Equation 11] In Expression [13], k _h and l _h are weights corresponding to the weights k _h and l _h in Expression [1] of the first embodiment. Note that the procedure for obtaining equation [13] is the same as that in the first embodiment, and the procedure for obtaining the weights k _h and l _h is also the same as in the first embodiment.
That is, the procedure up to this point of the facial expression reproducing method according to the second embodiment is that the “viewpoint”, “magnification”, and “rotation” are determined at the time of two-dimensional projection so as to minimize the error E as described above. Except for the above, it is the same as the first embodiment.
Using the weights k _h and l _h in equation [13], 3D
Determine the motion of vertex h of WFM. This will be described in detail.

[0110] Formula [13] of the 3-dimensional movement vector M _hb corresponding to the three-dimensional movement vector M _ha and 2-dimensional vector m _hb corresponding to the two-dimensional vector m _ha, to each weighted k _h and l _h, the The following equation is obtained by taking the sum.

[0111]

(Equation 12) Equation [14] corresponds to equation [7] in the first embodiment. That is, the procedure up to this point is the same as the procedure for obtaining equation [7] in the first embodiment. Note that subscripts 3-dimensional movement vector M _ha "h" indicates the vertex h of 3D WFM, subscript "a" indicates that the expression expression (basic facial expression) a was subjected to three-dimensional measurement. The same _{applies to the} three-dimensional movement vector M _hb .

Consider a three-dimensional error vector remaining as a result of GA. A three-dimensional moving vector M 3 dimensional error vector S _ha and 3-dimensional movement vector M 3 dimensional error vector S _hb of _hb of _ha, respectively weighted k _h and l _h, taking the sum, the following expression is obtained.

[0113]

(Equation 13) The subscript “h” of the three-dimensional error vectors S _ha and S _hb
a ”and“ hb ”are three-dimensional movement vectors M _ha and M _hb
Are the same as suffixes “ha” and “hb”. In GA, and the three-dimensional vector into a two-dimensional vector, performs the processing to obtain the three-dimensional error vector S _ha and S _hb by stepping on procedure in reverse.

That is, a three-dimensional error vector is obtained based on the error (m _qj -u _qj ) between the two-dimensional vector and the two-dimensional movement vector in equation [12].

3 Determines the Motion of Vertex h of 3D WFM
The dimensional vector X _h is calculated by using the vector shown in Expression [14] and
It is represented by the sum with the vector shown in Expression [15], and is as follows.

[0116]

[Equation 14] Based on the three-dimensional vector X _h represented by the formula [16], 3
The top of the D WFM is driven to reproduce the expression when a person expresses an arbitrary expression.

FIG. 11 illustrates equation [16]. Referring to FIG. 11, a vector Y _h indicates a vector represented by Expression [14]. That is, the vector Y _h is _obtained by weighting the three-dimensional movement vectors M _ha and M _hb by weights k _h and l _h , respectively, and taking the sum.

[0118] Vector S _h shows a vector represented by the formula [15]. That is, the vector _Sh is 3
_Weight k for the dimensional error vectors S _ha and S _hb respectively
the _h and l _h, is obtained by taking the sum. Vector S _h would that 3-dimensional error vector of the vector Y _h. Vector X _h that determines the movement of the vertex h of 3D WFM is represented as the sum of the vectors Y _h and the vector S _h.

FIG. 12 is a diagram showing a comparison between the expression reproduction result by the expression reproducing method according to the first embodiment and the expression reproduction result by the expression reproducing method according to the modification of the second embodiment. is there.

Referring to FIG. 12, FIG.
7 shows the results of the facial expression reproducing method according to the embodiment. FIG.
(B) shows the result of the facial expression reproducing method according to the first embodiment. As can be seen from FIG. 12, it can be seen that the reproduction quality of facial expressions is further improved in the second embodiment as compared with the first embodiment.

As described above, in the facial expression reproducing method according to the modified example of the second embodiment of the present invention, the “magnification”, Determine “viewpoint” and “rotation”, and
Considering the remaining three-dimensional error vector as a result of A, 3D
The vertex of the WFM is driven to reproduce the expression when the person expresses an arbitrary expression. That is, the error between the three-dimensional measurement system and the two-dimensional measurement system is minimized, and in the portion where the error cannot be removed, the facial expression is reproduced in consideration of the error when reproducing the facial expression.

For this reason, the facial expression reproducing method according to the modified example of the second embodiment of the present invention is further different from the facial expression reproducing method according to the first and second embodiments in reproducing the facial expression. The quality can be improved.

The difference between the expression reproducing method according to the second embodiment (a modification of the second embodiment) and the expression reproducing method according to the first embodiment is as follows. In the expression reproducing method according to the second embodiment (a modification of the second embodiment), the expression is reproduced in consideration of an error between the three-dimensional measurement system and the two-dimensional measurement system. In the expression reproducing method according to the first embodiment, this error is not considered. Otherwise, the first embodiment is the same as the second embodiment (a modified example of the second embodiment).

[0124]

As described above, in the state reproducing method according to the first aspect of the present invention, a two-dimensional movement vector representing an arbitrary movement of an arbitrary object is represented by a predetermined movement of a reference object. Expressed as the first sum of the two-dimensional vector obtained from the three-dimensional movement vector, and obtained based on the weights calculated at that time, the three-dimensional model when an arbitrary object makes an arbitrary movement In order to move the lattice points of the three-dimensional model in accordance with the vector that determines the movement of the lattice points and to drive the three-dimensional model, any motion other than the predetermined motion for which the three-dimensional movement vector is determined in advance is optional. Even when you do,
The reproducibility of the state of any object after the movement is improved.

In the state reproducing method according to the second aspect, two three-dimensional vectors respectively corresponding to predetermined motions represented by two two-dimensional vectors each representing a two-dimensional motion vector representing an arbitrary motion of an arbitrary object as a sum. The motion vector is weighted for use in expressing the two-dimensional motion vector as a first sum, a second sum of the two weighted three-dimensional motion vectors is obtained, and a grid that is the second sum is obtained. Since the three-dimensional model is driven by moving the grid points of the three-dimensional model based on the vector that determines the motion of the point, any object other than the predetermined motion for which the three-dimensional movement vector is determined in advance has been moved. Even at this time, the reproducibility of the object in any state after the movement is improved.

According to a third aspect of the present invention, in the state reproducing method, a three-dimensional motion vector represented by a second sum of two weighted three-dimensional motion vectors at a reference point corresponding to a feature point characterizing an arbitrary motion of an arbitrary object. Since the three-dimensional model is driven based on the vector that determines the movement of the lattice points of the model, the state of any object after any movement can be reproduced naturally.

In the state reproducing method according to the fourth aspect, the second of the two weighted three-dimensional movement vectors at a reference point near a reference point corresponding to a feature point characterizing an arbitrary movement of an arbitrary object. Since the three-dimensional model is driven based on the vector that determines the movement of the lattice points of the three-dimensional model represented by the sum, even after the arbitrary movement of any object, even when the number of feature points is smaller than the number of reference points Can be reproduced naturally.

According to the state reproducing method of the fifth aspect, the error between the two-dimensional vector and the two-dimensional movement vector is reduced, that is, the three-dimensional measurement system (three-dimensional movement vector measurement system) and the two-dimensional measurement vector are used. In order to drive the three-dimensional model by performing processing to reduce the error with respect to the system (measurement system of the two-dimensional movement vector), an arbitrary object of any object can be further compared with the state reproduction method of claim 1. The reproducibility of the state is improved.

In the state reproducing method according to the sixth aspect, the error between the two-dimensional vector and the two-dimensional movement vector is reduced, that is, the three-dimensional measurement system (three-dimensional movement vector measurement system) and the two-dimensional measurement vector In order to reduce the error with the system (measurement system of two-dimensional movement vector), the viewpoint, magnification, and rotation are determined for the measurement of the three-dimensional movement vector, and the three-dimensional model is driven. Compared with the state reproducing method of the first aspect, the reproducibility of an arbitrary state of an arbitrary object is improved.

According to the state reproducing method of the present invention, an error between a two-dimensional vector and a two-dimensional movement vector is stored for a plurality of reference points and some predetermined movements. The three-dimensional model can be driven, and the reproducibility of an arbitrary state of an arbitrary object can be improved as compared with the state reproducing method of the fifth aspect.

According to the state reproducing method of the present invention, a two-dimensional motion vector representing an arbitrary motion of an arbitrary object is converted into a two-dimensional motion vector corresponding to a predetermined motion represented by each of the two two-dimensional vectors expressing the first sum. The two three-dimensional motion vectors and the two three-dimensional error vectors representing the errors between the two three-dimensional motion vectors are weighted when the two-dimensional motion vector is expressed by the first sum. A second sum of the three-dimensional motion vectors and a third sum of the two weighted three-dimensional error vectors are obtained, and a fourth sum of the second sum and the third sum is obtained. Since the three-dimensional model is driven by moving the lattice points of the three-dimensional model on the basis of the vector that determines the movement of the lattice point, which is the sum of the four, the arbitrary object is further compared with the state reproduction method of claim 5. Any state of It is possible to improve the reproducibility.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a position of a marker attached to a person's face when a facial expression of a person is detected in the first embodiment.

FIG. 2 is a diagram illustrating positions of dots drawn on a person's face in order to perform three-dimensional measurement of the face at the time of expressing various facial expressions of the person's face in the first embodiment.

FIG. 3 shows a basic 3D WF in the first embodiment.
It is a figure showing M.

FIG. 4 shows the basic 3D of FIG. 3 in the first embodiment.
It is a figure showing 3D WFM which expanded WFM.

FIG. 5 shows a basic 3D WF in the first embodiment.
The two-dimensional movement vector u _{h of} the marker corresponding to vertex h of M
FIG. 9 is a diagram showing a two-dimensional vector m _hj obtained by projecting a three-dimensional movement vector obtained by three-dimensional measurement of a person's face onto a person's face.

FIG. 6 is an expanded view of FIG. 4 according to the first embodiment;
FIG. 9 is a diagram for explaining a method of determining the movement of a vertex without a dot corresponding to a marker in D WFM.

FIG. 7 is a diagram showing a flow of processing for reproducing a facial expression of a person in the first embodiment.

FIG. 8 is a diagram showing a face image when a facial expression of a person is reproduced using a facial expression reproducing method as a state reproducing method according to the first embodiment.

FIG. 9 is a diagram illustrating a state in which dots obtained by three-dimensional measurement are projected on a face image obtained by two-dimensional measurement of a human face with a marker.

FIG. 10 is a diagram for describing “viewpoint”, “magnification”, and “rotation” optimized by GA in the second embodiment.

FIG. 11 is a diagram illustrating a procedure for obtaining a vector for determining the motion of the vertex h of the 3D WFM in the second embodiment.

FIG. 12 is a diagram showing a comparison between a facial expression reproduction result of the facial expression reproducing method according to the first embodiment and a facial expression reproduction result of a facial expression reproducing method according to a modification of the second embodiment;

[Explanation of symbols]

 1,3 Face of a person schematically shown 2 Marker 4 Dot 5,7 Triangular patch 9 Semicircular cylinder

Continuing from the front page Patent application Article 30 (1) is applied. IEICE 1995 General Conference Proceedings Proceedings, Boundary, A-252, "Examination of facial expression reproduction method based on 3D measurement" , March 1995 (72) Inventor Noriko Suzuki 5th place, Sanraya, Inaya, Seika-cho, Kyoto, Japan Town No.5, Hiratani, Sanriya, A.T. A.T.R. Communication System Research Laboratories Co., Ltd. (56) References JP-A-8-96162 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB) G06T 1/00-17/50 H04N 7/13-7/15

Claims (8)

(57) [Claims] 1. A plurality of three-dimensional movement vectors representing the movement of the reference point of the reference object when the reference object provided with a plurality of reference points makes some predetermined movements. A plurality of two-dimensional vectors projected on an arbitrary object provided with a plurality of feature points represented by a two-dimensional image, and the arbitrary object moves when the arbitrary object moves, On the basis of the two-dimensional movement vector representing the motion of the feature point, the image of the reference target object consisting of a plurality of grid points including a plurality of grid points corresponding to the plurality of reference points of the reference target object is used as an image. A state reproduction method for reproducing a state when the arbitrary object makes the arbitrary movement by driving a three-dimensional model, wherein in each of the plurality of feature points corresponding to the plurality of reference points, The reference object is the front From among the plurality of two-dimensional vectors corresponding to the plurality of three-dimensional motion vectors when performing some predetermined motion, close to the two-dimensional motion vector, and sandwiching the two-dimensional motion vector, Extracting the two two-dimensional vectors; and weighting each of the two extracted two-dimensional vectors at each of the plurality of feature points corresponding to the plurality of reference points, thereby performing the two-dimensional movement. Calculating respective weights for the two extracted two-dimensional vectors so that the vector can be represented by a first sum of the two extracted two-dimensional vectors; Based on the weights calculated at each of the corresponding plurality of feature points, the movement of the plurality of grid points when the arbitrary object performs an arbitrary movement Determine a plurality of vectors to determine, by the movement of said plurality of said plurality of grid points based on the vector to determine the movement of the plurality of lattice points, and a step of driving the three-dimensional model, state reproduction method. 2. The step of driving the three-dimensional model, wherein the two three-dimensional motion vectors corresponding to the some predetermined motions respectively represented by the two extracted two-dimensional vectors are calculated. Weighting; determining a second sum of the weighted two-dimensional motion vectors; and determining the second sum of the two-dimensional motion vectors based on a vector that determines the movement of the lattice point. Moving the grid points. 3. The two three-dimensional motion vectors corresponding to the some predetermined movements respectively represented by the two extracted two-dimensional vectors are: a first one of the two extracted two-dimensional vectors. 3. The state reproduction method according to claim 2, wherein the three-dimensional movement vectors of the reference point corresponding to the feature points indicated by the two-dimensional movement vector represented by the sum are three. 4. The two three-dimensional motion vectors corresponding to the some predetermined motions respectively represented by the two extracted two-dimensional vectors are: a first one of the two extracted two-dimensional vectors. The state reproduction method according to claim 2, wherein there are two three-dimensional movement vectors of the reference point near the reference point corresponding to the feature point indicated by the two-dimensional movement vector represented by a sum. 5. The two-dimensional movement vector when the arbitrary object makes the predetermined movement with respect to the plurality of reference points and the some predetermined movements, and the reference object 2. The state according to claim 1, further comprising: determining a measurement condition of the three-dimensional movement vector such that a sum of squares of an error with the two-dimensional vector when the predetermined movement is performed is minimized. How to reproduce. 6. The state reproduction method according to claim 5, wherein the conditions for measuring the three-dimensional movement vector are a viewpoint, a magnification, and a rotation. 7. The state reproduction method according to claim 5, further comprising the step of storing the error for the plurality of reference points and the some predetermined movements. 8. The step of driving the three-dimensional model, wherein the two three-dimensional motion vectors corresponding to the some predetermined motions respectively represented by the two extracted two-dimensional vectors are calculated. Weighting; calculating a second sum of the weighted two three-dimensional motion vectors; and determining the predetermined number of the predetermined two-dimensional vectors represented by the two two-dimensional vectors based on the error. Obtaining two three-dimensional error vectors representing an error between the two three-dimensional motion vectors relative to motion; applying the calculated weights to the two three-dimensional error vectors; Obtaining a third sum of two three-dimensional error vectors; and obtaining a fourth sum of the second sum and the third sum The method according to any one of claims 5 to 7, further comprising: moving the grid point of the three-dimensional model based on a vector that determines the movement of the grid point that is the fourth sum. State reproduction method.