JP2003058906A - Figure animation creation system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate animation by estimating three-dimensional posture of a virtual figure from a two-dimensional line drawing drawn on a story board. SOLUTION: The three-dimensional posture is automatically estimated from the two-dimensional line drawing described on the story board. And continuous three-dimensional operations are estimated and generated from picture scripts of intermittent motions described on the story board by using previously stored sample motions. In addition, expression of timing or the like of dialogue among a plurality of virtual figures is enabled by adding a function that describes context on a time axis to the story board. Furthermore, support of collaborative works in the case of creating CG (computer graphics) is also enabled by sharing the story board by a plurality of CG animators.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving image generation system capable of generating a person moving image from a two-dimensional line drawing, and more particularly to a moving image generation system capable of describing a front-back relation on a time axis. .

[0002]

[Background Art] In the field of production of multimedia contents such as entertainment and education in recent years, the number of contents including interaction between multiple virtual persons and interaction between virtual persons by CG (computer graphics) is increasing. is doing. However, it is extremely difficult to describe and generate various interactions by a motion generation method such as a key frame animation used in conventional CG software or a motion description by a CG programming language. When creating a CG animation,
A general method is to specify intermittent motions by key frames and interpolate motions between frames by a simple method such as a spline curve. In order to generate a natural motion by such a method, it is necessary to set a large number of key frames, and skill is also required. In addition, since it is necessary to directly handle the movement in frame units, it is necessary to consider separately the method for assembling long movements along the story.

[0003] At the production site of CG animation, a paper storyboard is used as a tool for fixing a vague image of a work and integrating various ideas. In addition to dialogue, the storyboard describes the situation by writing in totos and various designation items (such as designation of color and movement) when creating animation. It is also used as a place for multiple animators to share images. The problem with paper storyboards, however, is that they are separate from the process of creating animations. Therefore, when creating an animation, it is necessary to recreate it from the beginning. In addition, interactive operations such as rewriting and editing during work cannot be performed.
As a method of describing a dialogue of a virtual person, it is general to describe a motion by a special script language such as TVML (TV program making language). However, it is difficult to express various actions only by the abstract description in the language. If you describe a detailed operation in an abstracted language, you will eventually need to describe it at the numerical level like a programming language.

[0004]

SUMMARY OF THE INVENTION An object of the present invention is to estimate a three-dimensional posture of a virtual person from a two-dimensional line drawing drawn on a storyboard and generate a three-dimensional animation.

[0005]

In order to achieve the above-mentioned object, the present invention is a moving image generation system capable of generating a moving image from a two-dimensional line drawing of a person. The storyboard description section has a data accumulation section that accumulates a plurality of motion segments, which are three-dimensional posture parameters for a series of motions, and describes a two-dimensional line drawing of a person to create a moving image. From the two-dimensional line drawing input in the storyboard description section,
A posture estimation unit that estimates a three-dimensional posture parameter, and a motion segment that is applied to the plurality of posture parameters from the data storage unit to interpolate between postures based on the plurality of posture parameters from the posture estimation unit. A motion generation unit that generates a motion by interpolating the motion and an image display unit that displays the generated motion. With the above configuration, it is possible to generate a moving image from a two-dimensional line drawing of a person described on a storyboard.

The motion generation unit projects a plurality of posture parameters obtained by the posture estimation unit onto an eigenspace, and selects a motion segment having the smallest distance from each point in the projected eigenspace. In addition, the storyboard description section can describe the relationship on the time axis, and further,
A plurality of storyboard description sections are provided, and the storyboards described by each storyboard description section can be stored in a common storage section. Further, the display state can be changed so as to show the progress state of the accumulated storyboard.

It is possible to further include a motion input unit for inputting a three-dimensional posture parameter in which a person is moving and dividing the posture parameter into motion segments and storing the motion segments in the data storage unit. The division of the three-dimensional posture parameter in the movement input unit is performed by projecting the posture parameter onto an eigenspace to generate a movement curve, and dividing the movement curve according to the minimum description criterion to obtain a movement segment.

It should be noted that a computer program for configuring the moving image generation system in the computer system and a recording medium recording the computer program are also the present invention.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION <Overview>
It is a system that generates a 3D CG animation from a 2D storyboard, and automatically estimates a 3D posture from a 2D line drawing described on a storyboard. Then, it is possible to generate a continuous three-dimensional motion by estimating from the intermittent motion described in the storyboard from the data of the sample motion that is sampled and accumulated in advance. In addition, by adding a function to describe the context on the time axis to the storyboard, the timing of dialogue between a plurality of virtual persons can be expressed. In addition, multiple C
By sharing the storyboard among G animators, it becomes possible to support joint work when creating CG. Hereinafter, embodiments of the system described above will be described in detail with reference to the drawings.

<Structure> FIG. 1 is a diagram showing the functional structure of a human moving image generation system according to the present invention. Operation signal input section 12
And a motion input unit 10 including a motion separation unit 14, a storyboard description unit 20 including storyboards 22, 24, 26, a three-dimensional posture estimation unit 32, and a three-dimensional motion interpolation unit 34.
It is composed of a motion generation unit 30, a data storage unit 40, and an image display unit 50. The motion input unit 10 inputs the three-dimensional posture of the sample motion using the motion signal input unit 12, and uses the motion separation unit 14 to automatically divide the input sample motion. The input data storage unit 40 stores parameters of each joint of the virtual person. The storyboard description unit 20 is used to describe a two-dimensional sketch of a scene in which the user wants to create a CG on the storyboards 22, 24 and 26. This is done, for example, by inputting in a storyboard field from a pointing device such as a mouse or a tablet. Motion generator 3
In 0, the 3D pose estimation unit 32 is used to estimate the 3D pose from the 2D pose, and the 3D motion interpolation unit 34 performs interpolation based on the sample motion. And the generated C
The G animation is displayed on the image display unit 50. next,
The operation process of the system of the present invention will be described for each process.

<Calculation of 3D pose from 2D pose> The first step to generate an animation from a storyboard is to estimate the pose of a 3D model from a 2D picture control. Regarding the method of estimating the three-dimensional posture from the storyboard of the two-dimensional posture described on the storyboard, a patent application filed by the inventor of the present invention is "moving image generation system (Japanese Patent Application No. 2001-6385).
4) ”described in paragraph [0010] and thereafter <
Estimation of human motion based on model> is used. The details will be described below.

In the motion generator 30, a posture is estimated by fitting a three-dimensional model of the human body. FIG. 2 shows the virtual person model 100 used in the motion generation unit 30. As shown in FIG. 2, the virtual human model 100 represents each part as an object and connects them to represent the whole. This virtual person model 100 has a head 10
1, body 102, upper right arm 103, upper left arm 104, right forearm 105, left forearm 106, right thigh 107, left thigh 108, right knee below 109, left knee below 110 (each object)
It consists of. The motion generation unit 30 estimates the posture by using such a virtual person model 100. The connection relationship of each part (each object) of the virtual human model 100 can be defined as a parent-child connection toward the terminal parts such as the arms and the head with the body 102 as the apex. When the parent part moves, the part corresponding to the child moves together in the connection relation defined by the coordinate system of the parent part. This virtual person model 100 will be described below. FIG. 3 is a diagram showing the relationship between the line drawing on the storyboard and the three-dimensional human model in the camera coordinate system and the world coordinate system. In this figure, the moving image is displayed on the image plane 150 described in the camera coordinate system. An arbitrary point on the virtual human model 100 is

## EQU00001 ## If p = (x, y, z) ^T ( ^T in the above equation, which is attached to the shoulder of the bracket, represents a transposed matrix), and a point on the image plane 150 projected onto the camera coordinate system ( X, Y) is

## EQU2 ## X = fx / z Y = fy / z (f is the focal length of the camera that generates the image plane 150).

(Posture estimation) In the following, extended Kalman
A method of matching the centerline extracted from the silhouette image with the body axis of the human model using a filter is described. An equation representing the posture a of each part of the virtual human model 100 in the world coordinate system is shown below.

Equation 3] a = [r, d] T d is a vector representing the movement component from the origin of the world coordinate. Further, r is a vector along the rotation axis that is each coordinate axis of world coordinates, and the magnitude of the vector is equal to the rotation angle.

FIG. 4 is a diagram showing the relationship between the line segments on the screen and the corresponding parts in the virtual human model 100. Image plane 1 on which a three-dimensional (3D) line segment P is projected
The two-dimensional (2D) predicted line segment on 50 is Q. The two-dimensional (2D) observation line segment Q ′ and the 2D predicted line segment Q in FIG.
Vectors q ₁ ′, q ₂ ′, q ₁ and q in the world coordinate system that represent the start and end points on the two-dimensional image plane 150.
_The formula represented by ₂ is shown below.

Q ′ = (q ₁ ′, q ₂ ′) Q = (q ₁ , q ₂ ) Further, the 3D line segment P is represented by vectors p ₁ and p representing the start and end points of the line segment in the world coordinate system. _The formula represented by ₂ is shown below.

Equation 5] _{P = (p 1, p 2} ) Then, 2D observed was observed at the initial position line segment Q 'and 3D
The set of line segments P is represented by the following formula.

## EQU00006 ## l = (P, Q ') A method for obtaining a set l of the 3D line segment P and the corresponding 2D observation line segment Q'on the image plane 150 will be described below with reference to FIG. To do. FIG. 5 is a diagram showing a process of estimating the initial position of the human model from the line drawing of the storyboard described on the storyboard.

(Estimation of the center line) When estimating the center line from the two-dimensional line drawing input from the storyboard, the posture cannot be estimated when parts such as the hands and feet are hidden by the body, so the storyboard cannot be estimated. This is performed using the silhouette image of the virtual person described in 1. The center line is estimated by the following procedure.

A silhouette image having a posture in which the center line is easily extracted is selected in advance and stored in the data storage section 40. At this time, the posture parameter of the human model corresponding to the silhouette image is also given. The posture parameters can be finely adjusted by the method described below,
A rough value is fine. As shown in FIGS. 5 (a), (i) and (ii), the input image sequence from the storyboard description unit 20 is matched with the silhouette image stored in the data storage unit 40 in advance, and the postures are similar. Select the frame you are viewing. In the selected silhouette image, since the rough posture of the human model is known, the area of each part can be projected on the image plane 150. The silhouette in each area is cut out, and the main axis is obtained and used as the center line. Figure 5
(B) (i) and (ii) show images displayed by applying the human model to the input image and extracting the center line of each part.

From the above method, the extracted center line can be used to obtain the set 1 of the 3D line segment P and the corresponding 2D observation line segment Q'on the image plane 150, which will be described later. With this, it is possible to accurately estimate the posture of the human model.

(Adjustment of Attitude) When the attitude a of each part is accurate, 3D is placed on the plane M formed by the 2D observation line segment Q'on the image plane 150 obtained from the observed image and the origin O of the camera.
A line segment P is included. However, when there is an error in the posture a of each part, as shown in FIG. 4A, planes M and 3 formed by the line segment Q ′ displayed on the image plane 150 and the origin O of the camera are set.
A distance occurs between the D line segments P. The shortest distance of this distance is
The prediction error is defined as h (a, l) and is expressed as the following equation.

[Equation 7] h (a, l) is a 2 × 1 vector. R is a 3 × 3 rotation matrix derived from r. In addition, n is a unit normal vector of the surface M

[Equation 8] Is. When a has no error, h (a, l) = 0, and the 2D predicted line segment Q on the image plane 150 overlaps with the 2D observed line segment Q ′ as shown in FIG. 4B. Then, the 3D line segment P is included on the plane formed by the 2D observation line segment Q ′ and the origin of the camera. This means that the posture a of the 3D line segment P at this time corresponds to the 2D observation line segment Q ′ displayed on the image plane 150. h (a, l) is so nonlinear and linear approximation by the observed value l = ^ _{l i} and the estimated value a = _^ 1 primary Taylor's expansion around _{a i-1.}

[Equation 9] However, ∂h / ∂a and ∂h / ∂l represent partial differentiation. These partial differential coefficients are used when the following Kalman filter is applied to obtain the optimum value of the posture a = [r, d] ^T of each part of the human model.

2D observation line segment Q'and 3D with correspondence
The human body posture a is corrected so that the error h (a, l) of the line segment P is minimized, and the position error distribution σ _xi is reduced. When the linear approximation formula of h (a, l _i ) shown immediately before is rewritten with zero, the following observation equation is obtained.

Where z _i = H _i a + ν _i

[Equation 11] Is. Here, the variance B _i of ν _i is obtained as follows.

[Equation 12]

From the above observation equation, the estimated value that minimizes the squared error of the posture x _i of the human body part is given by the Kalman filter.

[Equation 13] The human body posture a _i and the error distribution σ _i are updated each time a new line segment is associated. By applying the above posture estimation to each part of the human body, the posture in the three-dimensional model of the two-dimensional line drawing on the storyboard is estimated. In the system of the present invention, the posture of the torso is first obtained, and the upper arm, the upper leg, the lower arm, and the lower leg are matched with the center line in the order close to the torso to estimate the posture. In this way, FIG.
It is possible to estimate the initial posture in the input images of (a), (i) and (ii), and obtain an image in which the human model is superimposed as shown in FIGS. 5 (c), (i) and (ii).

(Association of 2D and 3D Line Segments) In the above-described method, the set 1 of the 2D observation line segment Q'corresponding to the 3D line segment P is utilized by utilizing the extracted center line by matching the silhouette images. I explained that I asked for it. However, if there are multiple line segments that can be extracted as the center line in the shooting environment,
There are a plurality of 2D line segment candidates for one 3D line segment. Therefore, the following two methods are used together to perform the association. First, when viewed on the 2D image surface 150,
A line segment whose projected image overlaps with the 2D observation line segment Q ′ when projected onto the 2D observation line segment Q ′ is selected as a candidate. Next, taking into consideration the error distribution of the estimated value and the error distribution of the 2D and 3D line segments, a line segment that can be determined to statistically match is selected.

Correspondence on a two-dimensional screen A procedure for making a correspondence on the two-dimensional image plane 150 using FIG.
Show the law. First, a 3D line segment P is drawn as shown in FIG.
2D prediction line segment Q = (q₁, Q _Two)
Meru. Also, from the observation image, the 2D observation line segment Q '=
(Q₁’, Q_Two’) For 2D observation line segment Q '
When the projected image of the projected 2D predicted line segment Q has an overlapping region
Are selected as correspondence candidates. Overlap area determination method and
Then, the 2D predicted line segment Q is projected onto the 2D observed line segment Q '.
Sometimes q₁Or q_TwoEither point is inside Q '
, The projection image of the 2D prediction line segment Q is the 2D observation line segment Q ′.
Can be considered as overlapping. First, q₁To
The coordinate c when projected onto Q'is the unit direction vector along Q '.
Cuttle

Equation 14] _{m = N [q 2 '-q} 1'] (N [] represents the normalized vectors) When,

(15) c = q ₁ ′ + m · (q ₁ −q ₁ ′) m. And the condition that c is included inside Q'is

## EQU16 ## (c−q ₁ ′) · (c−q ₂ ′) <0. This operation is performed for q ₁ , q ₂ , q ₁ ′, q ₂ ′, and line segments that do not satisfy the above conditions are rejected.

Corresponding human body posture error distribution σ _xi taking error distribution into consideration, and error distribution σ of 2D and 3D line segments
_A χ ² test is performed using _li , and line segments that are considered to be statistically inconsistent are rejected. Here, the hypothesis test is performed on the assumption that the 2D-3D line segments match each other. Human body posture a
= [R, d] ^T and the observed value l = (P, Q ′), the error distribution is represented by the normal distribution σ _ai ˜N (0, P ₀ ), σ _li ˜N.
Assume (0, L ₀ ). The first- and second-order statistics of h (^ a _i−1 , ^ l _i ) are

[Equation 17]

[Equation 18] Becomes From this, the Mahalanobis distance k can be obtained as follows.

[Formula 19] h (^ a_i-1, ^ L_i) Error distribution is h (^
a_i-1, ^ L_i) -N (0, Z _i), So k
(^ A_i-1, ^ L_i) ~ Χ^Two(Ζ) (however, ζ)
= Rank (Z_i)). χ^TwoBased on the probability from the (ζ) distribution
It is possible to set a threshold value. For example, k (^ a
_i-1, ^ L_i) <9.21 is Z_iRank 2 (ζ =
Means that 99% of line segments will be accepted in 2)
is doing. Lines above the threshold are statistically consistent
Unthinkable and can be rejected. Multiple line segments
Is left as a candidate, k (^ a_i-1, ^ L_i)
Select the one with the smallest distance value of. The above two methods
In combination, the 2D predicted line segment Q is closest to the 2D observed line segment Q '.
2D observation line segment corresponding to the 3D line segment P on the image plane 150
A set of Q'l = (P, Q ') can be obtained. Na
The posture of the person model described above is
Estimate the posture, project it on the image plane, and superimpose it on the input image
You may obtain by applying.

<Interpolation based on sample motion> When a continuous human motion is generated from a storyboard, since the motion intervals between frames are wide, it is not natural to simply interpolate the motion. There is. Therefore, the missing operation information is supplemented by applying the sample operation accumulated in the data accumulating unit 40. As a method of acquiring a sample motion, a human motion is recorded in a video image and manually or automatically recorded by moving image processing.
Dimensional posture can be obtained. It is also possible to use a motion capture device using a magnetic sensor or an optical sensor. Here, the number of parameters is reduced by projecting onto an eigenspace in order to reduce the calculation amount at the time of matching and to generate various actions by transforming the sample action. This will be described below. First, the posture parameters of the human body at time t are

[Equation 20] Express. m is the total number of posture parameters. In this specification, since the posture parameter is estimated from moving images taken at equal intervals, t corresponds to the frame number. When N is the total number of teaching motion samples, the average c of the postures of the human body is

[Equation 21] Is. If the matrix obtained by subtracting the average c of the postures from each posture of the teaching motion is X, the covariance matrix Q of the posture data string
Is

[The number 22 is a Q = ^X X ^T. Solve the following eigen equations to find the eigenvalues and eigenvectors.

The eigenspace of λ _i e _i = Q e _i k is obtained by using k eigenvectors selected in descending order from the eigenvalues as basis vectors. The cumulative contribution rate is the ratio of the n largest eigenvalues among the m eigenvalues.

[Equation 24] Call. It should be noted that, in this specification, the upper two eigenvectors are used to facilitate the polynomial approximation. The motion curve in the eigenspace is obtained by projecting the pose parameters into the eigenspace.

G _t = [e ₁ , e ₂ ] ^T (x _t −c) FIG. 6 shows an example of a motion curve of a human body in the eigenspace obtained by the above method and a three-dimensional posture corresponding to each coordinate value. FIG.

<Automatic division of operation curve> FIG. 7 is a diagram showing a method of dividing an operation curve in an eigenspace. FIG. 7A shows an operation curve before division and FIG. 7B shows an operation curve after division. Indicates.
After estimating the posture parameter of the person from the input image, the continuous motion is divided and described by the following two procedures. By using the method of obtaining the interpolation based on the sample motion described above, the posture parameter is projected to the eigenspace to reduce the number of dimensions, and the action trajectory is generated. The motion curve is divided according to the minimum description criterion. A polynomial model is used as a model for describing the curve. In the minimum description criterion, when a certain model is assumed, the description with the shortest description length by the model is selected, and the description of the measured data by fitting the model and the trade-off of the approximation error are handled. That is, when a long-time operation is described by a simple model, the description length of the model decreases, but the value out of the model increases. On the other hand, when the operation is divided into small time intervals and approximated by the model, the error from the input operation is reduced, but the number of bits required for describing the model is increased. Therefore, optimization is performed so that both the model description length and the outlier description length are minimized.

As a model for approximating the motion curve, a second-order polynomial is used and the coordinate values of the motion curve are G ₁ (t) and G ₂ (t), respectively.

G ₁ (t) = a ₁ t ² + a ₂ t + a ₃ G ₂ (t) = b ₁ t ² + b ₂ t + b ₃ . The coefficient of the quadratic polynomial of the above equation is estimated by the least square estimation.

If A = [a ₁ , a ₂ , a ₃ ] ^T B = [b ₁ , b ₂ , b ₃ ] ^T φ = [1, t, t ² ] ^T , then write as follows: be able to.

G ₁ (t) = φ ^T A, G ₂ (t) = φ ^T B Further, when A _x and A _y are obtained by the robust estimation method,

## EQU29 ## A = [ΣφWφ ^T ] ^{to 1} Σ [φWG ₁ (t)] B = [ΣφWφ ^T ] ^{to 1} Σ [φWG ₂ (t)] W used Canchy's weighting function. Σ represents an operation for calculating the sum between the start point and the end point of the motion segment.

Since the movement curve of a person often draws a complicated locus, it is difficult to approximate the whole with one polynomial. Therefore, a portion that greatly deviates from the polynomial curve is recursively divided to improve the approximation accuracy of each motion segment. The description length L _sk of the motion curve is composed of the following three items.

[ _Equation 30] L _sk = L _parameters + L _outliers + L _offset The description length of a real-valued parameter is expressed by α ( _logg , where α is the number of parameters and n is the threshold of parameters.
₂ n) / 2 can be obtained. Therefore, the description length of each parameter is as follows. Model description length (L _parameters ) In order to represent a polynomial, it is necessary to describe two end points and six parameters. | Ζ | is the parameter width of points in the eigenspace, | η | is the parameter width of polynomial coefficients,
When ε is the resolution, the model description length can be expressed by the following equation.

[Equation 31] Outlier description length (L _outliers ) If there are outliers, they can be described as random points in the eigenspace. The description length when there are λ outliers can be expressed by the following equation.

[Equation 32] Description length of miscalculation from _model (L _{model points} ) The description length of miscalculation from the model is Gaussian distribution ζ to N.
This corresponds to the description length when (0, σ ² ) is sampled at the resolution ε.

[Expression 33] In order to decide whether or not to divide at the division candidate points, the description lengths of the motion segment before the division and the motion segment after the division are compared. When the description length decreases, the motion segment is divided at the division point.

L _sk > L _sk1 + L _sk2 FIG. 8 is a diagram showing a series of operations projected onto an eigenspace. FIG. 9 is a diagram showing the operation shown in FIG. 8 automatically divided into frames 0 to 249 by the above method. The circles in the graph of FIG. 9 are shown and the numbers are added to the frames corresponding to the numbers. FIG. 10 is a diagram showing a three-dimensional motion corresponding to motion segments divided into 0 to 249 frames. Figure 10
(A) is a frame corresponding to a walking motion (frames 0 to 5)
9) and FIG. 10B are frames (frames 60 to 129) corresponding to the sitting motion, and FIG. 10C is a frame (frames 198 to 249) corresponding to the rising motion. The numbers shown in the lower right of each frame indicate the corresponding frame.

<Analysis and Reconstruction of Individuality of Human Motion> By deforming the motion segment in the eigenspace, the individuality of human motion is estimated and reconstructed. The average motion and the variance from the average motion are calculated by using a sample when the same motion is performed by multiple people. By randomly generating fluctuations from the average motion and deforming the motion segment, a motion with a personality different from the input sample motion is generated. Figure 1
1 averages the cases where the same behavior is applied to 3 models,
It is a figure which shows having generated the new motion by calculation.
FIG. 11A is a diagram showing a projection of the same motion of three persons on an eigenspace. And FIG.
In (b), after obtaining the average action of the model of three persons, the action having a new personality is generated by using the variance of the actions of the three persons.

<Interpolation of Motions by Motion Segments in Eigen Space> Using a motion segment previously stored in the data storage unit 40, a method of interpolating an intermittent posture described by a storyboard will be described below. To do. The three-dimensional posture is estimated from the two-dimensional line segment on the storyboard by using the method of obtaining the interpolation based on the sample operation described above. The obtained three-dimensional posture is projected on the eigenspace. The points in the eigenspace corresponding to each posture are p1, p2, ..., Pn. The motion segment having the smallest distance from the points p1, p2, ..., Pn in the eigenspace is selected. FIG. 12 is a diagram showing an example of the storyboard input to the storyboard and the CG animation generated by the above processing. FIG. 12A is a storyboard described on the storyboard, and is a story in the order of (i) to (iii). This is subjected to a process of interpolating the intermittent posture described above to generate a continuous CG animation as shown in FIGS. 12 (b) (i) to (vi).

<Description of Dialogue in Storyboard System> FIG. 13 is a diagram showing an example in which a function of controlling the speed of movement of each character is added to the storyboard. Although the method described above can represent the connection of motions between storyboards, it cannot represent the context or speed of motions when, for example, multiple characters are interacting in the same storyboard. . Therefore, a causal relationship between motions in the same frame can be described by adding a timeline description field corresponding to each character to the storyboard. As shown in FIG.
A column 220 for controlling a timeline corresponding to each character may be provided below the storyboard description section. In this column 220, for example, each line segment of the timeline of the motion (display of the motion time) estimated from the video image is associated with the start point and the end point of the motion segment.
Then, by moving this timeline (slider function), the operation of the associated operation segment is changed.
That is, if this timeline is lengthened, the action of the character corresponding to this becomes slower. Further, by moving the start point and the end point of this timeline, the start point and the end point of the operation can be controlled. In this way, by changing the timeline (line segment) displayed in this field, it is possible to adjust the subtle utterance timing.

<Support for Joint Work by Sharing Storyboard> As an application of the storyboard system, a plurality of CG animators share a storyboard to support joint work. It will also be used as a place for multiple animators to share images. FIG. 14 is a diagram showing a configuration example of a system for performing a collaborative work support work. Various data is stored in the data storage unit 40 in the server, and a client having storyboard description units 302, 304, 306 is connected to the server. Each worker can refer to the storyboard from these clients.

FIG. 15 is a diagram showing a work process management and message transmission method using a storyboard for supporting collaborative work by a plurality of creators. Work process management When creating a CG animation by a plurality of people, it is important to grasp the progress of the work process such as which part is progressing the work. In this system, the storyboard is used to describe the information necessary for managing the work process. For example, the fact that motion data and model data have been obtained is indicated by a color or the like to show how far the animation creation is progressing. Message Board As a method of supporting information exchange among a plurality of workers, it is possible to add a message attachment function by adding a message board 240 on the storyboard description section. For example, the description of each character's dialogue, the situation written in G, and various specified items when creating an animation are described. These messages can be sent from a manager or the like to a specific worker by linking with the mail sending function.

Each of the above-mentioned operation input section, storyboard description section, operation generation section, data storage section, and image display section can be executed on a computer. In addition, the configuration of the present invention can be realized by reading out from a recording medium storing a program for performing the processing, executing a program received through a communication line, or the like. Floppy (registered trademark)
There are disks, CD-ROMs, magnetic tapes, ROM cassettes, and the like. The communication line includes the Internet and the like.

[0034]

Since a three-dimensional CG animation can be generated from a two-dimensional sketch, it is possible to greatly improve the productivity of animation creation. The three-dimensional sample action can be automatically divided and accumulated, and the accumulated sample action can be used to generate a continuous action from an intermittent action by a storyboard. . Moreover, by adding a function of describing the context on the time axis to the storyboard, it becomes easy to describe the timing of the dialogue between the plurality of virtual persons. Plus, you can share storyboards among multiple CG animators,
It is possible to significantly improve the efficiency of collaborative work.

[Brief description of drawings]

FIG. 1 is a diagram showing a functional configuration of a system of the present invention.

FIG. 2 is a diagram showing a virtual person model.

FIG. 3 is a diagram showing a relationship between a line drawing (screen) on a storyboard and a three-dimensional human model.

FIG. 4 is a diagram showing a relationship between a line segment on the screen and a corresponding part of the virtual person model.

FIG. 5 is a diagram showing a process of processing for estimating an initial position of a human model from a line drawing of a storyboard described on a storyboard.

FIG. 6 is a diagram showing an example of a motion curve of an eigenspace and a three-dimensional posture corresponding to each coordinate value.

FIG. 7 is a diagram showing a method of dividing a motion curve of an eigenspace.

FIG. 8 is a diagram showing a series of operations projected onto an eigenspace.

FIG. 9 is a diagram showing a division of the operation shown in FIG. 8.

FIG. 10 is a diagram showing a three-dimensional operation corresponding to a divided operation segment.

FIG. 11 is a diagram showing an example in which three models are averaged when the same motion is performed, and a new motion is generated by calculation.

FIG. 12 is a diagram showing an example of a CG animation generated from a storyboard input to a storyboard.

FIG. 13 is a diagram showing an example in which a time frame (timeline) is added to a storyboard.

FIG. 14 is a diagram illustrating a configuration example of a system for performing a collaborative work support work.

FIG. 15 is a diagram showing a work process management and message transmission method using a storyboard.

[Explanation of symbols]

10 Motion input section 12 Operation signal input section 14 Motion separation unit 20 Storyboard description section 22, 24, 26 Story board 30 Motion generator 32 3D pose estimation unit 34 Three-dimensional motion interpolation unit 40 data storage 50 Image display section 100 virtual person model 150 image side 212, 214, 216 Storyboard 220 Timeline description field 222,224,226 Storyboard 240 Message field 302, 304, 306 Storyboard description section

Claims (9)

[Claims] 1. A moving image generation system capable of generating a moving image from a two-dimensional line drawing of a person, wherein the system stores a plurality of motion segments which are three-dimensional posture parameters for a series of motions of the person. A storyboard description part that describes a two-dimensional person's line drawing to create a moving image, and a two-dimensional line drawing input to the storyboard description part to create a moving image. A posture estimation unit that estimates the posture parameters of the posture estimation unit; and, based on the plurality of posture parameters from the posture estimation unit, a motion segment that applies to the plurality of posture parameters is used from the data storage unit to perform the interpolation between the postures. And a motion image generation unit that generates a motion by interpolating the motion image, and an image display unit that displays the generated motion. System. 2. The moving image generation system according to claim 1, wherein the motion generation unit projects a plurality of posture parameters obtained by the posture estimation unit onto an eigenspace, and A moving image generation system characterized by selecting a motion segment having the smallest distance to a point. 3. The moving image generation system according to claim 1, wherein the storyboard description unit can describe a relationship on a time axis. 4. The moving image generation system according to claim 1, further comprising a plurality of storyboard description parts, and the storyboards described by the storyboard description parts are accumulated in a common accumulation part. A moving image generation system characterized by being able to perform. 5. The moving image generation system according to claim 4, wherein a display state of the accumulated storyboard can be changed so as to indicate a progress state. 6. The moving image generation system according to claim 1, wherein a three-dimensional posture parameter in which a person is operating is input,
The moving image generation system further comprising a motion input unit that divides the posture parameter into motion segments and stores the motion segments in the data storage unit. 7. The moving image generation system according to claim 1, wherein the three-dimensional posture parameter division in the movement input unit projects the posture parameter onto an eigenspace to generate a movement curve. Then, the motion curve is divided according to the minimum description criterion,
A moving image generation system characterized by performing by obtaining a motion segment. 8. A program for causing a computer system to configure the moving image generation system according to any one of claims 1 to 7. 9. A recording medium storing a program for causing a computer system to configure the moving image generation system according to claim 8.