JP3337938B2 - Motion transmitting / receiving device having three-dimensional skeleton structure and motion transmitting / receiving method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to three-dimensional computer graphics (hereinafter, referred to as "3DCG") based on a network such as the Internet.
Computer graphics. The present invention relates to a motion transmitting / receiving apparatus having a three-dimensional skeleton structure and a motion transmitting / receiving method for efficiently transmitting / receiving a natural motion of a character having a complicated skeleton structure, such as a human, when producing or commercial-using the character. .

[0002]

2. Description of the Related Art Recently, as a field of application of 3DCG, WW
Attention has been focused on virtual malls (Virtual Mall) on the Internet such as W (World Wide Web) and various homepages. In particular, due to the rapid development of the Internet,
An environment for easily handling 3DCG such as games and movies at home is being prepared.

In the conventional WWW or the like, a plurality of machines called clients such as personal computers are connected to machines called servers such as personal computers and workstations via the Internet, so as to respond to requests from clients. If necessary, necessary information can be obtained by downloading data such as image, sound, text, and arrangement information provided by the server and reconstructing the data on the client side. The communication between the server and the client uses TCP / IP (Tranamission Control).
Protocol / Internet Protocol).

Conventionally, data provided from the server side is:
Mainly text data and image data only, but recently, VRML (Virtual Reality Modelin
g Language) and VRML browser standardization has progressed.
There is a movement to transfer 3DCG data itself such as a shape and a scene.

[0005] Here, the VRML will be briefly described. In a conventional data format such as HMTL (Hyper Text Markup Language), which mainly uses images and texts, enormous transfer time and transfer cost are required to transfer image data, particularly moving image data. . Therefore, in the current system, there is a restriction on network traffic. In contrast, in the conventional 3DCG, all of the viewpoint information and light source information, including the shape, are processed using three-dimensional data. With the advance of computer graphics (CG) technology, the image quality of images created by CG has rapidly improved, and transferring CG data as it is is much more efficient in terms of data volume. Efficiency has improved. Normally, the compression ratio is 1/100 or more of that when transmitting equivalent image data. Therefore, there is a movement to standardize a method of transferring 3DCG data via a network. As one of the efforts, standardization of 3DCG data called VRML has been proposed (VRML Ver.2.0). VRML Ver.2.0
Defines data formats such as shape data called primitives, various types of light source data, viewpoint data, and texture data, and a method of specifying the movement of a rigid body.

On the other hand, what has been recently attracting attention in the conventional CG field is an animation for creating an image in real time, that is, a so-called real animation. By adopting this real animation, CM (commercia
l message) and a movie. As one of them,
There is a method called a kinematics method for naturally reproducing complicated movements by expressing a complicated shape of a human or the like by a skeletal structure and defining a movement amount of a joint of the skeleton which changes every moment.

[0007] The kinematics method is a technique developed mainly in robotics. In the kinematics method , the rotation angle and the X-axis, the Y-axis, and the Z-axis directions of the joint portion of a skeleton model (consisting of a link and a joint (indirect)) having a linear link structure are used.
The posture of the skeleton model is determined by setting the movement amount (position) in the axis, Y-axis, and Z-axis directions.

[0008] By applying this kinematics method to a character having an arbitrary skeletal structure created by CG, compared to a conventional polygon-based animation (generated in combination with a key frame method or the like). At the same time, the amount of data can be reduced, and at the same time, it is becoming possible to reproduce natural movements of humans and dinosaurs.

[0009]

SUMMARY OF THE INVENTION However, VRM
The conventional 3DCG modeling language on the Internet centered on L does not consider a method of defining continuous time-series movement of an object. Further, in reality, there is no method for defining the movement of a complicated shape such as a human being, in other words, for example, a natural posture of a limb.

Furthermore, when generating a conventional animation based on the kinematics method, a large amount of motion data is required to generate a smooth motion. Therefore, in consideration of application to a network, the fact is that the problem of traffic has not yet been solved.

In order to cope with such a current situation, the present inventors define a complex structure such as a human with a skeleton structure, and designate a reference position of a designated skeleton (hereinafter, referred to as a “root”).
And a mechanism for compressing, restoring, and transmitting and receiving motion information of each joint position of the skeleton, a hierarchical relationship of the skeletal structure, an initial state of the skeletal structure, a constraint condition of the skeletal structure (for example, rotation of each joint). Motion data of the skeletal structure to be added to the initial information including the shape data associated with the skeleton and the data of the correspondence relationship with the skeletal structure. Element data (hereafter,
It is called “basic operation”. ), And a mechanism for transmitting or restoring the decomposed basic motion together with a connection method of a typical motion of the basic motion, and a posture of a skeletal structure required based on the transmitted and received animation information. A mechanism for deciding, and a mechanism for operating based on the avatar motion information transmitted and received, and a mechanism for interactively changing a motion operation, such as a motion direction or a basic motion sequence, according to a user's instruction, may be included. In the case of transmitting and receiving a three-dimensional movement of a character having a complicated structure like a human, a human hierarchy is defined by a skeletal structure, and a three-dimensional virtual space of a joint of a part of the defined skeletal structure is defined. the amount of movement of the inner, time series transmitted and received, or representative motion elements added how to connect elements of the representative motion can be transmitted and received, as a result, networks It can send and receive natural and smooth motions of characters in a click on the base and the transmission and reception system was conceived that it would be possible to greatly reduce the data amount at the time of transfer.

The present invention has been made on the basis of the above-mentioned idea, and comprises a human-like complicated shape (avatar) having a skeleton structure (hierarchical structure) between a client and a server connected to the Internet, and an avatar. And a motion transmission / reception apparatus having a three-dimensional skeleton structure and a motion transmission / reception method capable of transmitting and receiving motion data, which is the basis of the above, and enabling an avatar to be interactively operated in a three-dimensional virtual space on a network. Aim.

[0013]

According to a first aspect of the present invention , there is provided a motion transmitting / receiving apparatus having a three-dimensional skeletal structure , comprising:
Send a byte and motion data of the skeleton structure whose segments connecting them to the receiver, to generate based on the animation motion data of the transmitted <br/> skeletal structure at the receiving side
A that equipment, hierarchy is defined on the Symbol skeletal structure data
And the basic motion data that defines the movement of a part of the skeletal structure.
And connection method information between the basic operation data.
Transmitting means, the hierarchical data, and the basic operation data.
And the connection method information, and receives the hierarchical data,
And the posture of the skeletal structure based on the basic motion data
And connect the basic operation data to the connection
Receiving means for connecting according to legal information .

[0014] The motion transmitting device of a three-dimensional skeleton structure take some to claim 2 of the present invention, joints in three-dimensional computer graphics from a transmitting apparatus, endocytosis and
It transmits the motion data of the skeleton structure whose segments connecting them to the receiving side, to generate an animation based on the motion data of the transmitted skeleton structure at the receiving side system
Device in the system, wherein the receiving side has the skeletal structure.
Hierarchical data defining the above skeletal structure to determine the posture
Data and the basic motion that defines the motion of a part of the skeletal structure
Operation data and the above basic operation data.
Transmission means for transmitting connection method information to be used, and
It is a thing.

Furthermore, dynamic-out reception apparatus of a three-dimensional skeleton structure take some to claim 3 of the present invention, three-dimensional computer graphics
Joints, end sites, and connecting them
Receives motion data of a skeletal structure
Equipment in a system that generates animations
The hierarchical data defining the skeletal structure and the skeletal structure
Basic motion data that defines the movement of a part of
Connection method information used to connect between operation data
Receiving means for receiving the hierarchical data,
The basic motion data is used to determine the posture of the skeletal structure.
The connection method information and the basic operation data
Is used to connect

[0016] The motion transmission and reception method of the three-dimensional skeleton structure take some to claim 4 of the present invention, three-dimensional computer from the sender
Joints, end sites, and
Motion data of skeletal structure consisting of segments connecting them
To the receiving side, and the receiving side transmits the skeleton structure
This is a method for generating animations based on motion data.
The hierarchical data defining the skeleton structure and the skeleton
Basic motion data defining movement of a part of the structure;
Step of transmitting connection method information between the operation data
And the hierarchical data and the basic operation data and
Receiving the connection method information and the hierarchical data and the base
Determine the posture of the skeletal structure based on the motion data
And the basic operation data in the connection method information.
Therefore, connecting .

[0017]

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

DESCRIPTION OF THE PREFERRED EMBODIMENTS One embodiment of the present invention will be described below in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing a configuration of a motion transmitting / receiving apparatus having a three-dimensional skeleton structure according to an embodiment of the present invention. With reference to FIG. 1, a motion transmitting / receiving apparatus having a three-dimensional skeleton structure according to the present embodiment
This is for transmitting the skeleton structure motion data in the DCG to the receiving side and generating an animation based on the skeleton structure motion data transmitted from the receiving side.
It includes a G animation data transmission unit 1, a CG animation data transmission unit 2, and a CG animation data reception unit 3.

FIG. 2 is a block diagram showing an example of the configuration of the CG animation data transmission section. Referring to FIG.
The CG animation data transmitting unit 1 includes a skeleton structure defining unit 11, a motion data defining unit 12, and a motion data element decomposing unit 13. The CG animation data transmitting unit 1 defines a hierarchical relationship of the skeleton structure in response to a request from the receiving side. Data, the initial state of the hierarchical data, the constraint conditions of the skeleton joints, the shape data associated with the skeleton, the correspondence relationship data with the skeletal structure, a moving image data sequence for performing moving image mapping on a part of the shape data, and a mapping method. The motion data of the skeletal structure is transmitted as time-series data in addition to the initial information,
It is decomposed into basic operations that characterize the motion in the three-dimensional virtual space, and it is determined whether or not to transmit as data obtained by adding the connection method of the basic operations to the decomposed basic operations. Then, skeleton structure motion data is added to the initial information and transmitted.

Referring again to FIG. 1, the CG animation data transmission unit 2 adds the skeletal structure motion data to the initial information by the transmission program on the transmission side computer and the reception program on the reception side computer. The CG animation data is transmitted from the transmitting computer to the receiving computer.

FIG. 3 is a block diagram showing an example of the configuration of the CG animation data receiving section. Referring to FIG.
The CG animation data receiving unit 3 includes a skeleton structure defining unit 31, a shape data associating unit 32, and a motion data defining unit 3.
3. It includes a posture determination unit 34, a motion element data connection unit 35, and a motion connection unit 36, receives the CG animation data transmitted from the CG animation data transmission unit 2, and transmits the received CG animation data. When the mode is determined and the motion data of the skeletal structure added to the initial information is determined to have been transmitted as time-series data, the initial information and the motion data of the skeleton structure transmitted as the time-series data are interpreted. Then, the posture of the skeletal structure is determined based on the kinematics, and the motion data of the skeletal structure added to the initial information is decomposed into basic motions that are characteristic of the motion in the three-dimensional virtual space. If it is determined that the data has been transmitted as data obtained by adding the connection method of the basic operation to the basic operation, the initial information and the decomposition The interpreting movement data of the skeleton structure connection is transmitted as additional data of the basic operation in the basic operation, so as to determine the attitude of the skeletal structure based on kinematics.

Here, the operation of the motion transmitting / receiving apparatus having the three-dimensional skeleton structure will be described. In response to a request from the receiving side, the CG animation data transmitting unit 1 performs hierarchical data defining the hierarchical relationship of the skeletal structure, the initial state of the hierarchical data, the constraint conditions of the skeletal joints, the shape data associated with the skeleton, and the skeletal structure. The motion data of the skeletal structure is transmitted as time-series data by adding to the initial information including the correspondence relationship data, the moving image data string for moving image mapping to a part of the shape data, and the mapping method, It is decomposed into basic operations that characterize the motion in the virtual space, and it is determined whether or not to transmit as data obtained by adding the connection method of the basic operations to the decomposed basic operations. The skeleton structure motion data is added to the initial information and transmitted.

At this time, the skeleton structure definition unit 11 defines a CG character (having a complex skeleton such as a human) to be transmitted by a skeleton structure, and the defined skeleton structure is defined as a hierarchical structure. A route (reference position for calculation) is set at the position that becomes the parent of the structure. Here, the skeleton structure of a normal human has a hierarchical structure as shown in FIGS. 4 and 10A. On the other hand, the route is usually set at the center position of the body, for example, at the base of both feet, and the origin of the CG character's local coordinate system (hereinafter referred to as the Z-top right hand system) is set at this position. Put.

In FIG. 4, the portions surrounded by squares are called joints or joints of the skeletal structure, and the straight lines connecting the joints correspond to the parts such as the arms and the legs. This is called a segment. Furthermore, the portion surrounded by a thick square is a tip portion of a hand or a toe called an end site. In FIG.
Although the basic skeleton has been shown, the hierarchy of the skeleton structure is changed as necessary. For example, in the skeletal structure of FIG. 4, when it is desired to express the hand hierarchy in more detail, a hierarchy of five fingers and a finger joint included in each hierarchy may be provided below the left and right wrists.

In the skeletal structure, an initial state of the hierarchy is set in addition to the hierarchical relationship of the joints. For example, the lengths of the arms and legs and the initial directions of the lengths of the arms and legs are set.
In addition, the movable range of each joint is set in the skeletal structure. For example, how many times the right elbow can be rotated around the XYZ axes of the local coordinate system can be set.

FIG. 5 is a diagram showing an example of the data format of the skeleton structure, and FIG. 5A shows the hierarchy of the skeleton. Referring to FIG. 5 (a), for example, taking a human elbow as an example, the segment corresponds to the upper arm or the forearm, and there are portions called sites at both ends of the segment. Further, joints such as elbows (defined by the upper arm site and the forearm site) are joints. On the other hand, FIG. 5B shows an example of the definition of the skeletal structure. In the example of the definition of the skeletal structure in FIG. 5B, the names of the respective parts such as Neck and NeckBase are added as identifiers to the segments, sites, and joints, and the three-dimensional coordinate position (- p) and the rotation angle (-r) are set for each of the three components. Site coordinates are such that the origin of local coordinates is at the location of the parent site for each segment. For the joint, an identifier such as SkullBase and a hierarchical relationship (-c: parent-child relationship between which parent site and which child site are connected) are set. Furthermore, if necessary, the degree of freedom of rotation (for Ball, XY
Z), and a movable range (-l: angle) are set.

The motion data definition unit 12 sets the motion data of the main part of the skeleton structure set by the skeleton structure definition unit 11. Here, the main part is a part of the skeleton to be moved. For example, in the case of the whole body, the three-dimensional coordinate position or joint angle data of the joint of the skeleton included in the whole body is given as time-series data. can get. The format of the motion data usually consists of time data such as a frame number and the like, and entity data, each of which includes a joint name, an identifier for identifying angle data or coordinate data, and three elements (XYZ axis components). Consists of data.

The motion element data decomposing unit 13 decomposes, as necessary, into reference points and progress information necessary for expressing the movement of the skeleton, based on the time-series motion data sequence. The decomposition method and the like will be described later in detail.

The CG animation data transmission unit 2 transmits CG animation data in which skeletal structure motion data is added to initial information from a transmission computer by a transmission program on a transmission computer and a reception program on a reception computer. Transmit to the receiving computer. That is, the CG animation data transmission unit 1
The skeletal structure and motion data compressed by
Transmission between server and client according to IP.

The CG animation data reception unit 3
C transmitted from the G animation data transmission unit 2
When the G animation data is received, the transmission mode of the received CG animation data is determined, and when it is determined that the motion data of the skeletal structure added to the initial information has been transmitted as time-series data, the initial information and the time While interpreting the motion data of the skeletal structure transmitted as sequential data and determining the posture of the skeletal structure based on the kinematics, the motion data of the skeletal structure added to the initial information is obtained in the three-dimensional virtual space. When it is determined that the data has been transmitted as data obtained by adding the connection method of the basic operation to the decomposed basic operation, the initial information and the decomposed basic operation are divided into basic operations. Interpret the motion data of the skeletal structure sent as data to which the connection method of the basic motion is added, and based on the kinematics, To determine the momentum. At this time, the skeleton structure definition unit 31 receives the human skeleton structure data (see FIG. 4) defined by the skeleton structure definition unit 11, and interprets the received skeleton structure data.

The shape data associating section 32 groups which segment of the skeleton structure to be processed corresponds to which part of the previously transmitted shape data.
Here, the shape data is a portion of the skin (also referred to as “skin”) to be attached to the skeleton of the CG character, and is usually a polygon row or a set of mesh data. A group corresponding to the upper arm, a group corresponding to the foot, and the like are classified based on the polygon row and the mesh data, and how the classified data group corresponds to the upper arm and the foot of the skeletal structure is defined in advance. Normally, the correspondence between the shape and the skeleton is such that the origin of the local coordinate system of the segment is placed at the site on the parent side of the segment of the skeletal structure and in which direction (designated by the direction vector and the rotation vector) from the origin. Is determined by setting whether to assign the skin on a scale of. A skeleton segment is arranged at the center of a plurality of shape groups and is associated with the shortest distance from each shape group.

Further, in order to make the shape look natural, it is necessary to make settings so that the skin is not broken at the joints. In this case, a new shape group (called “flexor”) is set for the shape group of the joint portion,
This is realized by setting the weighting coefficient to the effect of the bending angle from the parent segment and the child segment in accordance with the bending angle of the joint, and deforming the shape group included in the flexor.

The motion data definition unit 33 receives the motion data transmitted from the motion data definition unit 12 or the motion element data decomposition unit 13, and restores the received motion data as needed. The motion element data connection unit 35 connects the basic operations of the motion data restored by the motion data definition unit 33 smoothly.

The posture determining section 34 is configured to transmit the skeleton structure data sent from the skeleton structure defining section 31 to the shape data associating section 3.
The posture of the skeleton at an arbitrary time is determined using the shape data grouped for each skeleton sent from 2 and the motion data sent from the motion data definition unit 33.
In addition, the posture determination unit 34 determines the posture of the skeleton at an arbitrary time, as necessary, by taking into account the movement data transmitted from the movement element data connection unit 35 and connecting the basic operations smoothly. I do. The movement connection unit 36 connects the posture of the skeleton determined by the posture determination unit 34.

Next, a method for determining the posture of the skeletal structure will be described. The method of determining the posture of the skeletal structure in FIG. 4 at an arbitrary time (frame) differs depending on the form of the motion data, but basically, kinematics (also referred to as “kinematics”) developed mainly with robotics. (See "Dynamics and control of robots" (Arimoto, Asakura Shoten)). The kinematics method determines the linear posture by giving the time change of the angle data of each joint to the initial position of the joint structure connected by the linear link and determining the joint position in order from the parent How to

In the case of the human skeleton shown in FIG. 4, usually, a total of five straight chains (for example, a right arm and a right foot) from the root to the right and left toes, the root to the left and right hands, and the root from the top are individually It is necessary to determine the posture.

Now, consider the case where the data sent from the motion data definition unit 33 is the angle data of each joint. Generally, all necessary joint angle data in FIG.
4, the posture of each straight line is uniquely determined. As described above, the method of determining the posture of the skeleton from the initial state of the skeleton structure and the joint angle data (rotation angle from the initial position in the local coordinate system) at an arbitrary time is described by direct kinematics (forward motion). Gaku) called the law.

In the case of the direct kinematics method, the posture determination unit 34 calculates the skeleton initial value data input from the skeleton structure definition unit 31 in order from the joint closest to the root. For example, a rotation matrix (3 * 4 matrix) determined by the rotation angle of the parent site given by the motion data definition unit 33 is applied to the parent site of the skeleton shown in FIG. 5 and the length of the segment. As a result, the position of the child site is determined. The operation of applying the rotation matrix to the parent site of the skeleton and the length of the segment is repeated by the number of joints.

As an attitude determination method other than the direct kinematics method, there is an inverse kinematics method. In such inverse kinematics, it is necessary to set a range to be calculated for the skeletal structure in FIG. 4 (an influence range for moving the skeleton, which is referred to as a “handle”). In FIG. 4, for example, the handle is set from the right collar to the right hand. Further, with respect to the set handle, the motion data definition unit 33 sends the tip of the handle, that is,
In this example, time-series data of a trajectory in the three-dimensional virtual space at the right end is provided. The trajectory data can set up to six elements of the three-dimensional coordinates (XYZ axis components) and the rotation angle (rotation components around the XYZ axes) of the tip, but usually, each of the three components is separately set in the posture determination unit. Give to 34.

In the case where the above inverse kinematics method is used to solve the posture of the skeleton (the range of the handle) from the trajectory of the tip portion of the handle using a mathematical expression, there may be a plurality of solutions without constraint conditions. That is, the posture of the skeleton in the three-dimensional virtual space is not uniquely determined. Therefore, usually, the calculation is performed based on an evaluation function (energy minimization) that minimizes the sum of squares of the minute changes of all the joint angles included in the handle. For specific calculation, an inverse matrix of a determinant (referred to as a “Jacobi matrix”, which is uniquely determined) indicating the relationship between the joint angle used in the above-described direct kinematics and the tip position is calculated. When the number of joints is N, this inverse matrix becomes a 6 * N matrix (6 is the maximum degree of freedom at the tip), and there is no inverse matrix as it is.
Therefore, under the above-mentioned constraints, the pseudo-Jacobi inverse matrix is calculated by a sweeping method or the like (see “Robot Engineering (Hirose, Shokabo)”).

In the attitude determination unit 34, both algorithms of the above inverse kinematics and direct kinematics can be used depending on the application. further,
In the posture determination unit 34, the skeleton structure definition unit 3
Based on the movable range data of an arbitrary joint input from step 1, the posture of the joint is limited by comparing it with the movable range set for the joint angle determined by kinematics. In other words, as a result of the posture calculation using kinematics, if the angle of a certain joint exceeds the movable range, a condition determination process for setting the angle to the value obtained by the posture calculation is added. By using the above kinematics method, the posture of the character having the skeletal structure can be determined.

Next, a method of assigning a skin to the posture of the skeleton will be described. As described above, in 3DCG,
The skin data is a polygon group or a free-form surface group. Generally, a general-purpose 3DCG modeling tool is used for modeling the shape of the skin. The output data format of this tool is Dxf (AUTO DESK's product name AUTO
Standard formats such as CAD file format and Obj (SGI product name Wave FRONT file format) are used.

The shape data association unit 32 sets which segment of the skeleton structure defined by the skeleton structure definition unit 31 is associated with the polygon group. For example, if the shape group of the human upper arm corresponds to the segment of the upper arm of the skeleton, the origin of the modeling coordinates of the shape group (such as the center of gravity) and the coordinate origin of the parent site of the segment match, and an offset ( XYZ axis direction three components) and shape scale (XYZ axis direction magnification)
And to fine-tune the placement relationship.

If the posture of the skeleton is determined using the above-described kinematics method, the position and direction of the parent site of the segment of interest are determined. When applied to the group shape data (vertex coordinates of polygons), the skin moves in conjunction with the movement of the skeleton.

The above is an explanation of a method for generating an animation of a CG character having a basic skeleton structure. In the above example, the motion data is the angle data of each joint (in the case of direct kinematics) or the trajectory data of the distal end portion (in the case of inverse kinematics). It is necessary to sequentially input the time series data of the trajectory data of the part from the motion data definition unit 33. Generally connected to the Internet,
In the transmission / reception system of CG animation data shown in FIG.
It is necessary to minimize communication traffic in the CG animation data transmission unit 2 as much as possible.

Next, a method of compressing motion data will be described. Compression ratio, the transfer performance and the transmission line, in accordance with the drawing performance of the transmit and receive side of the machine must be able to hierarchically control. FIG. 6 is a diagram for explaining a method of dividing the movement of the skeletal structure. Referring to FIG. 6, the motion data of the CG character is determined by (1) the motion of each joint of the skeletal structure, and (2) the position and direction data of the entire body in the three-dimensional virtual space. Hereinafter, the former is referred to as local information, and the latter is referred to as world information.

In normal CG character animation, world information is defined as information on the root of the skeletal structure shown in FIG. 4, and local information is used as joint information.

First, consider a three-dimensional virtual space surrounding a range in which a CG character can move as shown in FIG. The reference coordinates of the three-dimensional virtual space and the root position of the skeletal structure (in FIG. 6, the position of the human stomach) are matched, and the origin of the local coordinate system is set. With respect to the origin of the local coordinate system, a virtual operation space is set in each of the X-axis, Y-axis, and Z-axis directions. The overall size of the virtual operation space is determined by setting the size in the XYZ axis directions. Furthermore, the movement of the virtual motion space in the world coordinate system due to the movement of the body,
It can be defined by a matrix of translation and rotation of the character. The virtual motion space rotates or moves in accordance with the movement (world information) of the CG character. Qualitatively, the virtual motion space is the largest that the hand or foot of the CG character can move in the local coordinate system. Range. In FIG. 6, a rectangular parallelepiped is taken as an example of the virtual motion space, but a sphere centered on a CG character may be selected. In this case, it is more convenient to provide the division described below in a polar coordinate system.

Next, a sub-three-dimensional virtual space that divides this three-dimensional virtual space into a three-dimensional virtual space will be considered. This sub-three-dimensional virtual space divides the virtual motion space equally, and its size is a division pitch dx, dy, dz or XYZ in the XYZ axis direction in the local coordinate system.
The number of quantizations is determined by the number of divisions (l, m, n) in the axial direction.

When setting the movement of the CG character in the three-dimensional virtual space, it is necessary to set the trajectory of each joint position or the tip position of the skeleton structure. The trajectory is defined by an identification number of the sub three-dimensional virtual space or a coordinate position. If the division pitch of the sub three-dimensional virtual space is increased,
The quantization level at each joint position or tip position of the skeletal structure is reduced, and the data amount can be reduced.

The sub-three-dimensional virtual space is further divided hierarchically and combined as shown in FIG. 7 in accordance with the hardware of the transmission / reception side or the performance of the transmission system. In order to encode the trajectory of the continuous movement of the CG character in the three-dimensional virtual space, the progress point of the movement in the three-dimensional virtual space shown in FIG. 8 is specified in the sub three-dimensional virtual space. The minimum information required to reproduce the motion includes the position (or identification number) of the sub-three-dimensional virtual space at the start point of the motion, the position (or identification number) of the sub-three-dimensional virtual space at the end point, and the motion. (Linear interpolation, spline interpolation, etc.) on the interpolation method between the start point and the end point of. It should be noted that the more the information of the elapsed points shown in FIG. 8 is increased, the more smoothly the exercise can be reproduced, but the larger the data amount. When the identification number of the sub-three-dimensional virtual space is given, the receiving side converts the coordinates into coordinates in the three-dimensional virtual space by, for example, the center of gravity of the sub-three-dimensional virtual space and the origin of the local coordinates from the origin. Convert using relative position. In this case, a quantization error corresponding to the size of the sub three-dimensional virtual space occurs.

The data flow in the transmission / reception system of the skeleton data of the CG character shown in FIG.
First, in the motion element data decomposing unit 13, for a CG character to be transmitted, a movement / rotation matrix representing a position and a direction (world information) in the world coordinate system of the three-dimensional virtual space, The size and the division pitch (dx, dy, dz) into the sub-three-dimensional virtual space are transmitted. Next, if necessary,
In the unit of the arm or foot of the CG character, the position or the identification number of the sub-three-dimensional virtual space between the start point and the end point of the three-dimensional trajectory where the skeleton of the arm or foot moves, the sub-space position of the progress point or the progress point information, And a connection method between the start point and the end point of the three-dimensional trajectory is transferred as necessary. Then, the movement data element connection unit 35 on the receiving side interprets (restores) the transfer data and reproduces the movement of the CG character by interpolating the three-dimensional trajectory of the CG character shown in FIG.

In the above description, a method of setting only the trajectory of the tip portion has been described as an example for the sake of simplicity, but the posture determination unit 34 employs direct kinematics. Similarly, when the posture is determined, the angle information obtained from the progress positions of all the joints of the necessary skeleton may be transferred as information in the sub three-dimensional virtual space.

Next, the sub-three-dimensional virtual space is
FIG. 9 shows a method for obtaining joint angle data on the transmitting or receiving side when the signal is given to the transmitting side. Referring to FIG.
The initial position of the joint position (the center position of the sphere in FIG. 9) is set by the skeleton structure definition unit 11 and the skeleton structure definition unit 31. Next, the position in the sub-three-dimensional virtual space at a certain time,
Alternatively, when the identification number is given to the posture determination unit 34,
The coordinates of the position of the center of gravity of the sub three-dimensional virtual space can be calculated. The rotation angle (θx, θx) around each axis in the local coordinate system based on the position or the identification number in the sub-three-dimensional virtual space, the size of the sub-three-dimensional virtual space, and the length of the known segment. θy, θz). In this case, the order of rotation (such as XYZ) must be determined in advance.

[0068] In the posture determining unit 34 adopts the inverse kinematics method, when determining the posture, the C G animation data transmitting unit 1, a handle information skeletal structure, the locus information of the tip position of skeletons From the motion element data decomposing unit 13.

In the description so far, the information on the torsion of the skeleton (rotation around the Z axis in the local coordinate system) has not been accurately defined. In the definition part of the joint (Joint information in FIG. 5B), when the degree of freedom of the joint is Ball, the joint can be rotated around three axes of the XYZ axes.
Ignore rotation about the Z axis. However, in humans,
The torsion greatly affects the posture of the joints such as the waist, neck, and wrist. In particular, the CG character needs to accurately represent information such as whether the wrist is returning. In this case, separately from the above description, it is necessary to add torsion information (rotation angle about the Z axis) to the sub-three-dimensional virtual space information and transmit and receive the information separately. For example, in the example shown in FIG.
The twist angle of the wrist at to is θo, and the twist angle at time t1 is θ
₁ and receive, among joint angles of skeletons decided in the posture determining portion 34, twisting only portions necessary, replaced with angle data interpolated between the θo to theta _1.

Next, a method for connecting basic operations (continuous operations such as "raising the right hand" and "waving the hand") will be described. For example, as a basic motion of the CG character, (1) a motion in which the right hand moves from the virtual motion space at the lower left of the body to the virtual motion space at the center of the body and moves to the virtual motion space at the upper right. , (2) a case of connecting an operation of reaching for a hand from a virtual operation space in the upper right to a virtual operation space in the forward direction of the body.

FIG. 11A is a conceptual diagram illustrating the connection between two basic operations. Referring to FIG. 11A, at time t0
The movement from time t1 to time t1 corresponds to the basic operation of (1), and the movement from time t2 to time t3 corresponds to the basic operation of (2). The motion element data decomposing unit 13 decomposes the data into virtual motion space information and sub-three-dimensional virtual space information of the start point and sub-three-dimensional virtual space information of the end point, or, if necessary, between the start point and the end point. The sub-three-dimensional virtual space information of the lapse point is transferred together with the interpolation method. In the CG animation data receiving unit 3, the motion element data connecting unit 35 interprets the sub three-dimensional virtual space and restores the trajectory information for each basic motion. Further, the movement element data connection unit 35 connects the basic operation (1) and the basic operation (2).

FIG. 11B is a diagram showing a connection state between normal basic operations based on the time axis. Referring to FIG. 11 (b), basic operation (1) is obtained from the operation space information at time t1.
Can be restored. FIG. 11B is a graph obtained by converting this into the position of the end site of the joint of the right hand (tip of the right hand) or the joint angle data. Similarly, the right hand angle or the coordinate value at time t2 can be restored from the operation space position at time t2. As connection information for connecting these basic operations, information on how far back in the negative time direction (value in the negative time direction) of the basic operation (1), for example, t1 ', and the time of the basic operation (2) It is necessary to set how much time elapses in the forward direction, for example, t2 '. This value may have a fixed value as a default on the transmitting and receiving sides. Using these pieces of information, interpolation processing is performed using the coordinate values of the right hand tip at times t1 ', t1, t2, and t2' in FIG. In addition, although these values generally have three components on the XYZ axes in the white circle in FIG.
Here, for simplification of description, only one component is shown.

As the interpolation method for these points, information such as linear interpolation or spline interpolation is set in advance on the transmitting / receiving side. Furthermore, to make the connection smooth,
If necessary, sub-three-dimensional virtual space information serving as interpolation information is transferred between time t1 and time t2 (see time t1 and time t1 in FIG. 11).
point between t2). In the above example, the case where the posture determining unit 34 determines the posture of the skeleton by employing the inverse kinematics method has been described as an example. Only the information of the joint is provided to the posture determination unit 34, and the connection of the basic motion is exactly the same.

By using the above method, by linking a plurality of basic operations based on a plurality of previously transferred basic operations, or by sending only new virtual operation space information and interpolation information, A new motion in which the basic operation has been changed can be generated with an extremely small amount of data.

Next, the starting point and the ending point of the basic operation described above,
Alternatively, a method of creating key point information will be described. The basic operation of the CG character in the three-dimensional virtual space is, as described above, where the CG character of interest moves in the three-dimensional virtual space and along what trajectory (world information). What kind of posture (motion) the hand or foot of the CG character takes at a certain time on the trajectory of the CG character of interest (local information)
And divided into The former is the amount of translation of the root of the skeletal structure in FIG. 4 at an arbitrary time (Xtrans, Ytrans, Ztran
s) and direction (Xrot, Yrot, Zrot).
On the other hand, the latter is the angle of each joint or tip (Xrot,
Yrot, Zrot) or three-dimensional coordinates (Xtrans, Ytrans,
Ztrans). These pieces of information are both sent from the transmission system to the reception system as time-series data or the above-described virtual operation space information.

When the basic operation itself is set, a technique called motion capture is usually used. Hereinafter, a method of creating motion data using motion capture will be described first. In the motion capture, a plurality of cameras capture images while making a desired movement on a model in which a marker reflecting light is attached to a joint part of a body. 2
Based on a two-dimensional image of a marker captured by more than one camera, the three-dimensional coordinates of the marker or angle data corresponding to the skeletal structure is calculated (Japanese Patent Application No. 7-312).
005). Motion captured C
The motion of the G character is angle data (time series) of each joint of the skeletal structure shown in FIG. This data is shown in FIG.
In the example of (a), the trajectory 1 of the movement from time t0 to time t1
Or a locus 2 of the movement from time t2 to time t3.

The motion data defining unit 12 sends the angle data of each joint of the skeletal structure as a basic operation to the CG animation data transmitting unit 2 as time-series data.
What the motion data definition unit 33 interprets is a basic motion transmission / reception method. On the other hand, when it is desired to reduce the amount of motion data, there is a method of sampling progress point information on a trajectory and transmitting and receiving the sampled progress point information along with a connection method.

Further, when it is necessary to compress transmitted / received data and reuse it, it is necessary to decompose the data into virtual operation space data shown in FIG. In this case, the motion element data decomposing unit 13 quantizes (vector-quantizes) the virtual motion space of the CG character in FIG. , Or broken down into identification numbers. The motion element data connection unit 35 restores the trajectory of the motion captured by using the information on the connection method from the received virtual motion space data.

Next, a method of creating motion data without using motion capture will be briefly described. The skeleton structure of the CG character shown in FIG. 6 and a virtual motion space are set. The initial state of the CG character is
This is shown in FIG. Virtual operation space sub 3
The virtual space is divided into three-dimensional virtual spaces, and the position is designated by the identification number of the divided sub-three-dimensional virtual space.

FIG. 12 shows a simple setting example of the right hand posture. Referring to FIG. 12, first, as a step 1, the starting point position of the right hand is determined. For example, if the start point is the right front of the face, the number of the sub three-dimensional virtual space near the right front of the face is set. The result is displayed by determining the posture of the right hand, for example, by employing the inverse kinematics method. In addition,
In this case, a kinematic and a display system are required as a simple browser. If the position of the right hand is different from the desired one, the number of steps in the XYZ-axis direction (local coordinate system) to be moved is set from the current block number or position, and a new sub- 3
A two-dimensional virtual space is set (step 2). By repeating the operation of step 2, a desired position can be set. The advantage of this designation method is that the position can be roughly set, for example, to the right or behind a little, in the manner in which a human performs choreography of motion. This designation is set for the necessary three-dimensional virtual space of the start point position, the end point position, and the lapse point (step n, step n + 1, etc.).

In the description so far, mainly the setting of the movement of the skeleton has been mainly described. What the CG character actually wants is the skin information bound to the posture of the skeleton. As described above, the skin shape and the skeleton segment can be allocated by sharing the local coordinate position. In the kinematics method, it is necessary to set a skeleton in a portion that needs to be moved. FIG. 4 or FIG. 10 shows the hierarchical relationship of the skeleton generally used. However, for example, if you want to set the shape of a human fingertip such as the hand being open or the bending of the finger, it is sufficient to define the skeleton of the finger in the same way, but simply defining the skeleton of the finger, The data volume increases.

A method for fixing the shape of the hand and replacing desired shape data as necessary will be described in order to reduce the data amount. As the shape data to be bound to the right-hand and left-hand joints of the hierarchical structure shown in FIG. 4, for example, a shape in which the hand is spread or a shape in which the hand is gripped is prepared in advance. Further, as shown in FIG. 11, when a right hand motion is given, the motion element data
Along with the information of the three-dimensional virtual space, the type of shape data to be bound to the wrist and the twist information are added and transferred. The motion element data connecting unit 35, restores the trajectory of movement in the sub-three-dimensional virtual space Majo report, at the same time, judges that the twisting of the wrist, or the shape that is bound to the wrist looks like, this determination , The hand shape can be accurately reproduced.

Next, a description will be given of a method of expressing an expression that is important in reproducing a CG character. When expressing the facial expression realistically, the shape of the head can be bound to the joint corresponding to the neck of the hierarchy shown in FIG. 4, but in synchronization with the movement of the body, the facial expression such as "laughing" or "angry" Is difficult to reproduce. Therefore, for the expression, a method of mapping a live-action moving image to a head shape is used.

FIG. 10 is a conceptual diagram for explaining expression mapping. Referring to FIG. 10, the shape of the head when mapping is performed is basically a sphere or a cylinder.
Shape data that reproduces the unevenness of the face is used. A mapping method such as a mapping coordinate or a projection direction is set for this shape. In addition, in consideration of the case where the skeletal neck turns to the right or to the left, the human face is viewed from a plurality of camera positions (usually on a circle surrounding the face) as mapping data. The video sequence of the expression at the time (MPEG2 (Mot
ion Picture Experts Group phase 2) Prepare compressed images). When the direction of the neck is determined, it is determined from which moving image sequence (viewpoint-dependent data) the facial expression mapping data is to be selected. If necessary, the image data corresponding to the corresponding time from a plurality of moving image sequences is selected. The pixels are weighted with respect to the corresponding pixels by an angle from the line of sight, and are displayed in a so-called merged manner. For example, when the viewpoint for observing the CG character is between A and B shown in FIG. 10B, when A and B are selected as moving image sequence data and the center of gravity of the character is viewed from A, B and the viewpoint Is determined (proportional to the angle difference) in accordance with the angle difference of.

In the CG animation data transmitting section 1,
A mechanism for transferring the compressed data and camera position of the moving image sequence of the expression using MPEG2 or the like is required. on the other hand,
The CG animation data receiving unit 3 needs a mechanism for restoring the compressed and transferred moving image sequence and selecting and mapping which viewpoint image is based on the camera position.
Please refer to "Foley & Van
Dam et. Al.,: Computer Graphics Principles and Pra
ctice II, Addison Wesley ".

FIG. 13 is a diagram showing an example of script notation of movement for three handles of the right hand, the left hand, and the head. Referring to FIG. 13, the numbers of the sub-three-dimensional virtual space of the starting point, the ending point, and the lapse point are set as the trajectory information, and the actual reproduction time is transferred as additional information. In this example, a spline is specified as the interpolation method.

FIG. 14 is a diagram showing an example of a method of transmitting and receiving motion data having a CG skeleton structure. Referring to FIG. 14, between CG animation data transmitting unit 1 and CG animation data receiving unit 3 , the entity of the motion data represented in the sub-three-dimensional virtual space and the motion description language (for example, FIG. 13)
CG animation data can be transmitted and received without depending on the performance of various calculations and drawing, and hardware such as an OS (Operating System).

Finally, a brief description will be given of a method of checking a collision between two CG characters when there are a plurality of CG characters. As shown in FIG. 6, one of the advantages of expressing motion in the sub-three-dimensional virtual space is simplification of calculation of a collision check. For example, when roughly determining whether or not one arm and the other hand collide, consider the boundary of a segment included in the arm of interest of one character. The boundary is a rectangular parallelepiped determined by the parent site and the child site of the arm having the skeletal structure shown in FIG. The base area of the cuboid is determined by the size of the skin, and is a fixed value according to the physique in a normal human. If a collision is determined between the boundary and the sub-three-dimensional virtual space (rectangular parallelepiped) of the other character, rough determination can be easily performed. Further, when it is desired to determine the collision in detail, the collision may be determined for the shape data (all vertices of the polygon group) included in the boundary. By performing the collision determination (inside / outside determination between the boundary and the point of interest) in the sub-three-dimensional virtual space, the calculation of the collision check is greatly simplified (speeded up). It should be noted that the present invention is not limited to the above-described embodiment, and it is needless to say that various design changes and modifications can be made within the scope of the present invention.

[0089]

As is apparent from the above description, according to the present invention, the character 3 having a complicated structure like a human being can be obtained.
When transmitting and receiving three-dimensional movements, the human hierarchy is defined by a skeletal structure, and the structure is defined by the amount of movement of some joints in a three-dimensional virtual space in time series or representative movement elements and their connection methods. By additionally transmitting and receiving, it is possible to transmit and receive a natural and smooth movement of the character in the transmission / reception system based on the network, and it is possible to greatly reduce the data amount at the time of transfer.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a motion transmitting / receiving apparatus having a three-dimensional skeleton structure according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating an example of a configuration of a CG animation data transmission unit.

FIG. 3 is a block diagram illustrating an example of a configuration of a CG animation data receiving unit.

FIG. 4 is a diagram illustrating an example of a hierarchical relationship of a human skeleton structure.

FIG. 5 is a diagram illustrating an example of a data format of a skeleton structure.

FIG. 6 is a diagram for explaining a method of dividing the movement of the skeletal structure.

FIG. 7 is a diagram for describing a motion division sub-three-dimensional virtual space.

FIG. 8 is a diagram for explaining restoration of a motion from a sub-three-dimensional virtual space into a motion trajectory;

FIG. 9 is a diagram for explaining a method of changing to a joint angle when a sub three-dimensional virtual space is given.

FIG. 10 is a diagram showing an example of a method of mapping a basic human skeleton and a face shape.

FIG. 11 is a diagram for explaining connection between basic movements.

FIG. 12 is a diagram for explaining an example of a sequence for generating motion data from human motion.

FIG. 13 is a diagram illustrating an example of a script expression of motion data having a CG skeleton structure.

FIG. 14 is a diagram illustrating an example of a method of transmitting and receiving motion data having a CG skeleton structure.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 CG animation data transmission part 2 CG animation data transmission part 3 CG animation data reception part 11 Skeletal structure definition part 12 Motion data definition part 13 Motion element data decomposition part 31 Skeletal structure definition part 32 Shape data correspondence part 33 Motion data definition part 34 attitude determination unit 35 motion element data connection unit 36 motion connection unit

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-8-180208 (JP, A) JP-A-9-81782 (JP, A) JP-A-10-40419 (JP, A) JP-A-10-108 222698 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 1/00 G06T 11/60-17/50 G06F 13/00 JICST file (JOIS)

Claims (4)

(57) [Claims] [Claim 1] Joints and end sites in three-dimensional computer graphics from a transmitting side and connecting them
Transmits the motion data of the skeleton structure whose segments to the receiving side, a device for generating an animation based on the motion data of the transmitted skeleton structure at the receiving side, a hierarchical data for defining the skeleton structure, Of the above skeletal structure
Basic operation data defining a part of the exercise, and the basic operation
Transmitting means for transmitting connection method information between data, the hierarchical data, the basic operation data, and the connection
Receiving the method information, the hierarchical data, and the basic operation.
To determine the posture of the above skeletal structure based on crop data
In addition, the basic operation data is stored in accordance with the connection method information.
A motion transmitting / receiving apparatus having a three-dimensional skeletal structure , comprising: receiving means for connecting . 2. A joint, an end site and three-dimensional computer graphics in three-dimensional computer graphics are transmitted from a transmitting device.
Nag sends motion data of the skeleton structure whose segments to the receiving side, a <br/> apparatus in a system for generating an animation based on the motion data of the transmitted skeleton structure at the receiving side, the receiving Side to determine the posture of the above skeletal structure,
Hierarchical data defining the skeletal structure, and
Basic motion data that defines a part of the movement and the basic motion
Connection method information used to connect data,
Motion transmission instrumentation <br/> location of a three-dimensional skeleton structure characterized by comprising a transmission hand stage, for transmitting. 3. Three-dimensional computer graphics
Joints, end sites and segments connecting them
Of motion data of the skeletal structure consisting of
An apparatus for generating a skeleton structure, the hierarchical data defining the skeleton structure;
Basic motion data that defines a part of the movement and the basic motion
Connection method information used to connect data
Receiving means for receiving the hierarchical data and the basic operation data,
Used to determine the attitude of the structure, the connection method information above
The dynamic-out reception apparatus of a three-dimensional skeleton structure, characterized in that utilized to connect the basic operation data. 4. A three-dimensional computer graphic from a transmitting side.
Joints, end sites, and connecting them
Send motion data of skeleton structure consisting of segments to the receiving side
The receiver sends the transmitted skeleton structure motion data
A method for generating an animation based on the hierarchical data defining the skeletal structure;
Basic operation data defining a part of the exercise, and the basic operation
Transmitting connection method information between data, the hierarchical data, the basic operation data, and the connection
Receiving the method information, the hierarchical data, and the basic operation.
When the posture of the above skeletal structure is determined based on the work data
In both cases, the basic operation data is stored in accordance with the connection method information.
Motion transmission and reception side of the three-dimensional skeleton structure, which comprises the steps of: connecting Te
Law .