JPH10307930A - Animation production system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To continuously move an object of CG(computer graphics) in response to music and also to facilitate an editing job. SOLUTION: An object is contracted in a virtual space based on the model data that defined the shape of the object, and a series of motions are given to the object based on the motion data consisting of a series of frame data. A time frame Ta is set based on the beat time information included in the MIDI(musical instrument digital interface) data. Then the motion data are compressed or expanded so as to secure the matching between the frame Ta and a series of motions of the object, and the frame data corresponding to every display frame shown in the frame Ta are generated. The motion data which are compressed or expanded synchronously with reproduction of music are synthesized with the object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an animation creating system for moving an object in a virtual space generated by computer graphics (hereinafter referred to as CG) in synchronization with music.

[0002]

2. Description of the Related Art In order to synchronize animation with music, generally, a method of dividing frames according to the time element from music data, creating a still image of each frame, and arranging them in chronological order as a moving image is known. Be taken. For example, in order to fit a specific video at a specific timing of music, such a specific video is prepared as a key frame, and the music data is observed with a waveform along a time axis. , Etc.) and assigning key frames, and then interpolating between key frames to create an entire animation.

A major disadvantage of such a traditional method is that animation creation has no real-time property.
Evaluation of the production result is impossible only after all animations are completed. For this reason, the creation and editing work of the animation is trial and error and takes a lot of trouble. In addition, there is a problem that the degree of synchronization between the completed video and the music greatly depends on the experience and intuition of the creator. Furthermore, if the music is changed, the animation must be recreated.

On the other hand, in recent years, a system for editing animation by three-dimensional CG (hereinafter referred to as 3DCG) in real time has been developed along with the improvement of drawing performance of a CG chip (“Development and operation of a real-time character animation system”). , Kokura, NICOGRAPH Transactions 1993, p.
p131-139). Also, by controlling the frame control of moving images by 3DCG with dynamic time stamps, 3D
A real-time-capable CG capable of associating CG with real-time has also been proposed ("Real-time interactive 3D CG simulator", Suzuki et al., NICOGRAPH Transactions 199).
3, pp149-158). Furthermore, there has been proposed a system for generating a CG in which the performance information of music such as strength is correlated from a psychological point of view in accordance with the performance of an electronic musical instrument in real time ("Research on music visualization by computer graphics"). , Haruguchi, NICOGRAPH Transactions 1994, pp112-11
9).

[0005]

However, among the above-mentioned conventional systems, the first system is a system in which an operator operates a data glove to move a CG in accordance with an actual video or music. Is dependent on the skill level of the operation of the operator. In addition, for these reasons, the operation of the object is naturally limited. The second system focuses on matching the time difference between the CG and the real world by frame control of the CG image, and does not consider synchronization with music at all. Further, the third system generates a sound object (for example, a snare drum sound is an image of a scattered small sphere) for visualizing the timbre from a human psychological point of view, and generates the sound object using M.
The object is drawn in synchronization with the IDI event information.
Continuous movement cannot be synchronized with music.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and has as its object to provide an animation creating system that can continuously move CG objects in time with music and that can easily perform editing work. I do.

[0007]

A first animation creation system and an animation creation program stored in a medium according to the present invention construct an object in a virtual space based on model data defining the shape of the object. In giving a series of motions to the object based on motion data composed of a series of frame data, extracting the timing information from music data including timing information, and matching the rhythm of the music data based on the timing information. Setting the time frame, and compressing or expanding the motion data so that a series of movements of the object match the time frame to generate the frame data corresponding to each display frame displayed in the time frame. At the same time as the reproduction of musical tones by the music data. Is not characterized by being adapted to synthesize the object from said compressed or decompressed motion data and the model data.

A second animation creating system according to the present invention comprises: a model data storage means for storing model data defining a shape of an object in a virtual space; and a series of movements of the object based on a time series of frame data. A motion data storage means for storing the defined motion data; and a MIDI (Musical Instrument Digit) defined for generating a predetermined musical sound.
al Interface) MIDI data storage means for storing data, and MID stored in the MIDI data storage means
A tone generating means for generating a tone based on the I data; and a time frame set in units of the beat time based on beat time information included in the MIDI data, wherein a series of movements of the object are determined by the time frame. The motion data is compressed or decompressed so as to match the frame data to generate the frame data corresponding to each display frame displayed in the time frame, and the compression or expansion is performed in synchronization with the reproduction of the musical sound by the MIDI data. An animation creating means for combining the object from the decompressed motion data and the model data stored in the model data storage means, and a display means for displaying the object combined by the animation creating means. And

According to the first animation creating system of the present invention, the timing information is extracted from the music data including the timing information, and a time frame (for example, a time of one beat or one bar) that matches the timing information is set. At the same time, by compressing or expanding the motion data so that the motion of the object matches the time frame, a series of motions of the object are accommodated in the time frame. This makes it possible to easily realize continuous motion control such as making the object dance to music, for example. According to the present invention, an object is generated by model data defining its shape and motion data defining its operation, and data processing for synchronizing with music is performed only on motion data. The amount of data processing is small, and real-time processing is facilitated. Even when the music data is changed, the motion data is compressed or decompressed so as to match the music data based on the timing information included in the music data, so that it is not necessary to change the model data or the motion data. Further, even when the motion data or the model data is changed, other data is not affected.

As the music data, for example, MIDI data can be used. In the second animation system of the present invention, a time frame based on the beat time is set by extracting beat time information included in the MIDI data. By using the beat time as a reference, the movement of the object can be most matched to the music.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a diagram showing a configuration of an animation creating system according to one embodiment of the present invention. The computer 1 has a function of executing processing for creating / editing a 3DCG object as an animation element and MIDI data (MIDI file) as music data, and synchronizes the created object with the MIDI data. And a function of executing a process for synthesizing and creating an animation in real time. The computer 1 includes a computer main body 1a including software for executing the above processing;
It comprises a keyboard 1b and a mouse 1c as input means necessary for creating objects and MIDI data, and a display 1d for displaying the generated 3DCG. A data glove 2 is connected to the computer 1 via an interface 3 as external control means for designating a series of movements of an object. Also, a MIDI musical instrument 4 for creating MIDI data is connected to the computer 1 via a MIDI interface 5. The MIDI data read from the computer 1 is supplied to a MIDI sound source 6 via a MIDI interface 5 to become a musical tone signal, which is output as a musical tone via a speaker 7.

FIG. 2 is a functional block diagram of the computer main body 1a. The computer main body 1a includes a model data storage unit 11, a motion data storage unit 12, and a MIDI data storage unit 13 for storing model data, motion data, and MIDI data, respectively. These data are stored in a model data creating / editing unit 14, a motion data creating / editing unit 15, and MIDI.
The data creating / editing unit 16 creates and edits each independently. These data are synthesized and synchronized by the animation creating unit 17,
The display device 1d and the MIDI sound source 6 are used as a video signal and music data for displaying 3DCG, respectively.
It is supplied to.

The model data is constituted, for example, as follows. That is, as shown in FIG. 3, an object to be displayed (a human body model in the illustrated example) has a plurality of parts such as a face, a torso, and arms displayed by polygons connected by joints indicated by white circles in the figure. Has a link structure. The model data is composed of shape data and link data as shown in FIG. The shape data is a file that defines the shape of each part such as a face and a torso, as shown in FIG. 3A. For each of a plurality of polygons constituting the part, a vertex sequence, a face list, and a surface texture Etc. are defined. The link data is shown in FIG.
Is a file defining the link relation of each joint as shown in FIG. For example, in the case of a human body model, the pelvis is defined as the joint of the highest parent, the waist connected to it, the right crotch and the left crotch are defined as child joints, and the neck connected to the waist via the spine is defined as the grandchild's joint. Defines the link structure. Also, bones connecting the joints and parts corresponding to the bones are defined at the same time. For each joint, a three-dimensional angle is defined. As shown in FIG. 3, the joint angle is defined as a direction of a lower part based on the local coordinate system with the upper part (bone) as a local coordinate system. By defining such a link structure, the position of each joint can be obtained simply by giving the three-dimensional position of the parent hip bone in the virtual space.

As shown in FIG. 5, a series of movements can be given to the object by changing the joint angles of the joints in a time series. Motion data is data that defines such a series of motions, for example, as shown in FIG.
It is a file that defines the angles of each joint for each time. In addition, three-dimensional coordinate values of the parent at each time are also defined at the same time. Here, data such as the joint angle sequence at each time is referred to as “frame data”.

The MIDI data includes a list of MIDI events in which timing information is added to a MIDI message, meta events such as sequence numbers, text information, and tempo information. Time information necessary for animation synchronization is extracted from the MIDI data.

As the model data creating / editing unit 14 for creating / editing model data, known modeling software or the like can be used, and the model data is created using the keyboard 1b and the mouse 1b. . The motion data creating / editing unit 14 for creating motion data assigns each control element provided corresponding to each finger position of the data glove 2 to each joint of the object, and based on information obtained by the operation. For this purpose, a program for generating time-series key frame data is used. MI to create / edit MIDI data
As the DI data creating / editing unit 16, a MIDI sequence application is used. Here, a characteristic point is that the creation and editing operations of these data can be performed completely independently of each other.

As shown in FIG. 7, the animation creation unit 17 constructs an object in the virtual space from the model data, and compresses and expands the motion data from the time information contained in the MIDI data to generate a frame. A frame generation unit 32 for generating data;
An object drawing unit 33 for drawing an object for each frame based on the generated frame data;
A sound output instructing unit 34 that reads I data and sequentially outputs a MIDI message, and a control unit 35 that synchronizes the timing of frame generation, object construction, and object drawing based on the MIDI data with music. I have.

Next, a description will be given of an animation generation process in the animation creation unit 17 by synchronizing music and 3DCG. FIG. 8 is a diagram for explaining this processing. The MIDI data is configured based on a sequence of MIDI messages input in time units.
Since timing information is also added to the IDI event, the sound output instructing unit 34 that outputs a MIDI message based on the timing information and the control unit 35 perform necessary processing in synchronization with each other, so that the sound is output. Images can be synchronized. Here, a time frame Ta is set based on a beat time indicating one beat time, and the motion data is adjusted to the time frame Ta. For example, in the illustrated example, the beat time is set to the length of a quarter note because the time is 4/4. Since this length changes depending on the tempo information, the beat time is actually determined based on the information. In the first bar, the time frame Ta is set to the same time as the beat time, and in the second bar and thereafter, the time frame Ta is set to the beat time ×
4, that is, one bar. Also, the first
In the bar, motion data 1 is selected, and in the second bar, motion data 2 is selected.

Since the length of each of the motion data 1 and 2 does not coincide with the time frame Ta, the motion data 1 is compressed in the first bar, and the motion data 2 is expanded in the second bar and thereafter. Assign to Ta. In the compression / decompression processing, the number of frames displayed in the time frame Ta is set to N
a, when the number of frames of motion data is Nm,
A conversion rate r = Nm / Na is determined, and based on the conversion rate r, frame data is sequentially selected by compressing every r in the case of compression or by applying an appropriate interpolation process in the case of decompression. Good.

As described above, it is conceivable to change the motion or change the time frame Ta in the middle of the music.
As shown in the figure, the IDI data is pre-read for the next time frame Ta. If there is a change in the motion data or the time frame Ta due to the pre-read MIDI data, the IDI data is read before entering the next time frame. Frame number conversion processing is executed to create frame data for the number of frames Na.

FIG. 9 is a flowchart showing a frame creation process started at predetermined intervals according to the timing given by the control unit 35. First, the control unit 35 sets a time frame Ta (an appropriate default value at first).
The MIDI data of the minute is pre-read (S1), and information necessary for generating an animation and displaying its timing is extracted. Model data for constructing an object to be displayed and motion data for defining its operation are represented by MID.
It may be described in the I data, or may be set in the control unit 35 in advance. Control unit 35
Gives the model data number to be selected and the motion data number to the object construction unit 31 and the frame generation unit 32, respectively, when there is a change in the time frame Ta, motion data, and the like (S3). Control unit 35
Sets the time frame Ta from the read MIDI data (S4). That is, in the case of an SMF (Standed MIDI file), the beat time is obtained based on the timing information included in the MIDI event and the tempo information described as the meta event. Time frame Ta based on this beat time
Is set, and a series of movements of the object are adjusted to the time frame Ta. Also, which motion data is to be used may be arbitrarily set in advance, or may be embedded as a meta-event of the MIDI file. In the former case, the operation of playing the same song again with different objects and motions is easy,
In the latter case, there is an advantage that the operation of changing the movement of the object in the middle of the music becomes easy.

When the time frame Ta is determined, the control unit 35 calculates the number of frames Na to be displayed from the information of the time frame Ta (S5). Next, the number of frames Na, Nm
At the same time, a frame creation command is given to the frame generation unit 32. The frame generation unit 32 calculates the number of frames Na, N
m, a conversion rate r (= Nm / Na) is obtained.
, The motion data is compressed or decompressed to reconstruct the frame data (S7).

Subsequently, the control unit 35 sets the time frame T
The object drawing unit 33 is activated at the timing of the start of “a”. For this reason, the control unit 35 constantly monitors the current time of the sound output instructing unit 34 and obtains the timing of drawing the object. It is desirable that this timing is actually a timing preceding the note-on timing by the time required for drawing. In addition, as shown in FIG. 8, the display start timing (the start of each time frame) seems to match the rhythm when it precedes the note-on timing by τ. For this reason, the control unit 35 determines the amount of time preceding note-on in advance. The object drawing unit 33 sequentially reads out each frame data of the generated motion data. On the other hand, prior to this, the object construction unit 31 constructs an object in the virtual space based on the model data. Therefore, the object drawing unit 33 moves the position of each part of the object based on the sequentially read frame data and draws it in the memory. This drawing process is a continuous process independent of the frame generation process so that the 3DCG object is continuously displayed on the display device 1d. For this reason, the frame buffer used for frame generation has a ring buffer configuration, and the object drawing unit 33 reads frame data from the frame buffer of the frame generation unit 32 at a fixed timing.

By performing the above processing, as shown in FIG.
A series of motions of the object are compressed or decompressed so that they fall within a time frame based on the beat time of the music according to the MIDI data, and a continuous sound synchronized with the musical sound, for example, a human body model is danced to music. Animation with motion can be created. The above-described animation creation processing is realized by, for example, an animation creation processing program, and the program is provided by being recorded on a recording medium.

According to this system, the model data, motion data, and MIDI data can be created and edited completely independently of each other, so that they can be created in parallel, making the creation of animation extremely easy. become. Further, even when the music, the model data, and the motion data are changed, other data need not be changed. Further, since the model data and the motion data are completely independent, when different motions are given to the same object, only the motion data needs to be changed.

By the way, since motion data is a file in which a series of motions of an object is defined by a temporal change of a joint angle, multiplying a joint angle by a certain parameter makes the motion of the object as a whole a large motion. Or small movements. In addition, the motion of the object can be made faster or slower by temporal compression or expansion processing. By utilizing this, it is possible to reconstruct more deformed motion data with a smaller number of motion data.

For example, FIG. 10 shows an example in which motion data is converted by parameters of speed = 5 and width = 10. In this case, motion data having a speed 5 times and a width 10 times is generated from the basic motion data. Therefore, the contents of the data frames (key frames) at times 0, 1, 2,... ., The indirect angle) is increased by 10 times, and arranged as data frames at times 0, 5, 10,. Then, by interpolating between these data frames, converted motion data is obtained as shown in the figure. By performing such a conversion process, for example, as shown in FIG. 11, the operation of “walking” shown in FIG. The operation can be changed to "run" as shown in FIG.

If the motion data thus converted is subjected to the synchronization processing with the music data as described above, various deformation motions can be created by storing less basic motions. Therefore, for example, if the above-described parameters are written in the MIDI data, it is possible to easily give an operation change such as increasing the motion little by little from the middle of the music or conversely decreasing the motion.

Further, as shown in FIG. 6, the motion data is provided with point information (indicated by an asterisk in the figure) for synchronizing with the time signature of the music, and the display timing of the frame in which this point information is described By adjusting the operation start timing of the CG so as to match the first beat of the MIDI data, the synchronization adjustment between the music data and the CG is further simplified.

[0030]

As described above, according to the present invention,
By extracting the timing information from the music data including the timing information, setting a time frame that matches the timing information, and compressing or expanding the motion data so that the motion of the object matches the time frame, Since the series of movements of the object are set within the above-mentioned time frame, there is an effect that continuous operation control such as making the object dance to music can be easily realized.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an animation creation system according to an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of a computer main body in the system.

FIG. 3 is a diagram showing an example of an object displayed in the system.

FIG. 4 is a diagram showing a configuration of model data that defines the shape of the object.

FIG. 5 is a diagram for explaining the movement of the object.

FIG. 6 is a diagram showing the contents of motion data that defines the motion of the object.

FIG. 7 is a block diagram of an animation generation unit in the system.

FIG. 8 is a diagram for explaining a synchronization process between music and an object in the same system.

FIG. 9 is a flowchart of a frame generation process in the same process.

FIG. 10 is a diagram for explaining motion data deformation processing.

FIG. 11 is a diagram for describing motion data deformation processing.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Computer, 2 ... Data glove, 3 ... Interface, 4 ... MIDI musical instrument, 5 ... MIDI interface, 6 ... MIDI sound source, 7 ... Speaker, 11 ... Model data storage part, 12 ... Motion data storage part, 13 ... M
IDI data storage unit, 14: model data creation / editing unit, 15: motion data creation / editing unit, 16: MI
DI data creation / editing section, 17: Animation creation section.

Claims (3)

[Claims] An animation is created by constructing an object in a virtual space based on model data defining the shape of the object, and applying a series of motions to the object based on motion data composed of a series of frame data. In the system, the timing information is extracted from music data including timing information, a time frame matching the rhythm of the music data is set based on the timing information, and a series of movements of the object match the time frame. As described above, the motion data is compressed or decompressed to generate the frame data corresponding to each display frame displayed in the time frame, and the compressed or decompressed or decompressed data is synchronized with the reproduction of the musical sound by the music data. From the motion data and the model data, the object Animation production system is characterized in that so as to synthesize a-objects. 2. Model data storage means for storing model data defining the shape of an object in a virtual space; and motion data storage means for storing motion data in which a series of movements of the object are defined by a time series of frame data. MIDI (Musi) defined to generate a predetermined tone
MIDI data storage means for storing cal instrument digital interface (DATA) data; music tone generation means for generating a musical tone based on the MIDI data stored in the MIDI data storage means; and a beat time information included in the MIDI data. Setting a time frame in units of the beat time, compressing or expanding the motion data so that a series of movements of the object is aligned with the time frame, and setting each display frame to be displayed in the time frame. Animation creation for generating the corresponding frame data and synthesizing the object from the compressed or decompressed motion data and the model data stored in the model data storage means in synchronization with the reproduction of the musical sound by the MIDI data. Means and an object synthesized by the animation creating means Animation production system characterized by comprising a display means for displaying the door. 3. An object is constructed in a virtual space based on model data defining the shape of the object, and when a series of motions is given to the object based on motion data composed of a series of frame data, timing information is used. Extracting the timing information from the music data including the time data, setting a time frame that matches the rhythm of the music data based on the timing information, and setting the motion data so that a series of movements of the object matches the time frame. The frame data corresponding to each display frame displayed in the time frame by being compressed or expanded is generated, and the compressed or expanded motion data and the model data are synchronized with reproduction of a musical sound by the music data. Create an animation that combines the object from Medium storing the management program.