JP2003331310A - Image/voice interlocking output device and method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a moving image compiler for music producing an image whose CG object is changed by being interlocked with changes in voice information used in a musical piece. <P>SOLUTION: The voice information such as the intensity, the length, and the strength of the voice is obtained from MIDI (musical instrument digital interface) data in the musical piece, the CG object in the CG image is moved, rotated, changed in color, and changed in shape using the voice information. A recording medium recording a moving image compiling program for music is provided in an image reproduction procedure reproducing the image and the voice based on the image and voice data set thereby. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a computer and synchronized output of music and computer graphics video, and more particularly to the creation of computer graphics video using audio information used in music.

[0002]

2. Description of the Related Art With the recent increase in speed and cost of personal computers, digital editing technology for producing long-time moving images on a computer while interweaving video and audio materials has become widespread. Expressions using this technology are widely used for a variety of purposes, from television broadcasting to Internet homepages. Recently, an attempt has been made to stream music videos by matching the music with music created by music artists.
Attention has been paid to its use for V. In addition to MTV, for example, when a concert is held at a large venue,
Huge screens have been installed on both sides of the stage, and images using computer graphics (hereinafter CG) are being played at the same time as the performance. Also, in the entertainment space called a club where you can enjoy dancing to dance music (techno, hip hop, etc.), it is commonly known as VJ (Visua
A style established by a visual artist called “Jockey” that visually enhances the atmosphere of the venue by displaying CG images on the screen in time with music.

FIG. 1 shows a conventional embodiment for producing such a moving image for music. In the music production device 4, the MID that constitutes the audio information of one song or a part of the song.
Create music materials such as I data and voice waveform data.
The created music material is stored in the storage device 7. Commercial software is CUBASE from STEINBERG.
Such as composition software and SONIC FOUNDAR
Waveform editing software such as Y company's SOUND FORGE corresponds to this. The image creating device 7 creates an image material constituting image information for one song or a part of the song. The created video material is stored in the storage device 7.
Among commercially available software, this is software for creating CG images such as software TRUE SPACE of CARIGALI. In the video editing device 5, the audio material created by the audio creating device 4 and the video material created by the video creating device 7 are sequentially arranged in time sequence using the input device (FIG. 2) to create a music video. To do. Commercial software SHOCKWA from MARCOMEDIA
Authoring software such as VE, and MEDIA
This is the case with VJ software such as RAVE GEAR from VISION.

[0004]

However, in the CG image for music, the music piece is changed by changing the CG object in association with the change of the audio information (size, length, strength, etc.) used in the music piece. Although it is possible to create an effective image in which the development of is reflected in the image change, in the conventional embodiment, since the production process of the video material and the music material is independent as described above, It was not easy to create such a video because the audio information in the music could not be used. The CG object mentioned here is a 3D object representing a virtual object (for example, a car or a robot) arranged in a three-dimensional virtual space in 3DCG image creation software such as TRUE SPACE of CARIGALI, or PHO of ADOBE.
It means a 2D object having two-dimensional video data created by software for creating a 2D image such as TOSHOP and stored in the GIF format or the like. The CG object has shape information and color information in both 3D and 2D.

Therefore, an object of the present invention is to provide a moving image editing apparatus for music capable of creating a video in which a CG object changes in association with a change in audio information used in a music piece.

[0006]

In order to solve the above-mentioned problems, the invention of claim 1 specifies a CG object specifying procedure for specifying a CG object in synchronous output of music and CG video using a computer, and A CG object placement procedure for designating a display position of the designated CG object, a MIDI data designation procedure for designating MIDI data, a MIDI data placement procedure for designating a reproduction start position of the designated MIDI data, and video data including a CG object. Audio and video data associating procedures for associating audio data including MIDI data, parameter setting procedures for specifying information on changes in CG objects, and video and audio data based on the video and audio data set by the above procedure. And a CG image reproduction procedure for reproduction. It is music for the motion picture editing method and.

According to a second aspect of the invention, in synchronous output of music and CG video using a computer, a CG object designating procedure for designating a CG object and a CG object arranging procedure for designating a display position of the designated CG object. And MIDI that specifies MIDI data
Data specifying procedure, MIDI data arranging procedure for specifying the reproduction start position of the specified MIDI data, audio data / video data associating procedure for associating the video data including the CG object with the audio data including the MIDI data, and the change of the CG object And a CG video playback procedure for playing back video and audio based on the video data and audio data set by the above procedure, and a video editing apparatus for music. is there.

According to a third aspect of the invention, in synchronous output of music and CG video using a computer, a CG object designating procedure for designating a CG object and a CG object arranging procedure for designating a display position of the designated CG object. And MIDI that specifies MIDI data
Data specifying procedure, MIDI data arranging procedure for specifying the reproduction start position of the specified MIDI data, audio data / video data associating procedure for associating the video data including the CG object with the audio data including the MIDI data, and the change of the CG object And a CG video playback procedure for playing back video and audio based on the video data and audio data set by the above procedure, and a video editing program for music. It is a storage medium that has been stored.

[0009]

According to the present embodiment, it is possible to create an effective CG image in which a CG object changes in association with a change in audio information.

First, the data structure of MIDI data and CG object will be described. MIDI is a standard created for exchanging information that encodes common characteristics of musical instruments such as pitch and strength of playing a keyboard and performance information for controlling MIDI musical instruments between different MIDI musical instruments. As specified by MIDI standard 1.0, M
The IDI data is composed of a MIDI message string composed of a plurality of bytes, and the MIDI message has a data structure that is a set of a status byte indicating the type of message and a data byte indicating the actual data of the status (FIG. 3). M determined by the value of the status byte
The types of IDI messages include the start of pronunciation (note-on below), the stop of pronunciation (note-off below), and the relative change in pitch (pitch-pend change below). MID
In the data byte portion of the I message, different actual data is stored for each type of MIDI message. For example, in the data byte portion of the message in which the value of the status byte is note-on, the pitch (hereinafter note number) to be pronounced and the sound intensity (hereinafter velocity) are stored. The data used in the music is stored in the storage device as a set of a data unit including an index area, a tone generation timing area for designating the timing of producing a MIDI message, and a MIDI message area. (Figure 4)

The CG object is a virtual object arranged in the virtual space of the CG image as described above, and has shape information and color information. The CG object is
The index area, the type area for specifying the type of CG object, the position information area for specifying the position in the virtual space, and the CG object area are stored in the storage device in a data format (FIG. 5). CG object type is 2
In the case of D object, the value indicating 2D in the type area and the position in the two-dimensional virtual space (X coordinate / Y coordinate) in the display position area
Is held, and in the case of a 3D object, a value indicating 3D is held in the type area and information on the position (X coordinate, Y coordinate, Z coordinate) in the three-dimensional virtual space is held in the display position area.

FIG. 6 shows the overall structure of an embodiment of the present invention. The input device 12 is used by the operator to give commands to the device, such as reading MIDI data and reproducing a moving image for music. The storage device 17 stores MIDI data and CG object data that can be used in moving images, and association data described later.

The operator first uses the music making apparatus 14 similar to the above-mentioned music making apparatus 4 to use the MIDI used in the song.
After creating the data and creating the CG object to be used in the video in the song by using the video creating apparatus 16 similar to the above-described video creating apparatus 6, the music moving image editing apparatus 15 is used to create the music moving image. To do. The operator selects a CG object to be used in the moving image for music by using the input device 12, and while watching the display device 13, uses the input device 12 to perform CG.
Specify the display position of the object. Further, the operator selects the MIDI data to be used in the moving image for music with the input device 12, and while watching the display device 13, the input device 12
Is used to specify the sounding timing of MIDI data.
Next, when an instruction is issued from the input device 12 to associate the CG object with the MIDI data, the association data is created by the music action creating device 15 and stored in the storage device 17.

The association data is composed of a music area and a video area. In the music area, a MIDI data index designating MIDI data, a use parameter list for designating a data item used for changing a CG object in audio information obtained from a MIDI message, a maximum value and a minimum value of the data to be used. Holds the value. In the video area, a CG object index that specifies the CG object, movement data that specifies the movement width of the CG object, rotation data that specifies the angle at which the CG object rotates, and a color that specifies the ratio at which the color of the CG object changes. Information on change data and shape change data that specifies the rate at which the shape of the CG object changes is held. Furthermore, the operator uses the input device 12 to specify the MIDI data of the association data and the CG object,
The data item to be used is specified from the use parameter list of the associated data, and the maximum value, the minimum value, the movement data, the rotation data, the color change data, and the shape change data are specified.

When the reproduction is instructed from the input device 12, the music / moving image editing device 15 starts reproducing the moving image. The time code of the music video editing apparatus is incremented each time the reproduction progresses. The MIDI message data when the time code reaches the sounding timing is sent to the audio device 11 by the music video editing apparatus 15, and at the same time, the CG object associated with the MIDI data by the association data is changed based on the video parameter of the association data. The video information having the contents described above is sent from the audio / video editing device 15 to the display device 13. The audio device 11 outputs audio according to the contents of MIDI data, and the display device 13 outputs video according to video information.

The hardware configuration when the present invention is implemented by using a CPU is as shown in FIG. The computer is composed of a CPU 301, a RAM 302, and an auxiliary storage device 303 such as a hard disk, and a medium loading device 3 for loading a recording medium 308 such as a floppy (registered trademark) disk or a CD-ROM.
04, an input interface 305 for mounting an input device 309 such as a keyboard and a mouse, a screen display interface 306 for mounting a display 310, and a MIDI for mounting a MIDI sound source 311.
Interface 307 is installed. Further, the input device 308 is connected to the input interface 305, the display 310 is connected to the screen display interface 306, and the MIDI sound source 311 is set to MIDI.
Interface 307.

When the operator uses the input device 309 to instruct the execution of the program of the present invention while the recording medium 308 containing the program of the present invention is attached to the medium attachment device 304, the recording medium 308 is recorded. The program stored in is read. And the CPU 301
Writes the read program in the RAM 302 or the auxiliary storage device 303 and executes the program.

The overall structure of the program of the present invention will be described with reference to the flowchart of FIG. Here, as an example, MIDI data 1 composed of a MIDI message (note number D3, note on, velocity 100) whose sounding timing is 96 and a MIDI message (note number E3 note on, velocity 100) whose sounding timing is 192,
MIDI contained in MIDI data 1 using a 2D white semi-circle CG object 1 with a diameter of 100 pixels
Depending on the note number of the message, position CG object 1 from X = 100, Y = 100 to 1 in the X direction.
It is assumed that the color is changed to black and the size is changed to double by moving 100 in the 00Y direction and rotating 180 degrees.

First, while observing the display 310, the input device 309 is used to specify, from the CG objects in the auxiliary storage device 303, what is to be used for video output in the music (S101). Here, it is assumed that the CG object 1 has been read. Then, the CPU creates the video data 1 including the information of the CG object 1 and holds it in the RAM 302 or the auxiliary storage device 303. At this time, the value of the position information is empty. (State after completion of S101 in FIG. 10)

Next, the display position of this CG object in the virtual space is designated (S102). Where X =
Specify 100 and Y = 100. Then the CPU is CG
X = 100, Y = 10 in the position information area in the object
Set a value of 0. (State after completion of S102 in FIG. 10)

Next, while looking at the display 310, the input device 309 is used to specify, from the MIDI data in the auxiliary storage device 303, what is to be used for audio output in the music (S103). Here, MIDI data 1 is designated.
Then, the CPU creates the voice data 1 including the MIDI data 1 and holds it in the RAM 2 or the auxiliary storage device 3.
(State after completion of S103 in FIG. 11)

Next, the reproduction start position in the entire piece of MIDI data is designated (S104). Here, the playback start position is specified as 96. Then, the CPU recalculates the value of the tone generation timing area and resets the value to 192,288, respectively. (State after completion of S104 in FIG. 11)

Next, while looking at the display 310, the audio data and the video data to be associated are designated using the input device 309 (S105). Then, the CPU 301 newly creates association data and stores it in the RAM 302 or the auxiliary storage device 303. Also, the CPU 301 sets the value of the index area of the audio data in the MIDI index of the association data and sets the value of the CG object index area in the CG object index. Here, when the video data 1 and the audio data 1 are specified, the association data 1 is created, and the MIDI index is “md
"ad1" and "cgad1" are set in the CG object index (state 1 in S105 of FIG. 14).

Further, the CPU obtains the use parameter list from the contents of the voice data and sets it in the use parameter list of the association data.

FIG. 12 shows the procedure for acquiring the used parameter list.
It is shown in the flowchart. First, I = 1 (S20
1) The message type name is acquired from the status byte of the I-th MIDI message in the audio data (S202). Here, the first MIDI of audio data 1
The message type name “NOTE ON” is acquired from the message.

If the MIDI message includes actual data (S203), the sum of the message type name and the item name of the actual data is set for the number of items of the actual data (S204). Here, note-on has two items, note number and velocity, in the actual data part, so "note-on: note number" and "note-on: velocity" are set. If the MIDI message has no data bytes in the actual data part, S204 is skipped. For example, if the type name of the message is start (instructing to start playing the music), the MIDI message has no actual data, and S204 is skipped.

If the message type name of the same name is not included in the parameter list (S205), the message type name is added to the used parameter list (S206).
Here, add "Note On-> Note Number" and "Note On->Velocity" to the parameter list.

The value of I is incremented (S207),
If the MIDI data contains the I-th message, S
The processing returns to 202 to continue the processing, and if not, the processing ends. Here, since the audio data 1 has two MIDI data, the process is repeated until I = 2, and when I = 3, the process ends. (State after completion of S105 in FIG. 14)

Next, the operator selects and specifies the numerical value of each parameter of the association data using the input device 309 while looking at the display 310 (S106). Here, select "Note On-> Note Number" from the parameter list to use, minimum value C3, maximum value E3, movement range data (X = 100, Y = 100), rotation data (clockwise 180
Degree), color change data (black), and shape change data (1). (State after completion of S106 in FIG. 14)

After the operations of S101 to S106 are repeated to complete the arrangement of the audio data and the video data used in the moving image and the arrangement of all the materials is completed (S107), the reproduction is instructed using the input device 309. (S108), the time code advances as time advances, and the MIDI message when the time code reaches the sounding timing is the CPU.
It is sent to the MIDI sound source 311 via the MIDI interface 307 by 302. At the same time, CP
U302 is a C that has the MIDI data as a data source
The G object is changed using the parameter value of the association data, and the changed image information is sent to the display by the CPU 2 via the screen display interface 6. Here, when the time code reaches 192, CP
The U302 sends the first MIDI message of MIDI data 1 to the MIDI sound source 311 via the MIDI input / output interface 307. At the same time,
The CPU 302 uses this MIDI message to CGI
The object 1 is changed, and the changed image information is displayed on the display 3 via the image display interface 306.
Sent to 10.

FIG. 13 is a flow chart showing a method of changing a CG object using MIDI data.

First, the target data is read from the MIDI data based on the item selected in the use parameter list of the association data (S301), and the change rate is calculated from the maximum value and the minimum value of the association data (S302). .
When an item that does not have actual data is selected in the used parameter list, the rate of change is automatically set to 1. Here, MI
Since the note number of the DI data is read and the note number D3 is an intermediate value between the maximum value note number E3 and the minimum value note number C3, the rate of change is 0.5.

Next, the changed position of the CG object is set (S303). The changed position is the original position + (moving data of the association data × change rate). Here, the position after the change of the CG object 1 is X coordinate = original X
Coordinate 100 + (Change width 100 * Change rate 0.5) = 150, Y coordinate = Original Y
Coordinates 100 + (change width 100 * change rate 0.5) = 150 is set.

Next, in the same manner, the rotation angle of the CG object after change (= rotation data of association data × change rate)
0.5) (S304), color (= color data of associated data x
Change rate 0.5) (S305) and shape change (= 1 + shape change data of association data × change rate 0.5) (S306) are set. Here, the rotation angle after change of the CG object 1 is changed to 180 × 0.5 = 90 degrees, the color is changed to an intermediate color between white and black, and the shape change is changed to 1+ (1 × 0.5) = 1.5 times.

Next, the time code advances, and the time code
When it reaches 288, the MIDI message is also MID
It is sent to the I sound source 311 and, at the same time, the rate of change of the CG object is 1. Position information is X coordinate = 200 Y coordinate = 200, rotation angle 180
Degree, color = black, shape change = 2, display 31
Sent to 0.

The audio device 311 outputs audio according to the contents of the MIDI data, and the display 310 outputs video using the video information.

By adopting such an embodiment, voice information such as the volume, length, and strength of the sound is acquired from the MIDI data in the music, and the CG object in the CG image is utilized by using the voice information. It is possible to create a moving image for music that can be moved, rotated, changed in color, and changed in shape.

[0038]

[Brief description of drawings]

FIG. 1 is a conventional embodiment.

FIG. 2 is an image of arrangement of music material and video material by a conventional moving image editing apparatus.

FIG. 3 is a data structure of a MIDI message.

FIG. 4 is an example of audio data stored in a storage device.

FIG. 5 is an example of video data stored in a storage device.

FIG. 6 is an overall configuration of the present invention.

FIG. 7 is an example of association data.

FIG. 8 is a hardware configuration of a computer using a CPU.

FIG. 9 is a flowchart showing the configuration of a moving image production program.

FIG. 10 is a state of video data in the example.

FIG. 11 is a state of audio data in the example.

FIG. 12 is a flowchart of a procedure for acquiring a use parameter list.

FIG. 13 is a flowchart of a procedure for changing a CG object using MIDI data.

FIG. 14 is a state of video data in the example.

Claims (3)

[Claims] 1. A CG object designating procedure for designating a CG object, a CG object arranging procedure for designating a display position of the designated CG object, and a MID in synchronous output of music and CG video using a computer.
MIDI data designation procedure for designating I data, MIDI data placement procedure for designating playback start position of designated MIDI data, audio data / video data association procedure for associating video data including CG object and audio data including MIDI data And a parameter setting procedure for designating change information of the CG object, and a CG video reproduction procedure for reproducing video and audio based on the video data and audio data set by the above procedure. Video editing method for music 2. A CG object specifying procedure for specifying a CG object, a CG object arranging procedure for specifying a display position of the specified CG object, and a MID in synchronous output of music and CG video using a computer.
MIDI data designation procedure for designating I data, MIDI data placement procedure for designating playback start position of designated MIDI data, audio data / video data association procedure for associating video data including CG object and audio data including MIDI data And a parameter setting procedure for designating change information of the CG object, and a CG video reproduction procedure for reproducing video and audio based on the video data and audio data set by the above procedure. Video editing device for music 3. Synchronous output of music and CG video using a computer, a CG object designating procedure for designating a CG object, a CG object arranging procedure for designating a display position of the designated CG object, and a MID.
MIDI data designation procedure for designating I data, MIDI data placement procedure for designating playback start position of designated MIDI data, audio data / video data association procedure for associating video data including CG object and audio data including MIDI data And a parameter setting procedure for designating change information of the CG object, and a CG video reproduction procedure for reproducing video and audio based on the video data and audio data set by the above procedure. Storage medium storing a video editing program for music