JPH08293039A - Music/image conversion device - Google Patents

Abstract

PURPOSE: To provide a music/image conversion device which can convert the music into an image that has the concreteness of visual art. CONSTITUTION: An action database 107 stores the actions of an articulate body. An action generation part 106 retrieves the actions out of the database 107 based on the chord detected by a chord detection part 105 when the leading point of sound volume is detected by a peak detection part 103. Thus the part 106 generates the action data. Thereby, such images that have the concrete shapes and movement, e.g. an image of a human being who is dancing on the music can be generated.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a music image converter for converting music information into an image.

[0002]

2. Description of the Related Art As an example of a conventional technique for converting sound into an image, there is a method (Japanese Patent Application Laid-Open No. 63-184875) in which the motion of the sound of each part of a piece of music corresponds to the motion of an image having a basic shape. It was

[0003]

However, in the above-mentioned conventional technique, the motion of sound is directly corresponded to the motion of the image, so that the obtained image is an abstract image having unrelated motion as the whole image. The above-mentioned conventional art has a drawback in that the visual art aimed at is limited to abstract art.

It is an object of the present invention to provide a music image conversion apparatus for converting music as auditory art into an image as concrete visual art.

[0005]

In order to achieve this object, the present invention has a motion database for storing motions of articulated objects, and a peak detection unit for generating timing for switching motions from a music signal; It has a chord detection unit that detects chords, and a motion generation unit that selects a motion from a motion database based on the chord at the time of switching the motion.

[0006]

[Function] Some motions of a joint object representing a shape such as a human being performed within a certain time are stored in advance in a motion database, and the motion is searched based on the chord structure of the part triggered by the rising part of the sound. To do. The music information is converted into a series of motion information by appropriately switching the motion for each rising part of the sound, and the motion is applied to the image of the joint object to form an image.

[0007]

【Example】

(Embodiment 1) An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram of an embodiment of the present invention.

First, the music signal input section 101 inputs music as an electric signal of voice. The volume detector 102
Volume information is extracted from the music information input by the music signal input unit 101. When the input music information is an electric signal of voice, the squared value becomes the volume information, and the envelope is obtained as the volume information signal. Peak detector 10
3 is from the volume information signal obtained by the volume detection unit 102,
The rising point of the volume is detected. By detecting only the rising edge that exceeds the average sound volume in a certain time immediately before (a few seconds), it is possible to prevent the rising edge of a fine sound from being detected.

The pitch detection section 104 includes a music signal input section 10
Interval information is detected from the music information input by 1. Frequency analysis is performed on the electrical signal of voice by using Fourier transform. The pitch used in music is a value that can be normally expressed by (Equation 1).

[0010]

[Equation 1]

F (0,0) is a reference sound of 440 Hz, n is a pitch of a semitone interval, and m is a pitch of an octave. Therefore, the frequency f expressed by (Equation 1)
The frequency component I (n, m) of (n, m) is used as the pitch detection unit 10
Get at 4. The chord detection unit 105 detects the pitch detection unit 1 at the rising point of the volume obtained by the peak detection unit 103.
The chord is detected based on the pitch information obtained in 04.
First, consider that sounds that are exactly octave apart are the same sound,
The components I (n) are calculated as in (Equation 2).

[0012]

[Equation 2]

A certain number of n's are taken out in descending order of the component I (n), and the combination of the n's is detected as a chord.

The operation generator 106 includes a peak detector 103.
The motion of the joint object is searched from the motion database 107 based on the information on the rising portion of the sound volume detected in step 3 and the chord detected by the chord detection unit 105, and the motion is synthesized. FIG.
Shows an example in which humans are regarded as joint objects. The movement of a limb with respect to a joint object as shown in FIG. 4 is stored in the motion database 107 as a position coordinate of the joint or a numerical sequence of the angle of the joint. Peak detector 103
The rising point of the volume detected in step 3 is the change point of the operation. The type of chord detected by the chord detection unit 105 is associated with the action data stored in the action database 107, and the action after the point of change in action is searched for by the type of chord. The joint position coordinate or the joint angle numerical value sequence that is held as the movement data is subjected to spline processing or the like so that the movement before the change changes smoothly. The motion output unit 108 visualizes and displays the motion data generated by the motion generation unit 106.

As a result of the above, based on the motion data stored in advance, it is possible to display a joint object that moves in accordance with music.

(Embodiment 2) Next, an embodiment of the motion generator 106 of the present invention will be described with reference to FIG. The chord classification unit 201 classifies the types of chords detected by the chord detection unit 105.

An example of the operation of the chord classifying section 201 for determining the type of chord will be described with reference to the flowchart of FIG. Here, three n having a large chord component I (n) are selected to be n1, n2, and n3 (n1 <n2 <n3). Check the values of n2-n1 and n3-n2, and
If it is 3, it can be determined that it is a major chord, and if it is 3 and 4, it can be determined that it is a monotone chord. For example, if a Domiso chord is detected, n1, n2, n3
Are 3, 7 and 10, respectively, and can be determined to be major chords. In the case of the Radmi chord, n1, n2, and n3 are 0, 3 and 7, respectively, and thus can be determined to be monotone chords. In addition, some discrimination can be made from the values of these two differences.

If the judgment cannot be made, a new n
N1, n2, n3, n2, n3, n
The chord is turned by replacing it with 1 + 12, and the judgment is performed. For example, if there are chords such as la, de, and fa, n
Since 1, n2, and n3 are 0, 3, and 8 respectively, it is not possible to discriminate between major and monotone, so n1, n2 and n3 are newly added.
Are turned to 3, 8, and 12, respectively, to make a determination. If it cannot be discriminated by this, it is turned again and discriminated, and if it cannot be discriminated for the third time, it is discriminated as "unknown" and the process is terminated.

In FIG. 2, the motion search unit 202 searches the motion database 107 for motion data corresponding to the chord classification result obtained by the chord classification unit 201. The behavioral synthesis unit 203 smoothes the behavioral data retrieved by the behavioral retrieval unit 202 and the behavioral data up to that point by spline processing or the like, and synthesizes them into a series of behaviors. As a result, the motion of the articulated object can
That is, it is possible to change the sound atmosphere.

(Embodiment 3) Next, another embodiment of the motion generator 106 of the present invention will be described with reference to FIG. The bar recognizing unit 301 determines an action change point based on the rising point of the volume detected by the peak detecting unit 103.

An operation example of the bar recognizing section 301 will be described with reference to FIG. A bar is defined as the minimum unit of time for executing one operation. The graph represents the time of the rising point of the volume detected by the peak detection unit and the volume of the volume at that time. The vertical axis of the graph represents volume and the horizontal axis represents time.
The input is music information with a clear rhythm, and the rising point of the volume is approximately time 0, 1
It is assumed that one of a, 2a ... Further, in the figure, "bar time" represents the time for executing one operation, "intra-bar elapsed time" represents the time elapsed in the bar, and "bar start time" represents the head time of the bar.

It is assumed that the volume first rises at time 0. The operation starts from here. The next rising time is a. Time a is defined as the reference time when the volume at time a is higher than time 0, and otherwise time 0 is defined as the reference time. Further, the time a is defined as a unit time. Since the unit time is determined at the time a, the bar time at that time is a
Becomes When the time a is used as the reference time, the elapsed time within the bar is 0 and the bar start time is a.

At time 2a, since there is no input, it is judged to be in the middle of the bar. Therefore, the bar time is 2
a, the elapsed time in the bar becomes a, and the bar start time remains a.

At time 3a, there is an input, but since it is a little small input, it is difficult to determine whether it represents the beginning of the bar or the middle of the bar. Therefore, it is determined that the time is “beginning or midway” of the bar. The bar time is 2a or 3a, and the elapsed time within the bar is 0 (bar has changed) or 2a (bar has not changed). Since the input is small even at time 4a, the state at time 3a is inherited.

When a large input is input at time 5a, it can be regarded as the beginning of the bar, and the input at times 3a and 4a can be judged to be in the middle of the bar. So the bar time is 4a,
The start time within a bar can be set to 5a.

Thereafter, if there is a large input every time the time passes 4a, it is considered that there is no contradiction, and the bar time 4a is maintained. The volume is judged based on the average value of the volume for several seconds before that. The bar recognition unit 301 outputs the bar start time. Further, the unit time a is dynamically determined based on the sound volume rising point for several bars before each time.

The behavioral synthesis unit 203 determines how fast the behavior should be synthesized based on the time between the movement change points determined by the measure recognition unit 301. For example, if the motion reproduction time T is 4a, it is necessary to interpolate and correct the motion data so that one motion is completed at the time T = 4a. The motion data stored in the motion data 107 has data for M frames per motion, and the display system is D per second.
If the motion data of the frame is required, the motion data of the mth frame may be used as shown in (Equation 3) in order to display the dth frame on the display system.

[0028]

(Equation 3)

Since m is not an integer in general, the operation data to be displayed is obtained by interpolating the operation data of the (m's integer part) frame and the (m's integer part) +1 frame.

As a result, the change point of the motion can be roughly set according to the speed of the music, and the motion can be generated according to the music.

(Embodiment 4) Next, an embodiment of the operation output unit 108 in the present invention will be described. Operation output unit 108
Is displayed by applying the motion data output by the motion generation unit 106 to a three-dimensional shape model of a joint object generated by computer graphics. As the joint object, for example, a three-dimensional shape model representing the shape of a human being is used as shown in FIG.
It will be applied to the dimensional shape model.

As a result, it is possible to visualize a joint object having a specific shape, such as a human being, moving along with music.

(Fifth Embodiment) Next, a fifth embodiment of the present invention will be described. As music information, a signal corresponding to music performance control information, such as a MIDI control signal, is input to the music signal input unit 101. The MIDI control signal has pitch information and volume information for each sound. Volume detector 102
Then, at the timing when the pronunciation information is present, the volume information as music can be extracted by calculating the sum of the volume information of the sounds present at that timing. Since it is easy for the pitch detecting unit 104 to extract the pitch information, the same processing as that when an electric signal of voice is input as music information can be performed thereafter.

As a result of the above, it is possible to reduce the load of processing for pitch detection and volume detection.

[0035]

As described above, according to the present invention, the auditory art of music is a visual art with a specificity that is the movement of an object with a specific shape, such as a human dancing to music. Can be converted to.

[Brief description of drawings]

FIG. 1 is a block diagram showing the configuration of an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a configuration example of a motion generation unit according to the present exemplary embodiment.

FIG. 3 is a block diagram showing another example of the configuration of the motion generation unit according to the present embodiment.

FIG. 4 is a diagram showing a human being as an example of a joint object according to the present embodiment.

FIG. 5 is a flowchart showing the procedure of the chord classification unit of this embodiment.

FIG. 6 is a diagram showing an operation of a bar recognition unit that determines a change point of the operation from time-series volume data.

[Explanation of symbols]

 101 Music Signal Input Unit 102 Volume Detection Unit 103 Peak Detection Unit 104 Pitch Detection Unit 105 Chord Detection Unit 106 Motion Generation Unit 107 Motion Database 108 Motion Output Unit 201 Chord Classification Unit 202 Motion Search Unit 203 Motion Synthesis Unit 301 Measure Recognition Unit

Claims (5)

[Claims] 1. A music signal input section for inputting music information, a sound volume detecting section for detecting a sound volume from the music information input by the music signal input section, and a sound volume based on sound volume data detected by the sound volume detecting section. A peak detection unit that detects a rising point, a pitch detection unit that detects a pitch distribution from the music information input by the music signal input unit, and a point of the rising position of the sound detected by the peak detection unit, A chord detection unit that detects chords from the pitch distribution detected by the pitch detection unit and a motion database that stores the motion of the joint object are held, and the timing of the rising point of the sound detected by the peak detection unit and the chord detection. A motion generation unit that extracts motion data from the motion database based on the chord detected by the motion generation unit and generates a motion, and outputs the motion generated by the motion generation unit. Music image converter, characterized in that it is composed of a work output. 2. The motion generation unit extracts a motion data from the motion database based on the chord classification unit that classifies the chord data detected by the chord detection unit and the result classified by the chord classification unit. A music image conversion apparatus comprising a search unit and a motion synthesis unit that connects a plurality of motion data retrieved by the motion search unit and synthesizes a motion. 3. The motion generation section has a bar recognition section for recognizing a bar unit from the timing sequence of the rising portion of the sound detected by the peak detection section, and at the bar unit time determined by the bar recognition section. 3. The music image conversion apparatus according to claim 2, wherein the motion search unit searches for the motion to be performed, and the motion synthesis unit synthesizes the motion from the motion data sequence. 4. The music image conversion apparatus according to claim 1, wherein the motion output unit outputs the image of the joint object performing the motion generated by the motion generation unit. 5. The music image conversion apparatus according to claim 1, wherein the music signal input section inputs a signal corresponding to music performance control information.