KR100336269B1 - Apparatus for analyzing and visualizing music in real-time - Google Patents

Abstract

An apparatus for analyzing a music signal in real time and visualizing and displaying the characteristics thereof in various animations is provided. The apparatus includes an input unit for converting a music signal into digital audio data, a signal processor for extracting characteristics of a music signal from the input digital audio data, and generating a signal for various animation effects from the characteristics; And an output unit for visualizing and displaying a signal for the animation effect. The characteristics of the music signal include the loudness of the music signal, the fundamental frequency of the music signal, and the beat speed of the music signal. The animation effects include the music fountain effect, the space travel effect, the firecracker effect, the lightning and rain effects. do.

Description

Apparatus for analyzing and visualizing music in real-time}

The present invention relates to a real-time music analysis and visualization device, and more particularly, to a device for analyzing and displaying the characteristics of the music signal in real time by analyzing the music signal in real time.

As a method of analyzing and visualizing an audio signal and displaying it, there has conventionally been a bar graph showing the levels in each frequency band of an audio signal. This filter divides the audio signal into several frequency bands, extracts only the corresponding frequency band components, and then measures the level of the signal in the frequency band and shows it as a bar graph with a height proportional to the level.

Therefore, as the audio signal changes, these bar graphs continue to grow and shrink, which causes a visual decorative effect in listening to music.

In addition, as a method of visualizing an audio signal, there is a method in which the x-axis is used as the time axis and the y-axis is used as the level of the audio signal, thereby showing how the audio signal changes with time.

However, these conventional methods are more often used to analyze audio signals than to produce decorative effects due to their characteristics. In addition, because the change pattern is simple, it is often unsuitable to show a decorative effect in listening to music. In particular, because the gorgeous effects are appropriate in places such as karaoke and nightclubs, these methods are not suitable for creating decorative effects in this place.

On the other hand, the hearing impaired people can not hear the sound is often difficult to hang out with other people in places such as concerts or karaoke nightclubs. In this case, if the music can be shown visually and variously and colorfully so as not to lose interest, the deaf will be able to enjoy the music with the general public.

The present invention has been made in view of this point, and an object of the present invention is to provide a real-time music analysis and visualization device that can add interest to music by visualizing and displaying music in various patterns in real time.

In addition, it is an object of the present invention to provide a real-time music analysis and visualization device that can stimulate the entertainment of people listening to music by varying the lighting in the karaoke or nightclub in accordance with the current music.

1 is a schematic block diagram of a real-time music analysis and visualization apparatus of the present invention.

2 is a schematic block diagram showing various examples of an input unit of the apparatus of the present invention.

3 is a schematic block diagram showing a configuration of a signal processing unit of the apparatus of the present invention.

Figure 4 is a schematic block diagram showing a state in which the control unit of the present invention to extract the music characteristics from the digital audio data to create a visualization effect.

Figure 5 shows an algorithm for opening a sound card for input in a Windows 95 or later operating system.

Figure 6 shows an algorithm that implements a double buffering operation in software.

7 is an example of a screen by music fountain simulation.

8 is an example of a screen by space travel simulation.

9 is an example of a screen by the fireworks display simulation.

10 is an example of a screen by rain and lightning simulation.

The apparatus of the present invention for achieving the above object is to extract the characteristics of the music signal from the input unit for converting the music signal into digital audio data, the input digital audio data, and to generate a signal for various animation effects from the characteristics And a signal processing unit and an output unit for visualizing and displaying a signal for the animation effect from the signal processing unit.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1. Input section

The input unit 10 receives an audio signal and converts the audio signal into a digital data form that can be processed by the signal processing unit 20.

When the audio signal is output as an analog signal from an audio reproducing apparatus 11 such as a CD player or an audiotape player as shown in FIG. 2A, the input unit 10 outputs an analog output from the reproducing apparatus 11. And an A / D converter 12 for converting the signal into digital data form.

In addition, when the audio signal is a sound wave signal such as a human voice, not a form of an electric signal from the playback device, the sound wave signal is converted into an electric signal as shown in the microphone 13 as shown in FIG. The device converts the sonic signal into an analog electrical signal and then amplifies it to a level sufficient for use in the A / D converter 12 using the amplifier 14. The amplified signal is converted into a digital signal by the A / D converter 12 and output to the signal processor 20.

In addition, when the audio signal is output from the digital audio reproducing apparatus 15 for outputting a digital signal, the audio signal may be directly connected to the signal processing unit 20 in the digital audio reproducing apparatus 15 without the need for a separate A / D converter. have.

In addition, a sound card of a general personal computer (PC) may be used as the input unit. An example of a routine for opening a sound card for input in an operating system Windows 95 or later is shown in FIG. 5.

2. Signal processing unit

The signal processing unit 20 serves to control the output unit 30 to extract a characteristic of the audio signal from the digital audio data from the input unit 10 and to generate a visual effect based on the characteristic.

3 is a schematic block diagram showing an embodiment of the signal processing unit 20. Hereinafter, an operation of the signal processor 20 will be described in detail with reference to FIG. 3.

<Double buffering>

In Fig. 3, the digital audio data from the input portion enters one of the buffer 1 22 and the buffer 2 23 by the input side switch 21. While the digital audio data from the input portion enters one buffer, the digital audio data stored in the other buffer, that is, the digital audio data previously input from the input portion, is controlled through the output switch 24. Is output.

After the processing of the data stored in one buffer is completed, the control unit 25 controls both switches 21 and 24 to perform the processing of the data stored in the other buffer.

That is, while the input side switch 21 is connected to one buffer and data is being stored in the buffer, the output side switch 24 is connected to the other buffer and the control unit 25 and stored in the corresponding buffer in the control unit 25. Process the existing data. After the processing of the data, the switches 21 and 24 are inverted by the control signal of the control unit 25 so that the buffer in which the data is stored from the input unit and the buffer in which the data is output to the control unit are interchanged. By repeating such an operation, even if there is a change in the signal processing time in the control unit 25, the signal processing can be performed without missing data from the input unit 10.

This operation (double buffering) can be implemented in hardware, but can also be implemented in software. An example of software that implements double buffering in a typical PC is shown in FIG. 6.

<Audio signal characteristic extraction>

Next, a method of extracting the characteristics of the audio signal from the digital audio data in the control unit 25 will be described.

The characteristics of the audio signal include waveform, spectrum, loudness (the magnitude of the audio signal), fundamental frequency (pitch and pitch), and beat (beat rate). These characteristics are calculated | required by the following method.

(1) waveform

If the music signal is s [n], the window function is w [n], the window length is N, and the window moving distance is M, the music waveform of the kth frame is expressed by Equation 1.

(2) spectrum

The music spectrum of the kth frame is obtained by equation (2).

Where F {} represents a Fourier transform.

(3) loudness L (music loudness )

The loudness of the audio signal of the k-th frame is obtained by equation (3).

Here, A [jΩ] is the frequency characteristic of the human auditory system.

(4) fundamental frequency F ₀ (height of the music sound)

Extraction of F ₀ includes autocorrelation method and sepstrum analysis. Extraction of F ₀ using the septum analysis method is as follows.

S _k [ jΩ ]-> Power Spectrum-> log ()-> Inverse Fourier Transform->Cepstrum-> F ₀

(5) bit B (beat speed)

In order to automatically extract the speed of music, the following assumptions are made.

Assumption 1: The beat of a music is determined by the sound of a percussion instrument (drum, etc.).

Assumption 2: The frequency band of percussion sounds is more energy concentrated in the high frequency range than general music sounds, and has similar characteristics to white noise.

Assumption 3: The music section containing the beat of the percussion instrument is non-stationary compared to the other sections, and there is an impulsive signal component.

The above assumption is a common phenomenon seen in most music and does not apply to music with no percussion instruments or special percussion instruments.

The real-time bit automatic extraction algorithm using this assumption is as follows.

Step 1: Define and Initialize Variables

M: window size

fl: percussion low frequency cut-off frequency

fh: percussion high frequency cutoff frequency

E [n]: energy of the nth frame

P [n]: Peak value of the nth frame

FrameTime: One frame time (= window size / sample rate)

E _average [n]: Average energy of previous K frames

P _average [n]: Average peak value of previous K frames

BitTime: Percussion beat interval (global value)

LocalBitTime: Percussion beat interval (local value)

Step 2: Get the nth frame of the music signal

sn [0], sn [1], ........, sn [M-1]

Step 3: E [n], E _average [n], P [n], and P _average calculate [n]

Where S [f] is the Fourier transform of s [m].

P [n] = max (sn [0], sn [1], ........, sn [M-1])

Step 4: Determine the presence or absence of bits and calculate the local bit spacing

if E [n]> αE _average [n] and P [n]> βP _average [n]

LocalBitTime = BitCounter * FrameTime

BitCounter = 0

BitTime = BitTime + gain (LocalBitTime-BitTime)

else

BitCounter = BitCounter + 1

Where α and β> 1

Step 5: Substitute n + 1 for n and go to step 2.

<Visual effects>

The controller 25 generates various animation effects from the characteristic data of the extracted audio signal as shown in FIG.

Among the characteristics of the extracted audio signal, waveforms and spectra are conventionally used. As in the prior art, waveform shapes and levels at respective frequency bands may be expressed in bar graph form.

In the present invention, in addition to the waveform 110 and spectrum 120, using the loudness 130, the fundamental period 140, the beat 150, which is the music information extracted from the signal processor, the fraction 210, the space travel 220 By operating the animation, such as firecrackers 230, rain and lightning (240), the effect is as if the animation video is dancing according to the music.

Hereinafter, a description will be given of a music fountain simulation, space travel, fireworks, rain and lightning visualization algorithms.

(1) music fountain simulation

Music fountain simulation refers to the representation of music as images of fountains. An example of a screen by music fountain simulation is shown in FIG. 7.

The controller 25 represents the water pressure (height) of the fountain in E0, the color of the fountain in F0, and the left and right shakes of the fountain in B0.

That is, the height of the fountain is determined by the intensity of the sound, the color of the fountain is determined by the frequency of the sound, and the left and right shakes of the fountain are determined by the beat rate of the music. However, it is of course possible to take the determining factors of the height of the fountain, the color of the fountain, and the left and right shake of the fountain from characteristics different from the above example.

The algorithm for the simulation of fractions is as follows.

Step 1: Definition of Variables and Initialization of Fractions

cx: the width of the screen

cy: height of the screen

N: number of drops

Δt: fractional operation speed

G: gravitational acceleration

V0 [i]: Initial jet velocity of the i th drop

θ0 [i]: Initial ejection angle of the i th droplet

t [i]: Time elapsed after the i th drop

θavg: average ejection angle of water droplets

B: Bit Time of Music

L: Loudness of Music

F0: fundamental frequency of music

for each i (0 <i <N)

Set V0 [i] has uniform distribution in [0, Vmax]

Set θ0 [i] has gaussian distribution with zero mean

Set t [i] = 0

θavg = 0

Step 2: color the water droplets and calculate the average ejection angle

j = j + 0.01 / B

θavg = (π / 4) cos (2πj)

The larger the F0, the colder the color.

Step 3: calculate and draw each droplet position in the next frame

for each i (0 <i <N)

t [i] = t [i] + Δt

x = V0 [i] * cos (θ0 [i]) * t [i];

y = V0 [i] -G * t [i]) * t [i];

if (if you leave the screen)

Set V0 [i] has gaussian distribution with L mean

Set θ0 [i] has gaussian distribution with θavg mean

t [i] = 0

else

Draw each drop with the color specified in coordinates (x, y)

Step 4: Go to Step 1

(2) space travel simulation

Space travel simulation is like a space travel, where the stars appear close to me from afar and then disappear. An example of a screen by space travel simulation is shown in FIG. 8.

The controller 25 expresses the travel speed in E0 of the extracted music information, expresses the color of the star in F0, and expresses the brightness of the star in B0.

That is, the speed of the star approaching me slows down or speeds up depending on the volume of the sound, the color of the star changes according to the representative frequency of the sound, and the brightness of the star varies according to the beat rate of the music. However, it is of course possible to take the factors that determine the speed of the star, the color of the star, and the brightness of the star differently from the above example.

The algorithm for simulating space travel is as follows.

Step 1: Define the Variables and Initialize the Stars

MAX_STAR: maximum number of stars

typedef struct tagSTAR

short StarType; // type of star

BOOL IsStar; // presence

double distance; // distance from origin

double angle; // angle from origin

double radius; // radius of the star

short LifeTime; // elapsed time after creation (unit: frame)

BYTE R; // color of the star

BYTE G; // color of the star

BYTE B; // color of the star

STAR;

STAR * star; // star object

int starsize; // star size

double DIAG; // screen size

for (i = 0; i <MAX_STAR; i ++)

star [i] .distance has uniform distribution in [0, DIAG]

star [i] .angle has uniform distribution in [0, 2pi]

star [i] .radius has uniform distribution in [0, 3]

star [i]. LifeTime = 0;

star [i] .StarType has uniform distribution in [0, 3]

determine star [i] .R = F0

star [i] .G = determined by F0

star [i] .B = determined by F0

determined star [i] .StarType = B

Step 2: Type, Position, and Draw for Each Star

for (i = 0; i <starsize; i ++) // for each star

Calculate Star Coordinates

star [i] .LifeTime + = 1;

if (if star is within the screen)

Call the drawing routine after specifying the star type, color, and size

Calculate the position and size of the star in the next frame

else // if star is off screen

Go to step 2

Step 3: Repeat step 2 for the next frame.

(3) firecracker simulation

Firecracker simulation refers to expressing the image of a firecracker bursting with music. An example of the screen by the fireworks display simulation is shown in FIG.

The control unit 25 expresses the explosive power of the firecracker in E0, the color of the firecracker in F0, and the firing time interval of the firecracker in B0.

In other words, the range of firecrackers spreads according to the loudness of sound, and the firecracker's color changes according to the representative frequency of the sound, and the firing time interval of the firecracker varies according to the beat rate of music. For example, if the sound is loud, the firecrackers are loud. If the beat is fast, the firecrackers are often burst. However, it is of course possible to take the factors that determine the speed of the star, the color of the star, and the brightness of the star differently from the above example.

The algorithm for the simulation of the fireworks above is as follows.

Step 1: Define and Initialize Variables

Step 2: create firecrackers according to the music and determine the firecracker status

if (if current firecracker count is less than music size L)

for each i (0 <i <MAX_FIRE) // for each firecracker

if (i firecracker does not exist)

Create and initialize the i firecracker

Calculate explosion x, y coordinates

Sparklers color crystals

Step 3: Determine the presence and explosion of firecrackers and draw firecrackers

for (i = 0; i <MAX_FIRE; i ++) // For All Firecrackers

if (firecracker exists)

if (if not yet exploded)

Calculate the coordinates of the rocket

Draw a rocket.

else // if it exploded

if (first explosion)

Create debris

For each fragment

if (first explosion)

Set velocity, angle of debris

Shape the movement of the fragment

if (after some time has passed)

Reduce the number of debris

Step 4: Go to Step 2

(4) rain and lightning simulation

Rain and lightning simulation refers to expressing images of lightning and rain in time with music. An example of a screen by rain and lightning simulation is shown in FIG. 10.

The controller 25 expresses the amount of rain and the intensity of the lightning in E0, the color of the rain in F0, and the time interval of the lightning in B0.

That is, a large amount of rain and strong lightning strikes according to the volume of the sound, the color of the rain or wind blowing effect according to the representative frequency of the sound, and the lightning strikes according to the beat rate of music.

However, it is of course possible to take the determining factors of the amount of light and rain, color and intensity differently from the above example.

The algorithm for simulating lightning and rain is as follows.

Step 1: Define the Variables and Initialize Lightning and Rain

Step 2: Draw Lightning

Coordinate system setting

if (if the sound is above a certain level)

Against all lightning

if (i th lightning does not exist)

Lightning generation and initialization

Set each node of lightning

Against all lightning

If lightning exists

About every node of lightning

Coordinate setting

Determine the thickness of the lightning

Lightning drawing

Remove lightning when it's time to disappear

Step 3: draw non

rainnum = energy / 255. * MAX_RAIN

for (i = 0; i <rainnum; i ++) // Create Positive Ratio Proportional To Sound

if (if there is no i th rain)

Non-generated

for (i = 0; i <MAX_RAIN; i ++) // for all ratios

if (the i th rain)

Draw the line by calculating the following coordinates

if (off screen)

remove the i th rain

Step 4: Go to step 2 for the next frame.

3. Output section

The output unit 30 includes a display device such as a monitor, a projector, or the like, and a drive device for driving the display device. The visual effect control signal from the controller 20 is transmitted to the output unit 30 to control the driving device to be displayed as an image through a monitor or a projection device. Such an output device may use a conventional display device, and is not limited to a specific device.

On the other hand, when the present invention is applied to a lighting device in a karaoke room or a night club, the lighting may change the shape in which the fountain is spouted or the shape in which the firecrackers burst in accordance with music.

Although the present invention has been described in detail above, the present invention is not limited to the specific embodiments. Those skilled in the art will appreciate that many modifications and variations can be made without departing from the spirit of the invention.

According to the present invention, by visualizing and showing music in various patterns in real time, it is possible to add interest to music.

In addition, according to the present invention, since the music is visualized in various forms in real time, the visually impaired people can also enjoy the music together.

In addition, according to the present invention by varying the lighting in the karaoke or nightclubs in accordance with the music that is currently coming out there is an effect that can stimulate the excitement of people listening to music.

Claims (7)

delete An input unit for converting a music signal into digital audio data, A signal processing unit including two buffers for temporarily storing data from the input unit, a control unit reading data from the buffer to extract characteristics of a music signal, and generating signals for various animation effects from the characteristics; An output unit for visualizing a signal for the animation effect from the signal processor Equipped with The control unit receives one of the two buffers from the input unit and stores the data, the other outputs the previously stored data to the control unit, and all of the data stored in the other buffer is If the output to the control unit, the real-time music analysis and visualization device, characterized in that for controlling to change the role of the two buffers. The apparatus of claim 2, wherein the characteristics of the music signal include a loudness of the music signal, a fundamental frequency of the music signal, and a beat rate of the music signal. The method according to claim 2 or 3, As the animation effect, but expresses the image that the fountain is emitted from the characteristics of the music signal, Real-time music analysis and visualization device characterized by determining the height of the fountain in proportion to the intensity of the music, the color of the fountain according to the fundamental frequency of the music sound, and the left and right shake of the fountain according to the beat rate of the music . The method according to claim 2 or 3, As the animation effect, a star appears from a distance and disappears as if it is a space travel from a characteristic of a music signal. Real-time music analysis and visualization device, characterized in that the speed of the star in proportion to the intensity of the music, the color of the star according to the fundamental frequency of the music sound, and the brightness of the star in proportion to the beat rate of the music . The method according to claim 2 or 3, As the animation effect, the firecracker pops up from the characteristics of the music signal, Real-time music analysis and visualization characterized by setting the explosive power of the firecracker in proportion to the intensity of the music, the color of the firecracker in accordance with the basic frequency of the music sound, and setting the firing time interval of the firecracker in proportion to the beat rate of the music. Device. The method according to claim 2 or 3, As the animation effect, but expresses the image of lightning and rain from the characteristics of the music signal, Real-time music analysis characterized by determining the intensity and amount of rain in proportion to the intensity of the music sound, the color of the rain according to the representative frequency of the music sound, the lightning strikes according to the beat rate of the music and Visualization device.