KR100344776B1 - Real-time music visualization method - Google Patents

Abstract

The present invention provides a visualization method that can visualize music with various patterns of lighting and animation in real time, and produce a fantastic lighting effect of active and various colors according to music, and extracts the characteristics of the input audio music signal. A first step of counting an output time of the current spectrum animation, an output time of the current wallpaper animation, and a special effect processing time of the previous screen; A second step of determining a shape of a next spectral animation to be output immediately after the current spectral animation as the output time of the current spectrum animation counted, and then determining the output spectral animation and gradually changing to the spectral animation; A third image that determines the shape of the next wallpaper animation to be output immediately after the current wallpaper animation, and then changes the output wallpaper animation to a wallpaper animation according to the output time of the counted current wallpaper animation. step; A fourth step of determining a special effect for changing the shape of the current spectrum animation and the current background animation according to the elapse of time of processing the special effect of the counted previous screen, and then changing the shape by processing the previous screen with the special effect; And a fifth step of outputting a current wallpaper animation for a predetermined time, outputting a current spectrum animation on the outputting wallpaper, and then skipping to the first step.

Description

Real-time visualization of music {REAL-TIME MUSIC VISUALIZATION METHOD}

The present invention relates to a real-time visualization method of music, and more particularly, to a visualization method that can visualize the music characteristics analyzed by analyzing the music signal with a variety of lighting and animation (animation) and show in real time.

In order to analyze and visualize the audio signal, there has conventionally been a bar graph showing the levels of the audio signal in each frequency band. This divides the audio signal into several frequency bands, extracts only those frequency band components, and then measures the level of the signal in that frequency band and displays it as a bar graph of height proportional to that level. These bar graphs continue to grow and shrink as the audio signal changes, resulting in a visual decorative effect when listening to music.

In addition, as a conventional method of visualizing an audio signal, there has been a method in which the x-axis is used as the time axis and the y-axis is used as the level of the audio signal to show how the audio signal changes with time.

However, such a conventional visualization method is more often used to analyze an audio signal than to produce a decorative effect due to its characteristics, and because of its simple pattern of change, it shows a decorative effect when listening to music. There were many cases of inadequacy. Above all, because the colorful effect is suitable in places such as karaoke or night clubs, the conventional visualization method as described above is not suitable for producing a decorative effect in accordance with the music in places such as night clubs.

And, if you are deaf or hard of hearing, it is difficult to hang out with ordinary people in concerts, karaoke, and night clubs. You can enjoy the music with the general public. In order to meet these requirements, there has conventionally been a visual lighting effect by flashing or rotating a laser, a nightclub psych or commercial color light source.

However, the conventional lighting device implemented using such a method consumes enormous installation cost due to too much auxiliary equipment such as a lamp, and requires a manual operation, which causes inconvenience in use. Since the number of colors or the number of colors that can be produced is extremely limited, there is a problem that is insufficient to produce a decorative effect to meet the needs of users.

Accordingly, an object of the present invention is to provide a visualization method that can visualize music with various patterns of lighting and animation in real time, and produce a fantastic lighting effect of active and various colors according to the music.

In addition, the present invention is another object to provide a visualization method that can significantly increase the advertising effect by visualizing the indoor music in real time, such as a colorful animation, such as outdoor advertising signs.

1 is a block diagram of a device for real time visualization of music to which the present invention is applied.

FIG. 2 is a diagram illustrating a music analyzer and a signal processor of FIG. 1. FIG.

3 is a flow diagram of an embodiment of a real-time visualization method of music according to the present invention.

FIG. 4 is a flow diagram of an embodiment of a process for applying dynamic spectrum to three-dimensional spectral animation in FIG.

FIG. 5 is a flowchart illustrating a process of outputting a fractional spectral animation of FIG. 3. FIG.

FIG. 6 is a flowchart illustrating a process of outputting a 3D fractional spectral animation in FIG. 3. FIG.

FIG. 7 is a flowchart illustrating a process of outputting a spectrum animation of a road shape running in FIG. 3; FIG.

8 is a flowchart illustrating an example of a process of outputting a spectral animation of a piano shape in FIG. 3.

FIG. 9 is a flowchart illustrating a process of outputting a psychedelic spectral animation of FIG. 3. FIG.

FIG. 10 is a flowchart illustrating a process of adding a special effect to the spectral animation of FIG. 3. FIG.

11A and 11B are simulation diagrams by the visualization method in FIG. 3.

12A and 12B are simulation diagrams by the fraction shape visualization method in FIG. 5.

13 is a simulation diagram by the three-dimensional fraction shape visualization method in FIG.

14 is a simulation diagram by the piano shape visualization method in FIG.

15A and 15B are simulation diagrams by the special effect processing method in FIG.

Explanation of symbols on the main parts of the drawings

110: music analyzer 120: signal processor

130, 140: Projection unit 150: Animation display unit

In order to achieve the above object, the present invention provides a method for visualizing a music signal with various lighting and animations, and after extracting characteristics of an input audio music signal, the output time SC of the current spectrum animation, Counting the output time BC of the background animation and the special effect processing time DC of the previous screen; Determining the shape of the next spectral animation to be output immediately after the current spectral animation according to the elapsed output time SC of the counted current spectral animation, and then gradually determining the output spectral animation to the next spectral animation. Two steps; After the output time BC of the counted current wallpaper animation elapses, a shape of a next wallpaper animation to be output immediately after the current wallpaper animation is determined, and the output wallpaper animation is determined as the next wallpaper animation. A third step of changing in an instant; After the special effect processing time (DC) of the counted previous screen is elapsed, a special effect for changing the shape of the current spectrum animation and the current wallpaper animation is determined, and the previous screen is processed to change the shape by special effects. Fourth step; And a fifth step of outputting the current background animation for a predetermined time, outputting the current spectrum animation on the outputting background wallpaper, and then skipping to the first step.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the present invention.

1 is a block diagram of a device for real-time visualization of music to which the present invention is applied.

Referring to FIG. 1, the visualization apparatus to which the present invention is applied includes a music analyzer 110 for extracting characteristics of a music signal by analyzing an input music signal in real time, and music extracted by the music analyzer 110. The signal processor 120, the projection units 130 and 140 for projecting the illumination signal reproduced by the signal processor 120, and the signal processor 120 to reproduce the characteristic as an animation signal and an illumination signal. An animation display unit 150 for displaying a reproduced animation signal as an image is provided.

In addition, the visualization apparatus to which the present invention is applied further includes a storage unit 170 for storing a music file, which is output to the music analyzer 110 according to a user's command.

The projectors 130 and 140 may be implemented as beam projectors or cathode ray tubes (CRTs).

In addition, the animation display unit 150 is a display element used for outdoor commercial billboards and the like. The animation display unit 150 expresses an animation of a shape suitable for music generated indoors in real time. There is a pattern of squirting or popping firecrackers. However, the animation display unit 150 is not limited to only commercial billboards.

Referring to the operation of the visualization device to which the present invention having the structure as described above is applied in detail as follows.

When the music analyzer 110 analyzes the characteristics of the music signal input through the microphone 160 or the music signal transmitted from the storage unit 170 in real time and delivers the extracted music characteristics to the signal processor 120, the signal The processor 120 searches whether the lighting signal and the animation signal suitable for the transmitted music characteristics are stored in the internal memory, and if so, the processor 120 and the animation expression unit respectively store the stored lighting signal and the animation signal. Output to 150. In this case, the signal processor 120 converts the stored illumination signal into a signal suitable for projecting and outputs the signal, and also converts the stored animation signal into a signal suitable for displaying and outputs the converted animation signal.

If the lighting signal and animation signal suitable for the music characteristics extracted by the music analyzer 110 are not stored in the internal memory of the signal processor 120, the signal processor 120 may adjust the music characteristics according to a preset program. After processing, the lighting signal and animation signal suitable for the music characteristics are reproduced, and then the reproduced lighting signal and animation signal are converted into a signal form suitable for projection and display as described above, and output.

As such, the illumination signal output from the signal processor 120 is projected by the projection unit 130 to the screen 170, the monitor, or the like, and the illumination signal transmitted to the projection unit 140 is projected to the road or the like. do.

The animation display unit 150 expresses an animation signal transmitted from the signal processing unit 120. For example, the animation display unit 150 may express an animation such as rain or snow.

As described above, the lighting and animation is a visual analysis of the music input through the microphone 160 in real time.

FIG. 2 is a block diagram of the music analyzer and the signal processor of FIG. 1, which includes an A / D changer 211, memories 212 to 218, a bus arbiter 219, a playback controller 220, and a video processor 221. And an output processor 222.

Here, the memory 218 is a device that stores lighting signals and animation signals suitable for specific musical characteristics.

The bus arbiter 219 is controlled by the reproduction controller 220 and performs an arbitration role so that data inputted / outputted to the memories 212 to 217 to which the bus is shared does not collide, that is, data is stored in each memory. Conflicts occur when input / output is executed at the same time, so data is arbitrated to input and output to each memory at different times.

The reproduction controller 220 reproduces the signal in the signal processing unit 120 for reproducing an illumination signal and an animation signal matching the function and music characteristics of the music characteristic analyzing element in the music analyzing unit 110 for analyzing the characteristics of the input music signal. Perform the functions of the control element in parallel.

The operation of the music analyzer and the signal processor of FIG. 1 having the structure as described above will be described in detail as follows.

When the A / D converter 211 converts the analog music signal input through the microphone 160 or the analog music signal transmitted from the storage unit 170 into a digital signal and outputs the digital signal to the memories 212 and 213, the memory ( 212 and 213 temporarily store the transmitted digital audio signal and deliver the digital audio signal to the reproduction controller 220.

Subsequently, the playback controller 220 analyzes and extracts the characteristics of the audio signal transmitted through the memories 212 and 213, and then checks whether the lighting signal and the animation signal suitable for the extracted music characteristics are stored in the memory 218. If stored, the stored lighting signal and animation signal are read and stored in the memories 214 and 215.

If the lighting signal and the animation signal suitable for the extracted music characteristic are not stored in the memory 218, the playback controller 220 calculates and processes the extracted music characteristic according to a predetermined program and then the lighting signal and the animation signal according to the characteristic. Is stored and stored in the memories 214 and 215.

The video processor 221 reads the video light signal and the animation signal stored in the memories 214 and 215 to enlarge, reduce and rotate the light signal and the animation signal to diversify the shape. . In this way, the diversified illumination and animation signals are temporarily stored in the memories 216 and 217 and then passed to the output processor 222.

Finally, the output processor 222 converts and outputs an illumination signal transmitted from the memories 216 and 217 into a signal form suitable for projecting onto a monitoring element such as the screen 170, and the output processor 222 The animation signals transmitted from the memories 216 and 217 are converted into a signal form that can be displayed on the animation display unit 150 such as an advertisement board and output.

3 is a flowchart illustrating a method of real-time visualization of music according to the present invention, in which an audio music signal stored in a memory is read (301), and the characteristics of the read audio music signal are extracted (302). Here, the characteristics of the audio signal to be extracted include waveform, spectrum, loudness (magnitude of audio signal), fundamental frequency (pitch and pitch), and beat (rate of beat). Here, the memory stores the audio music signal input from the outside or temporarily stores the audio music signal input from another memory storing the music file. At this time, the music file stored in the other memory is output in accordance with the user's instructions.

After counting the output time SC of the current spectrum animation, the output time BC of the current wallpaper animation, and the special effect processing time DC of the previous screen (303), the output time SC of the counted current spectrum animation is the reference time. It is determined whether Ts (about 10 seconds) has elapsed (304), and if so, the output time SC of the spectral animation is initialized (305).

Thus, when the initialization state, after determining the shape of the next spectral animation to be output immediately following the current spectral animation, after declaring that the output spectral animation has started to change gradually to the spectral animation (306), the output It is determined whether the process of changing the shape of the spectral animation (307), and if it is a changing process, the horizontal (x-axis) coordinates of the intermediate shape between the current spectrum animation and the determined next spectral animation that appear in the process of changing the output spectrum animation and A vertical axis (y-axis) coordinate is calculated (308).

Once the coordinate axes have been calculated, the number of rotations of the determined spectral animation in the three-dimensional space is calculated (309), then it is determined whether the change in the output spectral animation is complete (310), and if the change is complete, the output spectrum It is declared that the change of animation is finished (311).

As such, after the end is declared, it is determined whether the output time BC of the current wallpaper animation counted in the counting process 303 has passed the reference time Tb (about 13 seconds) (312). The output time BC is initialized (313).

After the output time BC is initialized, the shape of the next wallpaper animation to be output immediately after the current wallpaper animation is determined, and then the output wallpaper animation is completely changed to the wallpaper animation at once (314).

Subsequently, it is determined whether the special effect processing time DC of the previous screen counted in the counting process 303 has passed the reference time Td (about 8 seconds) (315). In operation 317, a special effect for changing the shapes of the current spectrum animation and the current background animation is determined.

In addition, during the special processing time DC of the previous screen counted in the counting process 303, the shape is changed by applying a special effect to the previous screen (318), and during the output time BC of the counted current wallpaper animation, The current wallpaper animation is output (319). When the background image is output, during the output time SC of the current spectrum animation counted in the counting process 303, after outputting the current spectrum animation on the current background animation being output (320), a music signal is output. Proceed to the reading process 301.

If it is determined in step 304 that the elapsed time of the output time SC is determined, if the output time SC of the counted current spectrum animation does not pass the reference time Ts, the process proceeds to the change determination process 307 of the output spectrum animation. At this time, if the change in the shape of the output spectrum animation is completed as a result of the determination in the determination process 307, the process proceeds to the process 312 to determine the elapsed time of the output time BC. Alternatively, as a result of the judgment in step 310 of determining whether the change of the output spectral animation has ended while the shape of the output spectral animation is changed, even when the change of the output spectral animation is not finished, Proceeding to the process 312 of determining the elapsed time.

As a result of the determination in the determination process 312, if the output time BC of the counted current background animation does not pass the reference time Tb, the process proceeds to the process 315 of determining the elapsed time of the special effect processing time DC. If the result of the determination in the determination process 315 is that the special effect processing time DC of the counted previous screen does not pass the reference time Td, the process proceeds to the special effect processing process 318 of the previous screen.

In the process of extracting the characteristic of the audio signal 302, a method of obtaining the extracted characteristic will be described below.

First, if the music signal is s [n], the window function is w [n], the window length is N, and the window moving distance is M, the music waveform of the kth frame is obtained as shown in Equation (1).

Second, the music spectrum of the kth frame is obtained using Equation 2 below.

Here, F {} represents a Fourier transform.

Third, the loudness L of the audio signal of the k-th frame is obtained by the following equation (3).

Here, A [jΩ] is the frequency characteristic of the human auditory system.

Fourth, the extraction of the fundamental frequency F ₀ includes an autocorrelation method and a cepstrum analysis method. The extraction of the fundamental frequency F ₀ using the latter concept is calculated as follows.

S _k [ jΩ ]-> Power Spectrum-> log ()-> Inverse Fourier Transform->Cepstrum-> F ₀

Finally, to automatically extract the speed of music, beat B is obtained based on the following assumptions.

The first hypothesis assumes that the beat of the music is determined by the sound of a percussion instrument (eg drum).

The second hypothesis assumes that the frequency band of the percussion sound is more concentrated in the high frequency region than the general music sound, and exhibits properties similar to white noise.

The third assumption is that the music section including the beat of the percussion instrument is non-stationary compared to the other sections, and impulse signal components are present.

The first to third assumptions are general phenomena seen in most music, and do not apply to music in which no percussion instrument or special percussion instrument is used.

The real-time bit automatic extraction algorithm applying these assumptions is performed according to the following steps.

In the first step, we define a variable and initialize it as follows:

M: window size

fl: percussion low frequency cut-off frequency

fh: percussion high frequency cutoff frequency

E [n]: energy of the nth frame

P [n]: Peak value of the nth frame

FrameTime: One frame time (= window size / sample rate)

E _average [n]: Average energy of previous K frames

P _average [n]: Average peak value of previous K frames

BitTime: Percussion beat interval (global value)

LocalBitTime: Percussion beat interval (local value)

In the second step, the nth frame of the music signal is obtained as follows.

sn [0], sn [1], ........, sn [M-1]

In the third step, E [n], Eaverage [n], P [n] and Paverage [n] are calculated as follows.

P [n] = max (sn [0], sn [1], ........, sn [M-1])

Here, S [f] is a Fourier transform of s [m].

In the fourth step, the presence or absence of bits is determined as follows, and the local bit interval is calculated.

if E [n]> αE _average [n] and P [n]> βP _average [n]

LocalBitTime = BitCounter * FrameTime

BitCounter = 0

BitTime = BitTime + gain (LocalBitTime-BitTime)

else

BitCounter = BitCounter + 1

Where α and β> 1

The fifth step substitutes n + 1 for n and goes to the second step.

FIG. 11A simulates a logo by the visualization method of FIG. 3 as described above.

FIG. 11A simulates fireworks by the visualization method of FIG. 3 as described above.

FIG. 4 is a flowchart illustrating a process of applying dynamic spectrum to 3D spectrum animation in FIG. 3. Referring to this, a method of applying dynamic spectrum to 3D spectrum animation according to the present invention will be described in detail as follows. .

First, a T th frame among T frames (where T is a natural number of two or more than t) is input (401), and a dynamic spectrum of an i frequency band is applied to the 3D spectrum animation in the input frame (402). . Here, the dynamic spectrum is a spectrum consisting of N frequency bands (where N is a natural number of 2 or more, indicating a frequency band higher than i). That is, the shape of the three-dimensional spectral animation is changed by applying the spectrum for each frequency band to the three-dimensional spectral animation.

After calculating the x-axis coordinate (Sx) and the y-axis coordinate (Sy) of the three-dimensional spectral animation on which the dynamic spectrum is applied (403), the calculated x-axis coordinates and the y-axis coordinates are connected (404).

At this time, the x-axis coordinate (Sx) and y-axis coordinate (Sy) is calculated as shown in Equation (4) and (Equation 5), respectively.

Here, SP (t, i) is the energy of the spectral animation obtained by changing the shape by applying the spectrum of the i-frequency band to the three-dimensional spectral animation in the t-th frame.

If all coordinates are connected, it is determined whether the i frequency band of the dynamic spectrum applied to the 3D spectrum animation is lower than the N frequency band (405), and if so, the process proceeds to the coordinate calculation process (403).

If the i frequency band is not lower than the N frequency band, it is determined whether the input t th frame is the same frame as the last T th frame (406), and if not, the process proceeds to the spectrum coating process (402). At this time, if all T frames are input and it is determined that the dynamic spectrum is completely coated on each frame, the spectrum coating process is terminated.

FIG. 5 is a flowchart illustrating a process of outputting a fractional spectral animation in FIG. 3, wherein the method changes the positions of several fractions over time and divides the entire spectrum into several frequency bands. This is to express the water pressure of the corresponding fraction according to the star energy.

First, after counting the output time WC of the currently displayed fractional spectral animation (501), it is determined whether the output time WC of the counted current fractional spectral animation has passed the reference time Tw (502). The output time WC of the fractional spectrum animation is initialized (503).

In this way, when the initialization state, the fraction of the next spectral animation to be output immediately after the fractional spectral animation that is currently output is determined, and then the output spectral animation is determined and then gradually changed to the fraction of the spectral animation. After declaring (504), it is determined whether or not the process of changing the fraction shape of the output spectrum animation (505), and if it is the process of changing, calculating the position of the fraction array of the next spectrum animation to be output immediately following the output of the current spectrum animation. (506).

If the fractional array position has been calculated, it is determined whether the change in the fractional array position of the output spectrum animation is completely made (507), and if the change is made completely, it is declared that the change of the fractional array position of the output spectrum animation has ended (508).

Thus, after the end is declared, after dividing the frequency bands of the fractional spectrum in the current frame (509), i (where i is a natural number of two or more), i-th (where i is The average energy of each frequency band of the fractional spectrum is calculated (510). After calculating the average energy for each frequency band, the i-th fractional spectrum animation is output (511).

At this time, it is determined whether the output i-th spectral animation of the spectral animation is a P-th fraction spectral animation (512), or otherwise, the process proceeds to the average energy calculation process (510). If the i-th fractional spectral animation output is the P-th fractional spectral animation, the output of the fractional spectral animation is terminated.

On the other hand, in the process 502 of determining the elapsed time of the output time WC, if the output time WC of the counted current spectrum animation does not pass the reference time Tw, the process proceeds to the change determination process 505 of the output spectrum animation. At this time, if the change of the shape of the output spectrum animation is completed as a result of the determination in the determination process 505, the process proceeds to the frequency band classification process 509. Alternatively, even when the change of the output spectrum animation is not finished, in the state in which the shape of the output spectrum animation is changed, the determination of whether the change of the output spectrum animation is finished is performed. Proceed to 509.

In addition, torch-shaped spectral animation can be implemented through the same process as described above.

FIG. 12A simulates a fraction by the fraction shape visualization method of FIG. 5 as described above.

FIG. 12B simulates a torch by the visualization method of FIG. 5 as described above.

FIG. 6 is a flowchart illustrating a process of outputting a 3D fractional spectral animation in FIG. 3.

Referring to FIG. 6, the characteristics of the analyzed audio music signal (that is, the loudness of music, the fundamental frequency of music, and the time signature of the music) are input (601), and the three-dimensional fraction is determined according to the loudness and fundamental frequency of the input music. After determining the number and color of the falling water droplets (602), the start of searching for the presence of water droplets on the screen is declared (603).

Check if the i-th droplet (where i is a natural number) is on the screen (604), and if it is not on the screen, generate the i-th droplet and eject the droplet generated according to the loudness and beat of the input music. After determining the speed and the angle (605), it is determined whether the i-th droplet and the number of droplets determined in the determination process (602) are the same (606).

Here, if it is not the same, increase the number of droplets on the screen by a certain number (about 1) (607), and then determine whether the number of increased droplets is less than the maximum number of droplets that can be output on the screen (608). Proceed to the droplet check process 604.

If the water droplets are present on the screen in the water droplet checking process 604, the process proceeds to the process of increasing water droplets 607.

If the number of drops determined by the i-th drop in the drop number determination process 606 is the same or the number of drops increased in the determination step 608 of the maximum drop number is not less than the maximum number of drops that can be output on the screen, the determination process (602). In step 609, the search for whether the water droplet determined in the present invention exists on the screen is declared.

It is checked whether the i-th droplet determined in the determination process 602 is on the screen (610), and if it is on the screen, after calculating coordinates of the i-th droplet existing on the screen (611), the i-th droplet is displayed on the screen. It is determined whether the out of step (612). At this time, if it is determined that the screen is out of the i-drop, it is removed (613).

Subsequently, the number of output droplets is increased by a predetermined number (about 1) (614), and then it is determined whether the number of increased output droplets is less than the maximum number of droplets that can be output on the screen (615). Prints to the screen. Here, if less, the flow proceeds to the droplet check process (610).

On the other hand, when the i-th droplet is not on the screen in the droplet checking process 610 or the droplet is off the screen in the droplet determination process 612 on the screen, the flow proceeds to the droplet increasing process 614.

FIG. 13 simulates a three-dimensional fraction by the three-dimensional fraction shape visualization method of FIG. 6 as described above.

FIG. 7 is a flowchart illustrating a process of outputting a spectral animation of a driving road shape in FIG. 3 to provide a visual effect such as driving a car by causing a yellow center line to come out of the screen.

As illustrated in FIG. 7, the energy of the music signal calculated by the signal processor 120 of FIG. 1 is input (701), and the output screen is enlarged in proportion to the energy of the input music signal (702). Next, the output time RC of the currently output road-shaped spectral animation is counted (703).

In this way, it is determined whether the output time RC of the counted current road-shaped spectrum animation has passed the reference time Tr (704). If so, after outputting the yellow center line-shaped spectrum animation at the center of the screen currently being output (705). In step 706, it is determined whether the counted output time RC has passed the maximum reference time Trmax. At this time, if it has not elapsed, the output time RC of the current road-shaped spectral animation is initialized (707), and the output of the road-shaped spectral animation is terminated. If the output time RC has not passed the maximum reference time Trmax, the output of the road-shaped spectral animation is similarly terminated.

On the other hand, in the time determination process 704, if the output time RC has not passed the reference time Tr, the process proceeds to the time determination process 706.

In most cases, a musical signal is composed of harmonics instead of a single note. For example, many of the basic notes are mixed, such as "Domisol", "Dopa", "Shiresol", etc. . Extracting all of these pitches can be used for animation visualization to enhance the visual impact. That is, an animation in which piano keys bounce in accordance with music may be made through a process as shown in FIG. 8.

Next, a process of outputting the spectral animation of the piano shape in FIG. 3 will be described in detail with reference to FIG. 8.

First, the audio music signal stored in the memory is read (801), and then windowing is performed by multiplying the audio music signal by a window function (802), and then the fast Fourier transform is performed on the windowed audio signal. Take a Fourier Transform (803).

The power of the fast Fourier transformed spectrum is obtained (804), the natural log is taken to obtain the logarithm of the spectral magnitude (805), and the inverse Fourier transform is applied to the logarithm to the music. A sepstrum having periodic information about the signal is calculated (806).

In this way, the obtained periodic information of the septum is divided into bands, and the energy for each piano key representing each pitch of "Doremipasolassi" is calculated (807), and then a spectral animation like that of the piano keyboard is output. (808).

At this time, the energy E (N) for each piano key in the calculation process 807 is calculated by calculating the following equation (6). Where N is a piano key number.

Here, D _low (N) and D _high (N) are calculated by calculating the following Equations (7) and (8), respectively.

Where S is the sampling rate of the music signal, Q _l (N) is the minimum quaternity of the N piano's keyboard calculated by the following equation (9), and Q _h (N) Is the maximum capacity of the N-piano keyboard calculated by Equation (10).

Where F _l (N) is the minimum frequency of the N piano keys, calculated by the following equation (11), and F _h (N) is the maximum frequency of the N piano keys, calculated by the following equation (12): to be.

FIG. 14 simulates a piano by the piano shape visualization method of FIG. 8 as described above.

FIG. 9 is a flowchart illustrating a process of outputting a psychedelic spectral animation in FIG. 3, and a method of outputting a psychedelic spectral animation will be described in detail with reference to FIG. 3.

First, after counting the output time PC of the currently output psychedelic spectral animation (901), the output time PC of the current psychedelic spectral animation counted is determined by the minimum reference time Tmin (Tmin is the ON time), and if not, a black screen is output (903).

After the black screen is output, the output time PC determines whether the maximum reference time Tmax has elapsed (Tmax is an OFF time for the psychedelic spectral animation output) (904). After initializing the output time PC of the spectral animation (905), the output of the psychedelic spectral animation is terminated. Here, if the output time PC has passed the maximum reference time Tmax, the output of the psychedelic spectral animation is immediately terminated.

If the output time PC is determined to have passed the minimum reference time Tmin in the time lapse determination process 902, after outputting a white screen 906, the process proceeds to the time determination process 904.

As described above, the psychedelic effect used in the output nightclub, etc., when the present invention is applied to the lighting, it is possible to obtain a psychedelic lighting effect by projecting white and black beams according to the beat of the music.

In addition, the present invention implements spectral animation colorfully and abstractly by applying various special effects. For example, when the enlargement ratio is enlarged in proportion to the size of the music in the screen magnification mode, the full spectrum animation image is more out of music. Gives a special effect that comes out quickly. Such a special effect processing method will be described in detail with reference to FIG. 10 as follows.

First, after determining a screen distortion mode for applying a special effect to the current screen being output (1001), it is determined whether the determined screen distortion mode is a screen blanking mode (1002), and if the screen blanking mode, the current screen is output as it is. End the annotation processing.

As a result of the determination in the determination process 1002, if it is not the screen blanking mode, it is determined whether the determined screen distortion mode is the black output mode (1003), and if it is the black output mode, a black image is output on the spectrum animation being output. After outputting (1004), the special effect processing ends.

As a result of the determination in the determination process 1003, if it is not the black output mode, it is determined whether the determined screen distortion mode is a specific color image output mode (1005). After outputting the specific color image calculated by the equation (1006), the special effect processing is terminated.

As a result of the determination in the determination process 1005, if it is not the specific color image output mode, it is determined whether the determined screen distortion mode is the screen shift mode (1007), and if the screen shift mode, the output spectrum animation is moved in one direction of the screen. After the next spectral animation is output on the screen (1008), the special effect processing ends.

As a result of the determination in the determination process 1007, if it is not the screen shift mode, it is determined whether the determined screen distortion mode is the screen reduction mode (1009), and if it is the screen reduction mode, the output spectrum animation is reduced and the next spectrum is displayed on the screen. After outputting the animation (1010), the special effect processing ends.

As a result of the determination in the determination process 1009, if it is not the screen reduction mode, it is determined whether the determined screen distortion mode is the screen magnification mode (1011). If the screen magnification mode, the output spectrum animation is enlarged and the next spectrum is displayed on the screen. After outputting the animation (1012), the special effect processing ends.

As a result of the determination in the determination process 1011, if it is not the screen magnification mode, it is determined whether the determined screen distortion mode is the screen folding mode (1013). After being folded to the vertical or horizontal axis) and outputting the next spectrum animation on the screen (1014), the special effect processing ends.

As a result of the determination in the determination process 1013, if it is not the screen folding mode, it is determined whether the determined screen distortion mode is the screen stretching mode (1015), and if it is the screen stretching mode, the spectral animation being output is displayed in one direction (for example, on the screen). The image is rolled out in a vertical or horizontal direction), and the next spectrum animation is output on the screen (1016), and the special effect processing ends.

If it is determined in the determination process 1015 that the screen is not in the screen spreading mode, the special effect processing ends.

15A and 15B are simulated by the special effect processing method as described above in FIG.

Although the technical spirit of the present invention has been described in detail according to the above-described preferred embodiment, it should be noted that the above-described embodiment is for the purpose of description and not of limitation. In addition, those skilled in the art will understand that various embodiments are possible within the scope of the technical idea of the present invention.

As described above, the present invention has the following effects by visualizing music in various patterns of lighting and animation in real time.

First, by visualizing and displaying various music in real time, it is possible to significantly increase the satisfaction of listening to music.

Second, by varying the lighting in the karaoke, night clubs, etc. to the music that is currently coming out, it can be more exciting to the people listening to music.

Third, by visualizing the music in various forms and showing it in real time, the hearing impaired can enjoy and enjoy the music like ordinary people.

Claims (17)

In a method of visualizing music signals with various lighting and animations, After extracting the characteristics of the input audio music signal, the first counting the output time (SC) of the current spectrum animation, the output time (BC) of the current wallpaper animation and the special effects processing time (DC) of the previous screen (counting) step; Determining the shape of the next spectral animation to be output immediately after the current spectral animation according to the elapsed output time SC of the counted current spectral animation, and then gradually determining the output spectral animation to the next spectral animation. Two steps; After the output time BC of the counted current wallpaper animation elapses, a shape of a next wallpaper animation to be output immediately after the current wallpaper animation is determined, and the output wallpaper animation is determined as the next wallpaper animation. A third step of changing in an instant; After the special effect processing time (DC) of the counted previous screen is elapsed, a special effect for changing the shape of the current spectrum animation and the current wallpaper animation is determined, and the previous screen is processed to change the shape by special effects. Fourth step; And A fifth step of outputting the current background animation for a predetermined time, outputting the current spectrum animation on the outputting background wallpaper, and then skipping to the first step Real time visualization method of music comprising. The method of claim 1, The first step is, 1-1 step of reading an audio music signal stored in the memory; Extracting characteristics of the read audio music signal; And Steps 1 to 3 counting an output time SC of the current spectrum animation, an output time BC of the current wallpaper animation, and a special effect processing time DC of the previous screen Real time visualization method of music comprising. The method of claim 1, The second step, A step 2-1 of determining whether the output time SC of the counted current spectrum animation has passed a predetermined reference time Ts; A second step of initializing an output time of a spectral animation if it has passed as a result of the determination; Determining the shape of the next spectral animation to be output immediately following the current spectral animation, and then declaring that the output spectral animation has begun to slowly change to the next spectral animation; Determining whether the shape of the output spectrum animation is changing; If it has not been determined as a result of the step 2-1, the step 2-5 to step 2-4; As a result of the determination in the step 2-4, when the shape of the output spectrum animation is changed, the horizontal axis (x-axis) of the intermediate shape between the current spectrum animation and the determined next spectrum animation appearing in the process of changing the output spectrum animation. A second step 6-6 calculating the coordinates and the vertical axis (y-axis) coordinates; If the coordinate axis is calculated, calculating the number of rotations of the determined next spectral animation in three-dimensional space; As a result of the determination in the second to fourth steps, when the shape change of the output spectrum animation is completed, the second to eighth steps to the third step; Steps 2-9 of determining whether the change of the output spectrum animation is completely made; As a result of the determination in the second to second steps, if the change is being made, the second to tenth steps to the third step; And Steps 2-11 for declaring that the change in the output spectrum animation is complete when the change is completely made as a result of the determination in steps 2-9. Real time visualization method of music comprising. The method of claim 1, The third step, Determining whether the output time BC of the counted current background image animation has passed a predetermined reference time Tb; A third step of initializing the output time of the background screen animation if it has passed as a result of the determination; In a state in which the output time is initialized, determining a shape of a next wallpaper animation to be output immediately after the current wallpaper animation, and then determining the output wallpaper animation to be the next wallpaper animation in an instant; And If the result of the determination has not passed, step 3-4 proceeding to the fourth step Real time visualization method of music comprising. The method of claim 1, The fourth step, A fourth step of determining whether the special effect processing time DC of the counted previous screen has passed a predetermined reference time Td; A step 4-2 of initializing the special effect processing time if it has passed as a result of the determination; Determining a special effect for changing the shapes of the current spectrum animation and the current background animation; A fourth step of changing a shape by applying a special effect to the previous screen during the special effect processing time (DC) of the counted previous screen; And If the result of the determination has not elapsed, step 4-5 proceeding to step 4-4. Real time visualization method of music comprising. The method of claim 1, The fifth step, A fifth step of outputting the current wallpaper animation during the output time BC of the counted current wallpaper animation; And A step 5-2 of outputting the current spectrum animation on the background screen being output during the output time SC of the counted current spectrum animation, and then proceeding to the first step; Real time visualization method of music comprising. The method of claim 1, The third step, Step 3-1 of receiving a t-th frame (where t is a natural number) among T frames (where T is a natural number of two or more); Step 3-2 of applying a dynamic spectrum of a frequency band i (where i is a natural number) to a three-dimensional spectral animation in the input frame; Step 3-3 of calculating an x-axis coordinate Sx and a y-axis coordinate Sy of the three-dimensional spectral animation with the dynamic spectrum on; Step 3-4 connecting the calculated x-axis coordinates and y-axis coordinates; Determining whether the i frequency band of the dynamic spectrum applied to the three-dimensional spectral animation is lower than the N frequency band (where N is a natural number of two or more) when all the coordinates are connected; If the i-frequency band is lower than the N-frequency band as a result of the determination, steps 3-6 to step 3-3; As a result of the determination, if the i frequency band is not lower than the N frequency band, determining whether the input t th frame is the same frame as the last T th frame; 3-8, if the t-th frame is the same frame as the last T-th frame as a result of the determination in steps 3-7, terminating a dynamic spectrum coating process; And As a result of the determination in step 3-7, if the t-th frame is not the same frame as the last T-th frame, step 3-9 proceeds to step 3-2. Real time visualization method of music comprising. The method of claim 1, The third step, A third step of counting an output time WC of a fractional spectral animation currently being output; After the output time WC of the counted current fractional spectral animation elapses, the fractional shape of the next spectral animation to be output immediately after the current fractional spectral animation is determined, and then the output spectral animation is determined. Step 3-2 gradually changing to a fractional shape of the spectral animation; Step 3-3 of classifying a frequency band of the fractional spectrum in the current frame; A third step of calculating an average energy of each frequency band of the i-th (where i is a natural number less than or equal to P) fractional spectrum among P (where P is a natural number of 2 or more); A 3-5th step of outputting an i-th fractional spectral animation proportional to the calculated magnitude of the average energy for each frequency band; Step 3-6 of determining whether the output i-th spectral animation is a spectral animation of the P-th fraction; If it is determined that the i-th fractional spectral animation is different from the P-th fractional spectral animation, steps 3-7, the method proceeds to step 3-4; And In operation 3-8, when the i-th fractional spectral animation is identical to the P-th fractional spectral animation, the output of the fractional spectral animation is terminated. Real time visualization method of music comprising. The method of claim 8, The third step 2, Determining whether the output time WC of the counted current fractional spectral animation has elapsed by a predetermined reference time Tw; A third to tenth step of initializing an output time of a spectral animation having a fractional shape when the output time WC has passed the predetermined reference time Tw as a result of the determination in the third to third steps; Determining the fraction shape of the next spectral animation to be output immediately after the currently displayed fractional spectral animation, and declaring that the output spectral animation is gradually changed to the fraction shape of the spectral animation after the determination; 3-12 steps of determining whether the process of changing the fraction shape of the output spectrum animation; A third to thirteenth step of calculating a fraction arrangement position of a next spectral animation to be output immediately after the output of the current spectral animation if the fraction of the output spectral animation is changed as a result of the determination of the third to third spectral animations; If the fractional arrangement position has been calculated, determining whether the change of the fractional arrangement position of the output spectrum animation is completely made; A step 3-15 of declaring that the change of the fractional arrangement position of the output spectrum animation is complete when the change of the fractional arrangement position is completely made as a result of the determination in the step 3-14; As a result of the determination in step 3-9, if the output time WC has not passed the predetermined reference time Tw, step 3-16 to step 3-12; As a result of the determination of the step 3-12, if the change of the fraction shape is completely made, the method proceeds to the step 3-3; And As a result of the determination in step 3-14, if the change of the fractional arrangement position is not completed, step 3-18 proceeds to step 3-3. Real time visualization method of music comprising. The method of claim 1, The third step, A third step of expanding the output screen in proportion to the calculated energy of the music signal; Step 3-2 of counting an output time RC of the currently output road-shaped spectral animation; Determining whether the output time RC of the counted current road-shaped spectral animation has passed a predetermined reference time Tr; 3-4, if the output time RC has not passed a predetermined reference time Tr, outputting a spectral animation in the shape of a yellow center line to a central portion of a screen currently being output; A third step of determining whether the counted output time RC has passed a predetermined maximum reference time Trmax; As a result of the determination in step 3-5, if the counted output time RC has passed a predetermined maximum reference time Trmax, the output time of the current road shape spectrum animation is initialized and then the road shape spectrum is initialized. 3-6 step of ending the output of the animation; A third to seventh step if the output time RC has passed a predetermined reference time Tr as a result of the determination in the third to third steps; If the counted output time RC has not passed a predetermined maximum reference time Trmax as a result of the determination in the third to third steps, the output of the road-shaped spectral animation is terminated. Real time visualization method of music comprising. The method of claim 1, The third step, Step 3-1 of performing windowing by multiplying an input audio music signal by a window function; Step 3-2 of performing a fast Fourier transform on the windowed audio signal; Obtaining a power of the fast Fourier transformed spectrum (step 3-3); Step 3-4 of taking a natural log at the obtained size to obtain a log value of a spectral size; A 3-5 step of calculating a Sepstrum having periodic information about a music signal by taking an inverse Fourier transform on the obtained log value; A step 3-6 of dividing the calculated period information of the septum by bands to calculate energy for each piano key representing each pitch of "Doremi Pasolassi"; Steps 3-7 for outputting a spectral animation of a piano keyboard shape compared to the calculated energy Real time visualization method of music comprising. The method of claim 1, The third step, Step 3-1 of counting an output time PC of the currently output psychedelic spectrum animation; A third step of determining whether an output time PC of the counted current psychic spectral animation has elapsed a predetermined minimum reference time Tmin; A third step of outputting a black screen if the output time PC has not passed the predetermined minimum reference time Tmin as a result of the determination; A third step of outputting a white screen when the output time PC has passed the predetermined minimum reference time Tmin as a result of the determination; Determining whether the output time PC of the counted current psychedelic spectral animation has passed a predetermined maximum reference time Tmax; A third to sixth step of initializing the output time of the spectral animation if the output time PC has not passed the predetermined maximum reference time Tmax as a result of the determination in the third step; And As a result of the determination in step 3-5, when the output time PC has passed the predetermined maximum reference time Tmax, steps 3-7 of ending the output of the psychedelic spectrum animation Real time visualization method of music comprising. The method of claim 1, The third step, 3-1, which receives the characteristics of the loudness, the fundamental frequency, and the beat of the analyzed audio music signal, and determines the number and color of water drops falling from the three-dimensional fountain according to the loudness and the fundamental frequency of the input music, respectively. step; Generating water droplets as many as the determined number of droplets and determining the ejection speed and angle of the droplets generated according to the loudness and beat of the input music; And A third to third step of outputting the determined water droplets on the screen according to the determined ejection speed and angle; Real time visualization method of music comprising. The method of claim 13, The third step 2, 3-4 step of declaring the start of the search for the presence of water droplets on the screen; a fifth step of checking whether an i th drop (i is natural water) is on the screen; If the i-th droplet is not on the screen as a result of the checking in step 3-5, the i-th droplet is generated, and the ejection speed and angle of the droplet generated according to the loudness and beat of the input music are respectively determined. Step 3-6; 3 to 7 determining whether the number of i-drops and the number of drops determined in step 3-1 are the same; As a result of the determination in step 3-7, if the number of i-drops and the number of drops determined in step 3-1 are not the same, increasing the number of drops on the screen by a predetermined number; Determining whether the increased number of droplets is less than the maximum number of droplets output on the screen; If the i-th water drop is on the screen as a result of the checking in step 3-5, step 3-10; If the i-th droplet is determined to be the same as the number of droplets determined in the step 3-1, the method proceeds to the step 3-3 if the determination in the step 3-7 is performed; As a result of the determination in step 3-9, if the increased number of drops is less than the maximum number of drops, steps 3-12, the method proceeds to step 3-5; And As a result of the determination in step 3-9, if the increased number of drops is less than the maximum number of drops, step 3-13, the process proceeds to step 3-3. Real time visualization method of music comprising. The method of claim 13, Step 3-3, 3-4 step of declaring the start of the search for the presence of the output droplet on the screen; 3-5 steps of checking whether the i-th water droplet determined in the 3-1 step is on the screen; A third to sixth step of calculating coordinates of the i-th water drop existing on the screen when the determined i-th water drop is on the screen as a result of the checking in the third step; Determining whether the i-th water droplet has left the screen as a result of the coordinate calculation; A third step of removing the i-th drop out of the screen if the i-th drop is out of the screen as a result of the determination in step 3-7; Steps 3-9 of increasing the number of output droplets by a predetermined number; Determining whether the number of output droplets increased in steps 3-9 is less than the maximum number of droplets output on the screen; As a result of the determination in step 3-10, when the number of output water droplets is less than the maximum number of droplets, steps 3-11 to step 3-6; If it is determined in step 3-5 that the determined i-th droplet is not on the screen, step 3-12, the method proceeds to step 3-9; As a result of the determination in step 3-7, if the i-th water droplet has not left the screen, proceeding to step 3-9; And As a result of the determination in step 3-10, if the number of output water droplets is not less than the maximum number of droplets, steps 3-14 for outputting the number of droplets increased in step 3-9 on the screen. Real time visualization method of music comprising. The method of claim 1, The fourth step, Determining a screen distortion mode for applying a special effect to the current screen being output; 4-2 determining whether the determined screen distortion mode is a screen blanking mode; A fourth step of outputting the current screen as it is if the determined screen distortion mode is the screen blanking mode as a result of the determination in step 4-2; A fourth step of determining whether the determined screen distortion mode is a black output mode if the determined screen distortion mode is not the screen blanking mode as a result of the determination in step 4-2; A fourth step of outputting a black image on the spectral animation being output if the determined screen distortion mode is the black output mode as a result of the determination in step 4-4; As a result of the determination in step 4-4, if the determined screen distortion mode is not the black output mode, determining whether the determined screen distortion mode is a specific color image output mode; As a result of the determination in step 4-6, if the determined screen distortion mode is the specific color image output mode, steps 4-7 for outputting a specific color image calculated by a predetermined equation on the spectrum animation being output; A fourth step of determining whether the determined screen distortion mode is the screen shift mode when the determined screen distortion mode is not the specific color image output mode as a result of the determination in step 4-6; As a result of the determination in step 4-8, if the determined screen distortion mode is the screen shift mode, 4-9-9 which moves the output spectrum animation in one direction of the screen and outputs the next spectrum animation on the screen. step; A fourth step of determining whether the determined screen distortion mode is a screen reduction mode when the determined screen distortion mode is not the screen moving mode as a result of the determination in the step 4-8; As a result of the determination in step 4-10, if the determined screen distortion mode is the screen reduction mode, steps 4-11 for reducing an output spectrum animation and outputting the next spectrum animation on the screen; As a result of the determination in step 4-10, if the determined screen distortion mode is not the screen reduction mode, determining whether the determined screen distortion mode is a screen magnification mode; As a result of the determination in step 4-12, if the determined screen distortion mode is the screen magnification mode, step 4-13 of enlarging an output spectrum animation and outputting a next spectrum animation on the screen; Determining whether the determined screen distortion mode is the screen folding mode if the determined screen distortion mode is not the screen magnification mode; As a result of the determination in the step 4-14, if the determined screen distortion mode is the screen folding mode, the output spectrum animation is folded so as to be rolled in one direction on the screen and the next spectrum animation is output on the screen. Step 15; As a result of the determination in step 4-14, if the determined screen distortion mode is not the screen folding mode, determining whether the determined screen distortion mode is a screen unfolding mode; As a result of the determination in the step 4-16, if the determined screen distortion mode is the screen spreading mode, the output spectrum animation is rolled out in one direction on the screen and the next 4-spectral animation is output on the screen. Step 17; And As a result of the determination in steps 4-16, if the determined screen distortion mode is not the screen spreading mode, steps 4-18 of terminating the special effect processing are performed. Real time visualization method of music comprising. The method of claim 1, The audio music signal is stored in a memory and is randomly output by a user.