KR101095968B1 - Method for Spectral Analysis of Polyphonic Audio by Using Variable Length Window - Google Patents

Abstract

The present invention relates to a frequency analysis method using an analysis window having a fluid length in an audio signal having several sounds.
To this end, the present invention provides a first step of resampling an input audio signal, a second step of converting an audio signal input on a time domain basis into a signal on a frequency domain basis, and a second step of each frame obtained through the second step. A third step of extracting a peak value of an amplitude and a frequency value of a position where the peak value appears in a spectrum, and a fourth step of resetting a range in which a melody pitch of each frame exists based on values extracted through the third step And a fifth step of obtaining the dynamic change information of the melody pitch by determining the autocorrelation coefficient between the frames in the result obtained by converting the signal to the frequency domain reference signal through the second step, and resetting through the fourth step. Information on the range in which the melody pitch of each frame exists, and dynamic change information of the melody pitch obtained through the fifth step. A sixth step of setting the length of the analysis window corresponding to each frame by using the second frame; a seventh step of removing the base signal and the low frequency signal from the audio signal using a high frequency pass filter; and a seventh step In order to increase the resolution of the short-time Fourier transformed audio signal through the eighth step and the eighth step by using an analysis window corresponding to each frame whose length is set in the sixth step, A ninth step of passing the multi-rate filter bank is included.

Description

Method for Spectral Analysis of Polyphonic Audio by Using Variable Length Window}

The present invention relates to a method of analyzing a frequency using an analysis window having a fluid length in an audio signal having several sounds.

Thanks to the development of computers and the digitization of information, many materials and documents are easily stored and managed. In the case of music, digitization and the development of equipment made it easier to compose and arrange music, which led to the production of new and diverse music.

Meanwhile, in order to conveniently and systematically manage a large number of newly created music, many companies and organizations store and manage information on song titles, singers, composers, eras, lyrics, and genres as text-based digital information. .

This digitized music management makes it easy for us to find and search for the music we want, which is the basis of the music search system that is now widely used on the Internet and mobile phones.

The music retrieval system widely used today is a text-based music information retrieval system for retrieving desired music by inputting information about music titles, lyrics, and composers in a text format.

However, thanks to the development of music technology and equipment, a search system using music contents such as melody, mood, chord, and rhythm of music, as well as simple textual information, is now attracting attention. It is called (Content-Based Music Information Retrieval System).

Melody information is one of the most important information used to classify and analyze music in the content-based music retrieval system. Although the technical definition of melody is not clearly defined so far, the general definition of melody is as follows.

Melody refers to the voice of a typical person's voice or an instrument that people think of when they listen to music mixed with the sounds and sounds of various instruments. Melody also has a harmonic structure and its energy magnitude tends to prevail compared to other sounds.

In order to extract reliable melody information, the next audio signal with multiple notes must be correctly analyzed and interpreted. In general, an audio signal may be divided into a method of analyzing an audio signal in a time domain and a method of analyzing a signal in a frequency domain according to a reference.

Here, in order to analyze the information of music contents based on the melody and other rhythms, chords, and moods of an audio signal having various sounds, an analysis method in the frequency domain using Fourier transform is widely used. Therefore, a more accurate and detailed method of analyzing frequency components is needed to analyze the next correct audio signal.

On the other hand, one of the obstacles for extracting the melody of the reliable result from the audio signal having a multi-tone is the dynamic variation of the melody pitch (Dynamic Variation of Melody). Since the melody of music is not always constant, a change occurs over time, and sometimes a sudden change in sound occurs. In other words, while playing a particular note, switching to another note occurs continuously

The dynamic change that is converted in this way is determined by the score of the music, so there is a problem that it is difficult to predict in advance. In addition, there is a problem in that it is difficult to find the melody pitch frequency since the fundamental frequency changes quickly in the section in which the melody is called and the melody starts and ends as described above.

The present invention has been made to solve the above problems, and an object thereof is to provide a more reliable and accurate frequency information analysis method for extracting correct melody and music content information from an audio signal having several sounds.

To this end, the present invention is to analyze the flexible length of the melody having the dynamic change as described above to obtain the melody pitch information of the result more reliable than the frequency analysis method using the conventional fixed length analysis window Provides a frequency analysis method using the window.

In order to achieve the above object, a frequency analysis method according to the present invention includes a first step of resampling an input audio signal, and a frequency domain of an input audio signal on a time domain basis. Domain); a third step of extracting a peak value of an amplitude and a frequency value of a position where the peak value appears from the spectrum of each frame obtained through the second step; A fourth step of resetting a range in which the melody pitch of each frame exists based on the values extracted through the step; and identifying autocorrelation coefficients between frames in the result of conversion into a signal based on a frequency domain through the second step A fifth step of obtaining dynamic change information of the melody pitch by the method and a melody pitch of each frame reset through the fourth step Is a sixth step of setting the length of the analysis window corresponding to each frame by using the information on the range and the dynamic change information of the melody pitch obtained through the fifth step, and using the high frequency pass filter (HPF) in the audio signal. The seventh step of removing the base signal and the low frequency signal, and the audio signal passed through the high frequency pass filter (HPF) through the seventh step using an analysis window corresponding to each frame whose length is set in the sixth step for a short time Passing the short-term Fourier transform (STFT) audio signal through a multi-rate filter bank to increase the resolution of the frequency domain through the eighth and eighth steps of Fourier transform (STFT) The ninth step is included.

The second step converts an audio signal input on a time domain basis into a signal on a frequency domain basis by applying a short time Fourier transform (STFT) using a fixed length analysis window.

The frequency value of the position where the peak value appears in the third step is extracted using the number of frequency bins, the number of bins of the discrete Fourier transform (DFT), and the resampling frequency. The frequency value of the position where the peak value of the third step appears can be obtained by Equation 1 below.

[Equation 1]

In the fourth step, a first process of extracting the top K peak values and the frequency value at which the peak values appear by arranging the values of the melody peaks in order of magnitude, and the peak value and the peak extracted through the first process A second process of finding all peak values existing in the melody range initially set among the frequency values at the position where the value appears and selecting them as potential melody candidates, and a melody based on the potential melody candidates selected through the second process. To reset the range, the frequency of each candidate in the potential melody candidate is regarded as the fundamental frequency of the melody, and corresponds to the harmonic component of the fundamental frequency among the frequency values at which the upper K peak values extracted through the first process appear. Peak value around the frequency, and the peak value is added to the fundamental frequency peak value. In the method, the third process of investigating the harmonic component and adding the peak values to all the peak values existing within the melody range initially set is multiplied by the smaller values as the fundamental frequency is lowered to the peak values added through the third process. As the fundamental frequency is higher, the fourth process of finding N candidates in the order of the higher values among the results obtained by multiplying the higher values, and the first to fourth processes are applied for each frame, and the melody pitch exists in the entire frame. Include the fifth step of finding out the range.

In the fifth step, the dynamic change information of the melody pitch is obtained by determining the autocorrelation coefficient between frames in the result of conversion to the signal based on the frequency domain through the second step, and the autocorrelation coefficient is Can be obtained by

[Equation 2]

In the sixth step, the length of the analysis window corresponding to each frame is set, and the length of the analysis window is lower as the melody pitch value is kept constant and the longer analysis window is used. Can be.

In addition, the frequency analysis method using the analysis window having a flexible length in the audio signal having a plurality of sounds according to the present invention can be used in the form of a computer-readable recording medium recording a computer program.

As described above, according to the frequency analysis method according to the present invention, the harmonic property of the melody and the energy component of the melody are most prominent to predict the range of the melody pitch, and the dynamic of the melody using the autocorrelation coefficient. It is effective to identify the area of change.

In addition, by combining the two results and performing an audio signal frequency analysis using a flexible length analysis window, it is possible to extract more reliable melody pitch information from an audio signal having several sounds.

1 is a flowchart of a frequency analysis method using an analysis window having a fluid length in an audio signal having various sounds according to the present invention.
FIG. 2 is a flowchart illustrating a process of passing a multi-rate filter bank of an audio signal having various sounds.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention; First, it should be noted that the same components or parts among the drawings denote the same reference numerals whenever possible. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted so as not to obscure the subject matter of the present invention.

The method to be proposed in the present invention is largely as follows. First, estimating the range of the melody pitch in the audio signal having a plurality of notes, identifying a change section of the melody, and finally the length analysis window that changes based on the information about the range of the melody pitch and the change section of the melody Using a short time Fourier transform (STFT) is performed. Look in more detail below.

1 is a flowchart of a frequency analysis method using an analysis window having a fluid length in an audio signal having various sounds according to the present invention. As shown in FIG. 1, the frequency analysis method according to the present invention includes each step including the first step S10 to the ninth step S90.

First, a first step S10 is a step of resampling the audio signal in order to convert the input audio signal into a format suitable for analysis.

In the present invention, a multi-rate filter bank is used to increase the resolution of the frequency domain. For this purpose, a sampling frequency of 2 square kilohertz (2 ⁿ ) is used. kHz, n = natural number). Therefore, it is necessary to resample the input audio signal having various sounds to have a proper sampling frequency.

For example, a signal having a sampling frequency of 44.1 kHz may be resampled at a sampling frequency of 32 kHz, and a signal having a sampling frequency of 22.1 kHz may be resampled at a sampling frequency of 16 kHz.

Next, in order to analyze an audio signal having various sounds using an analysis window having a varying length, it is necessary to predict a range in which a melody pitch exists. In order to predict the range in which the melody pitch exists, the second step S20 converts an audio signal input based on a time domain to a signal based on a frequency domain.

To this end, the second step S20 converts an audio signal input based on a time domain into a signal based on a frequency domain by applying a short time Fourier transform (STFT) using a fixed length analysis window. do.

Since the spectrum for each frame (frequency characteristic graph of the corresponding frame signal) can be obtained through the second step S20, in the third step S30, the amplitude of the spectrum of each frame is local maximum value. The frequency value of the portion where the local maximum value and the local maximum value appear is extracted by finding only a part forming the subfield.

That is, in the third step S30, the peak value of the amplitude and the frequency value of the position where the peak value appears in the spectrum of each frame obtained through the second step S20 are extracted and stored.

Here, the frequency value of the position where the peak value appears in the third step (S30) can be extracted using the number of frequency bins, the number of bins of the Discrete Fourier Transform (DFT) and the resampling frequency, The frequency value of the position where the peak value appears can be obtained by Equation 1 below.

Next, in the fourth step S40, the range in which the melody pitch of each frame exists is reset based on the values extracted through the third step S30, and thus the process of resetting the range in which the melody pitch exists. Is composed of the following first process (S41) to fifth process (S45).

First, in the first step S41, the values of the melody peaks are arranged in order of magnitude to extract the top K peak values and the frequency values at which the peak values appear. Here, K becomes smaller in size as the number of bins used for the Discrete Fourier Transform (DFT) decreases, but is generally set to about 100 to 200. However, if the total number of bins is too small, the resolution in the frequency domain is low and it is difficult to predict the exact peak point. Therefore, a sufficient number of frequency bins should be used.

Next, when the upper K peak values and the frequency values of the positions where the peak values appear are extracted, all peak values existing within the first melody range are found. For example, if you first set the melody range between 100 Hz and 1200 Hz, you are looking for peaks between them. Generally, only base signals are present below 100 to 150 Hz, and most melody signals are within 4 octaves of 80 Hz to 1280 Hz.

That is, in the second process S42, all peak values existing within the melody range initially set among the peak values extracted through the first process S41 and the frequency value of the position where the peak values appear are found and found as potential melody. To be nominated.

Next, the melody range should be reset based on the potential melody candidates selected through the second process (S42). To this end, in the present invention, the melody has a harmonic structure and uses the property that the energy of the melody component is most prominent.

Meanwhile, in order to reset the melody range, the frequency of each candidate in potential melody candidates is regarded as the fundamental frequency of the melody, and the harmonic component of the existing frequency among the frequency values in which the upper K peak values extracted through the first process appear. If there is a peak around the frequency corresponding to, add that peak value to the fundamental frequency peak value.

For example, if there is a peak corresponding to 100 Hz, it is set as the fundamental frequency of the melody, and 200 Hz and 300 Hz where the harmonic component of the melody should be located among the upper K peak values extracted through the first process. Check for the presence of peak values around, 400 Hz, 500 Hz, ..., 3200 Hz. In this case, if there is a peak near the harmonic component frequency, then the magnitude of the peak is added to the 100 Hz peak. If no peak value exists near the frequency at which the harmonic component is to be located, it is assumed that there is no peak of the harmonic component. As described above, the harmonic components are investigated and all peaks are added to all peaks existing within the melody range initially set.

That is, the third process (S43) is to replace the frequency of each candidate in the potential melody candidate to the base frequency of the melody in order to reset the melody range based on the potential melody candidates selected through the second process (S42) And the harmonic component is examined for all peak values that exist within the initially set melody range by adding a peak value around the frequency corresponding to the harmonic component of the fundamental frequency and adding the peak value to the fundamental frequency peak value. The process of adding the peak values is performed.

On the other hand, there is a problem in the method of adding the melody peak values as described above before resetting the melody range. That is, the low frequency fundamental frequency has a large number of harmonic components, and the high frequency fundamental frequency has a small number of harmonic components.

For example, if the sampling frequency is 32 kHz and the fundamental frequency is 200 Hz, the harmonics appear at 400 Hz, 600 Hz, 800 Hz, ..., 32 kHz, so there may be a total of 159 harmonics. On the other hand, when the fundamental frequency is 1 kHz, only 31 harmonics may exist, as the harmonics are 2 kHz, 3 kHz, 4 kHz, ..., 32 kHZ.

Therefore, in order to compensate for the above problem, in the fourth process S44, the peak value added through the third process S43 is multiplied by a smaller value as the fundamental frequency is lower, and the value is obtained from the multiplication result. N candidates are found in this large order.

This is because the greater the sum of the peaks, the more likely the frequency of the peak is the melody fundamental frequency. Accordingly, the N peak frequency values selected based on the peak sum result can be melody pitch candidates.

Finally, a method of finding N melody pitch candidates by applying the fundamental frequency and its harmonic components from the spectral peak is applied for each frame. Thus, N melody pitch candidates are found in each frame, and the range of melody pitches in the entire frame can be determined using the N melody pitch candidates extracted in each frame.

That is, in the fifth process S45, the first processes S41 to S44 are applied to each frame to find a range in which the melody pitch exists in the entire frame. The newly found melody pitch range may have a narrower range than the melody range initially set.

On the other hand, after resetting the melody pitch range, it is necessary to analyze the dynamic change section in which the melody suddenly changes. In order to analyze the melody dynamic change region, an autocorrelation coefficient between frames is determined from the short time Fourier transform (STFT) result obtained in the second step S20.

In this case, in order to obtain the autocorrelation coefficient, it is not necessary to consider all frequency bins between frames, but only an area where the harmonic component of the melody appears best. The lowest melody pitch value in the given music section may be known through the fourth step S40, that is, the melody pitch range resetting step.

Meanwhile, the highest frequency of the section in which the harmonic component of the melody is well represented is usually 2 to 3 kHz. In general, bins in the frequency region between 200 Hz and 2 kHz are used to obtain autocorrelation coefficients.

That is, in the fifth step S50, the dynamic change information of the melody pitch is extracted by extracting the autocorrelation coefficient between the frames from the result of the conversion to the signal based on the frequency domain through the second step S20. The correlation coefficient can be obtained by Equation 2 below.

Where X _l (k) is the k-th coefficient of the l-th frame of the Discrete Fourier Transform (DFT).

As described above, the autocorrelation coefficients between frames have a constant melody pitch as the autocorrelation coefficient is close to 1, and on the contrary, the dynamic change is severe as the autocorrelation coefficient is closer to 0.

Next, in the sixth step (S60), the information on the range of the melody pitch of each frame reset through the fourth step (S40) and the dynamic change of the melody pitch extracted through the fifth step (S50) The length of the analysis window corresponding to each frame is set using the information.

In this case, the length of the analysis window is determined to use a longer analysis window as the melody pitch value is lower and constant, and a shorter analysis window as the melody pitch value is high and the dynamic change is severe. If the pitch is high even if the melody is constant, a short analysis window is used. If the melody pitch is high, the signal of sufficient period (more than 5 cycles) may be contained even if the length of the analysis window is short, so that the pitch can be accurately predicted. In addition, when the dynamic change of the melody is severe, it is difficult to predict the exact pitch when using the long analysis window because the melody pitch of the signal located at the beginning and the end of the analysis window is different. Therefore, even in this case, a short analysis window is used. In general, the analysis window preferably uses an analysis window between 32 ms and 92 ms, but the present invention is not limited thereto.

Next, in the seventh step (S70), the audio signal having multiple notes passes the signal components in the range including the fundamental frequency and harmonic information of the melody pitch as it is, and the signals in the lower frequency range can be removed. A pass filter (HPF) is used to remove the base and low frequency signals from the audio signal.

That is, the seventh step (S70) using the more detailed melody pitch range obtained in the fourth step (S40), before using the analysis window having a variable length obtained in the sixth step (S60), The base signal and the low frequency region signals are removed from the input audio signal.

Next, in the eighth step S80, an analysis window corresponding to each frame whose length is set in the sixth step S60 of the audio signal passing through the high frequency pass filter HPF through the seventh step S70 is opened. Short-term Fourier transform (STFT).

Lastly, in the ninth step S90, a multi-rate filter bank is used to increase the resolution of the short-term Fourier transform (STFT) audio signal in the frequency domain through the eighth step S80. Pass it through. (Same as M.Goto method)

On the other hand, the configuration of the multi-rate filter bank (Multi-Rate Filter Bank) is as shown in FIG. The reason for passing the multi-rate filter bank will be described in more detail as follows.

In general, the height of a human recognized melody is the log scale. For example, between 440 Hz and 880 Hz one octave, and between 880 Hz and 1720 Hz one octave. There is also an octave between 1720 Hz and 3440 Hz. In this manner, as the melody becomes higher, the interval between the frequencies occupied by a piano keyboard increases. However, when the Discrete Fourier Transform (DFT) having N bins in general is performed, the frequency interval indicated by one bin is constant at the sampling frequency / N Hz (Fs / N Hz).

This results in the fact that a small number of frequency bins are allocated between low frequency notes and a large number of frequency bins are allocated between high frequency notes, so that the frequency resolution varies depending on the pitch. have. In order to solve this problem, a multi-rate filter bank as shown in FIG. 2 is used.

Through the steps as described above, it is possible to analyze the frequency of the audio signal having various sounds by using the analysis window of the flexible length. And through this frequency analysis method it is possible to more accurately and easily extract the melody information of the audio signal having multiple sounds.

As described above with reference to the drawings illustrating a frequency analysis method using an analysis window having a fluid length in the audio signal having various sounds according to the present invention, the present invention is limited by the embodiments and drawings disclosed herein Of course, various modifications can be made by those skilled in the art within the technical scope of the present invention.

Claims (8)

Resampling the input audio signal (Resampling);
Converting an audio signal input based on a time domain to a signal based on a frequency domain;
A third step of extracting a peak value of an amplitude from the spectrum of each frame obtained through the second step and a frequency value at a position where the peak value appears;
A fourth step of resetting a range in which a melody pitch of each frame exists based on the values extracted through the third step;
A fifth step of extracting dynamic change information of a melody pitch by a method of determining autocorrelation coefficients between frames in a result of conversion to a signal based on a frequency domain through the second step;
Setting the length of the analysis window corresponding to each frame using information on the range in which the melody pitch of each frame reset through the fourth step exists and dynamic change information of the melody pitch extracted through the fifth step; Step 6;
A seventh step of removing the base signal and the low frequency signal from the audio signal using a high frequency pass filter (HPF);
An eighth step of performing a short time Fourier transform (STFT) on the audio signal passing through the high frequency pass filter (HPF) through the seventh step using an analysis window corresponding to each frame whose length is set in the sixth step; And
And a ninth step of passing a short-time Fourier transform (STFT) audio signal through a multi-rate filter bank to increase resolution in a frequency domain through the eighth step. A frequency analysis method using an analysis window having a floating length in an audio signal having several sounds.
The method according to claim 1,
In the second step, a short time Fourier transform (STFT) using a fixed length analysis window is applied to convert an audio signal input based on a time domain into a signal based on a frequency domain. A frequency analysis method using an analysis window having a floating length in an audio signal having sound.
The method according to claim 1,
The frequency value of the position where the peak value appears in the third step is extracted using the number of frequency bins, the number of bins of the Discrete Fourier Transform (DFT), and the resampling frequency. Frequency analysis method using an analysis window having a length.
The method according to claim 3,
The frequency value of the position where the peak value appears in the third step is a frequency analysis method using an analysis window having a fluid length in the audio signal having a multiple sound, characterized in that obtained by the following equation (1).
[Equation 1]

The method according to claim 1,
The fourth step may include: extracting a top K peak value and a frequency value at which the peak value appears by arranging the values of the melody peaks in order of magnitude;
A second step of finding all peak values existing within the melody range initially set among the peak values extracted through the first process and the frequency values at the positions where the peak values appear, and selecting them as potential melody candidates;
In order to reset the melody range based on the potential melody candidates selected through the second process, the frequency of each candidate in the potential melody candidate is regarded as the fundamental frequency of the melody, and the upper K extracted through the first process. The harmonic component is investigated for all peak values existing within the first set melody range by adding a peak value to a fundamental frequency peak value around a frequency corresponding to the harmonic component of the fundamental frequency among the frequency values at which two peak values appear. A third step of adding a peak value;
A fourth step of finding N candidates in order of increasing values among the results obtained by multiplying the peak value added through the third process by multiplying a smaller value with a lower fundamental frequency and a higher value with a higher fundamental frequency; And
And a fifth process of determining a range in which a melody pitch exists in the entire frame by applying the first to fourth processes to each frame. Frequency analysis method used.
The method according to claim 1,
The fifth step extracts the dynamic change information of the melody pitch as a method of determining the autocorrelation coefficient between frames in the result of conversion to the signal based on the frequency domain through the second step, and the autocorrelation coefficient is A frequency analysis method using an analysis window having a fluid length in an audio signal having multiple sounds, which is obtained by 2.
[Equation 2]

Where X _l (k) is the k-th coefficient of the l-th frame of the Discrete Fourier Transform (DFT). The method according to claim 1,
In the sixth step, the length of the analysis window corresponding to each frame is set, and the length of the analysis window is lower as the melody pitch value is kept constant and the longer analysis window is used. Frequency analysis method using an analysis window having a fluid length in the audio signal having a plurality of sounds, characterized in that.
Claim 8 was abandoned when the registration fee was paid. A computer-readable recording medium having recorded thereon a computer program relating to a frequency analysis method using an analysis window having a fluid length in an audio signal having multiple sounds according to claims 1 to 7.

Images