KR101106185B1 - An apparatus and a method for Melody Extraction of Polyphonic Audio by using Harmonic Structure Model and Variable Length Window - Google Patents

Abstract

The present invention relates to a method for extracting a melody from an audio signal having several sounds, and a technical problem to be solved is to provide a method for extracting only melody components from music having several sounds, such as a song or an orchestra.
To this end, the present invention, the frequency analysis step of analyzing the frequency using an analysis window having a fluid length, when the audio signal having a plurality of sounds, and Melody Pitch candidate extraction step for extracting melody pitch candidates using a harmonic structure model from the audio signal having various sounds analyzed through the frequency analysis step and melody pitch candidates extracted in the melody pitch candidate extraction step Disclosed is a method of extracting a melody using a harmonic structure model and an analysis window having a fluid length in an audio signal having a plurality of sounds, the method comprising determining a melody line.

Description

An apparatus and a method for Melody Extraction of Polyphonic Audio by using Harmonic Structure Model and Variable Length Window}

The present invention relates to a method for extracting a melody component from a next audio signal, i.e. an audio signal having several tones.

Recently, with the development of audio technology, music recording technology, and playback devices, the digitalization of music is possible, and as a result, new music is increasing every year.

As a practical example, in 2001, about 30,000 albums were released worldwide in a year, but in 2007, about 80,000 albums were released, more than double the number.

Recently, many songs are released as new albums, such as digital album formats, which are released from CDs (Compact Disk) albums on the Internet and users download and use songs. As the number increased, so did new genres and types of music.

As a result, consumers have more access to music than ever before, and they have more choices based on their tastes and interests.

However, as the genre of music and the number of records released increase, it becomes difficult to manage the music systematically, and as so many songs are created every day, many songs disappear before people listen or recognize their existence. As the number of records increased, it became difficult to manage problems such as plagiarism.

In order to efficiently use and manage the ever-increasing variety of music, there is a need for a technology capable of extracting and analyzing different features among music, thus solving the problems of music management described by many companies and organizations. To this end, information on song titles, singers, singers' gender, composers, periods, lyrics, and the like has been stored and managed as digital information based on a text format. In addition, these data provided a foundation for building a text-based music information retieval system.

However, in the character-based music database described above, there is a problem in that music information stored in the form of characters requires too much time and effort in that a person actually writes all the information directly into the database.

And as time goes on, more and more music flows around the world every day, so more time and effort is needed to build a corresponding character-based database.

Due to the limitations of such character-based databases, there is a growing interest in systems that automatically extract and analyze music contents information such as rhythm, mood, chords, and melody of music. As music interpretation technology advances, interest and possibility for a system for automatically interpreting music contents are increasing.

The music content information may include elements such as rhythm, mood, chord, melody, etc. Among the elements, the most useful element for music analysis and interpretation is melody. This is because the melody is the most easily and long-lasting part of people listening to music, and the melody best represents the characteristics of the music and is also suitable for comparing the similarities between music.

On the other hand, such a melody is not clearly defined technically, the definition of a commonly used melody can be described as follows.

When most people listen to music with different instruments or voices and think about it, they tend to think of the score of a single note rather than the score of all the instruments. For example, music that sings a person, such as a song or pop song, remembers the sound of a person, and in the case of orchestra music where several instruments are played at the same time, it remembers the sound of an instrument that leads the entire music. Tend to. Here the human voice or the sound of the leading instrument can be defined as a melody in general.

As described above, people tend to remember only the melody of the music when they listen to the music and remember the music, and the melody is more important in that it is a characteristic that can best represent the characteristics of the music.

Therefore, it can be said that the necessity of accurately extracting and interpreting the melody containing such important information from the audio signal is very high.

The present invention has been invented to solve the above problems, and based on the property that the melody component has a harmonic component and the energy of the melody component is most prominent, only the melody component in music having various notes such as a music or orchestra The purpose is to provide a method that can be extracted.

In order to achieve the above object, the method of extracting a melody according to the present invention comprises: a frequency analysis step of analyzing a frequency using an analysis window having a fluid length when an audio signal having several sounds is input, and the frequency analysis step Melody pitch candidate extraction step for extracting melody pitch candidates using a harmonic structure model from multiple sound audio signals analyzed through a melody pitch and melody pitch candidates extracted in the melody pitch candidate extraction step to determine a melody line A line determination step.

First, the frequency analysis step may include a first step of resampling an audio signal having a plurality of input sounds, and an audio signal input based on a time domain as a signal based on a frequency domain. A second step of converting, a third step of extracting a peak value of an amplitude and a frequency value of a position at which the peak value appears in a spectrum of each frame obtained through the second step, and a value extracted through the third step And a method of recognizing the autocorrelation coefficient between frames in the result of the conversion of the melody pitch of each frame to the signal based on the frequency domain through the second step. Information on a range in which the melody pitch of each frame reset through the fourth step is present; A sixth step of setting the length of the analysis window corresponding to each frame by using the dynamic change information of the melody pitch extracted through the fifth step; and a base signal and a low frequency in the audio signal using a high frequency pass filter (HPF). In the seventh step of removing the signal, and the audio signal passing through the high-frequency pass filter (HPF) through the seventh step using the analysis window corresponding to each frame whose length is set in the sixth step using a short time Fourier transform ( The eighth step and the eighth step of passing the short-term Fourier transform (STFT) audio signal through a multi-rate filter bank in order to increase the resolution of the frequency domain. Include.

In the second step, a short time Fourier transform (STFT) using a fixed length analysis window is applied to convert an audio signal input on a time domain basis into a signal on a frequency domain basis.

The frequency value of the position where the peak value appears in the third step is extracted using the number of frequency bins, the number of bins of the discrete Fourier transform (DFT), and the resampling frequency. The frequency value of the position where the peak value of the third step appears can be obtained by Equation 1 below.

[Equation 1]

The fourth step may include a first process of extracting the upper K peak values and the frequency value at which the peak values appear, and a first one of the peak values extracted through the first process and the frequency value at the position at which the peak value appears. A second process of finding all peak values existing within the set melody range and selecting them as potential melody candidates, and resetting the melody range based on the potential melody candidates selected through the second process. In the candidate, the frequency of each candidate is regarded as the fundamental frequency of the melody, and if there is a peak value around a frequency corresponding to the harmonic component of the fundamental frequency among the upper K peak values extracted through the first process, the peak value is used. All peaks within the initially set melody range by adding to the frequency peak value And a third step of investigating harmonic components and adding peak values, and a result obtained by multiplying the peak value added through the third step with a smaller value at a lower fundamental frequency and a higher value at a higher fundamental frequency. And a fourth process of finding N candidates in order of increasing values, and a fifth process of finding a range in which a melody pitch exists in all frames by applying the first to fourth processes for each frame.

The fifth step extracts the dynamic change information of the melody pitch as a method of determining the autocorrelation coefficient between frames in the result of conversion to the signal based on the frequency domain through the second step, and the autocorrelation coefficient is It can be obtained by 2.

[Equation 2]

In the sixth step, the length of the analysis window corresponding to each frame is set. The length of the analysis window is lower as the melody pitch value is kept constant and the longer analysis window is used, and the higher the melody pitch value and the dynamic change is shorter, the shorter analysis window is used. Melody extraction method using a harmonic structure model and an analysis window having a fluid length in the audio signal having multiple sounds.

Next, the extracting the melody pitch candidate may include a first step of performing a Fourier transform (STFT) on the input audio signal when the audio signal having various sounds analyzed through the frequency analysis step is input, and the first step; A second step of converting the unit of the short-term Fourier transform (STFT) audio signal through the first step from Hertz (Hz) to cents, and the audio signal converted to the cent unit by the second step In the third step of extracting the peak point for each frame, the fourth step of measuring the weight of the harmonic structure model contained by the melody pitch candidates, and the weight information measured in the fourth step, Comprising a fifth step of compensating the energy level difference according to the frequency and the sixth step of extracting N melody pitch candidates in each frame.

In the second step, the unit of the short-time Fourier transform (STFT) audio signal is converted from hertz (Hz) to cents through the first step, and the conversion is based on Equation 3 below.

&Quot; (3) "

In the third step, a peak point is extracted for each frame of the audio signal converted into cents through the second step, and the peak point is a local maximum value of each frame.

The fourth step is performed by inner product of the short-term Fourier transform (STFT) result and harmonic structure model which is set to 0 except for the local maximum value extracted through the third step. Measure the weight of how many melody pitch candidates contain the harmonic structure model.

In the fifth step, in the weight information measured in the fourth step, a difference in energy level according to frequency is compensated for, and the compensation is performed by multiplying a small value in a low frequency region by a larger value in a high frequency region. To do it.

The sixth step extracts N melody pitch candidates in each frame based on the result of the weight information compensated through the fifth step.

Lastly, the determining of the melody line may include a first step of receiving information about N melody pitch candidates and weights of the pitch candidates in each frame, and a second step of setting a start frame of the melody line. And a third step of selecting the N melody pitch candidates received in the first step from the start frame set through the second step and arranging them in the order of increasing weight, and the two neighboring melody pitch candidates A fourth step of determining whether the predetermined melody line criterion is satisfied; and if it is determined that the two neighboring melody pitch candidates satisfy the predetermined melody line criterion through the fourth step, each melody section is connected through the melody line connection. A fifth step of selecting the number of melody line candidates and a maximum of N melody line candidates selected through the fifth step And a sixth step of selecting an enemy melody line and a seventh step of removing and correcting a melody pitch value or a wrong melody pitch value protruding from the selected melody line through the sixth step.

The second step sets the first frame for determining the melody line as the start frame.

The melody line reference of the fourth step is set based on the characteristics of the melody line and is set differently according to the genre or characteristics of the music.

In the fifth step, if it is determined through the fourth step that two neighboring melody pitch candidates satisfy a predetermined melody line criterion, the melody line is connected to the next frame pitch candidates from the first candidate.

In the fifth step, when the melody lines of the section based on the first candidate are connected, the melody lines are connected to the Nth melody pitch of the start frame in this way to connect a total of N melody lines in one melody line section. N melody line candidates are selected by setting a next frame at which the melody line ends as another start frame and repeating the processes of steps 3 to 4 to form a new melody line candidate.

The sixth step selects an optimal melody line from the N melody line candidates selected through the fifth step, and determines the optimal melody line based on weight information of pitch values of the melody line.

In addition, the method of extracting a melody using a harmonic structure model and an analysis window having a flexible length according to the present invention may be used in the form of a computer-readable recording medium recording a computer program.

In addition, the melody extraction method using the harmonic structure model and the analysis window having a fluid length according to the present invention can be applied to the melody extraction system.

In addition, the method of extracting a melody using a harmonic structure model and an analysis window having a flexible length can be applied to an automatic music content information extraction system.

In addition, the method of extracting a melody using a harmonic structure model and an analysis window having a flexible length according to the present invention can be applied to a content-based music retrieval system.

As described above, according to the method of extracting melody according to the present invention, an audio signal having multiple sounds is based on the characteristic that the melody has a harmonic model and that the energy of the melody is most prominent compared to the energy of other sounds. It is effective to extract the melody component from.

In addition, the present invention can reduce the effects of harmonic interference and octave mismatch caused by the harmonic components of the signal other than the melody component and the base signal, and extract the melody component at a reliable level. It has an effect.

Accordingly, the method of extracting melody according to the present invention has an effect of enabling a system for automatically extracting music content information such as a melody or a content-based music information retrieval system.

1 is a flowchart of a method of extracting a melody using a harmonic structure model and an analysis window having a fluid length according to the present invention.
2 is a flowchart showing each detailed step of the frequency analysis step.
3 is a flowchart illustrating a process of passing a multi-rate filter bank of an audio signal having a plurality of sounds.
4 is a flowchart showing each detailed step of the melody pitch candidate extraction step.
5 is a graph illustrating an example of a harmonic structural model.
6 is a flowchart showing each detailed step of the melody line determination step.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention; First, it should be noted that the same components or parts among the drawings denote the same reference numerals whenever possible. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted so as not to obscure the subject matter of the present invention.

The method for extracting melody according to the present invention is based on the property that the melody component has harmonic properties, and the energy of the melody component appears most prominently.

However, there are various problems that must be overcome in order to extract a reliable result melody from the next audio signal with several notes. The most common problems are as follows.

-Harmonic Interference: In the case of music with various notes, it is difficult to determine the harmonic component of the melody because the harmonic component of the various instruments and the harmonic component of the melody pitch to be found interfere with each other.

Octqve Mismatch: For music, a note and an octave higher than that note have two times the fundamental frequency. In this case, a large portion of the harmonic components overlap each other, which causes difficulties when trying to find the pitch of the melody.

Dynamic Variation of Melody: Melody pitch is a variation of the fundamental frequency due to the nature of the music. In other words, you play a particular note and then switch to another note. It is difficult to find the melody pitch frequency because the fundamental frequency changes quickly in the transition period and the melody start and end sections.

The present invention relates to a method for enabling reliable melody extraction even in the above-described problems, Figure 1 is a flow chart of the melody extraction method using a harmonic structure model and an analysis window having a fluid length in accordance with the present invention have. As shown in FIG. 1, the method for extracting a melody according to the present invention includes a frequency analysis step S100, a melody pitch candidate extraction step S200, and a melody line determination step S300. Hereinafter, the above steps will be described in more detail.

The frequency analysis step S100 is a step of analyzing a frequency using an analysis window having a fluid length when an audio signal having several sounds is input. A flowchart of each detailed step of the frequency analysis step S100 is illustrated in FIG. 2.

First, a first step S110 is a step of resampling the audio signal in order to convert the input audio signal into a format suitable for analysis.

In the present invention, a multi-rate filter bank is used to increase the resolution of the frequency domain. For this purpose, a sampling frequency of 2 square kilohertz (2 ⁿ ) is used. kHz, n = natural number). Therefore, it is necessary to resample the input audio signal having various sounds to have a proper sampling frequency.

For example, a signal having a sampling frequency of 44.1 kHz is resampled at a sampling frequency of 32 kHz, and a signal having a sampling frequency of 22.1 kHz is resampled at a sampling frequency of 16 kHz.

Next, in order to analyze an audio signal having various sounds using an analysis window having a varying length, it is necessary to predict a range in which a melody pitch exists. In order to predict the range in which the melody pitch exists, the second step S120 converts an audio signal input based on a time domain to a signal based on a frequency domain.

To this end, the second step S120 converts an audio signal input based on a time domain into a signal based on a frequency domain by applying a short time Fourier transform (STFT) using a fixed length of a fixed Lenght Analysis Window. do.

Since the spectrum (frequency characteristic graph of the corresponding frame signal) for each frame can be obtained through the second step S120, in the third step S130, the amplitude is localized in the spectrum of each frame. The frequency value of the portion where the local maximum value and the local maximum value appear is extracted by finding only a part forming the subfield.

That is, in the third step S130, the peak value of the amplitude and the frequency value of the position where the peak value appears in the spectrum of each frame obtained through the second step S120 are extracted and stored.

Here, the frequency value of the position where the peak value appears in the third step (S130) may be extracted using the number of frequency bins, the number of bins of the Discrete Fourier Transform (DFT) and the resampling frequency. The frequency value of the position where the peak value appears can be obtained by Equation 1 below.

Next, in the fourth step S140, the range in which the melody pitch of each frame exists is reset based on the values extracted through the third step S130, and thus the process of resetting the range in which the melody pitch exists. Is composed of the following first process (S141) to fifth process (S145).

First, in a first step (S141), the peak point of the melody is obtained, and then the upper K peak values and the frequency values at which the peak values appear are extracted. Here, K becomes smaller in size as the number of bins used for the Discrete Fourier Transform (DFT) decreases, but is generally set to about 100 to 200. However, if the total number of bins is too small, the resolution in the frequency domain is low, so that it is difficult to predict the exact peak point. Therefore, the total number of bins needs to be moderately large. In general, at the 32 kHz sampling frequency, it is preferable to set the total number of bins to 4096 or more.

Next, when the upper K peak values and the frequency values of the positions where the peak values appear are extracted, all peak values existing within the melody range first set among them are found. For example, if the range of the first melody is set between 100 Hz and 1200 Hz, the peak existing between them is found. Generally, only base signals are present below 100 to 150 Hz, and most melody signals are within 4 octaves of 80 Hz to 1280 Hz.

That is, in the second process S142, all peak values existing within the melody range initially set among the peak value extracted through the first process S141 and the frequency value of the position where the peak value appears are found and potentials are found. Selected as a melody candidate.

Next, the melody range should be reset based on the potential melody candidates selected through the second process (S142). To this end, in the present invention, the melody has a harmonic structure and uses the property that the energy of the melody component is most prominent.

Meanwhile, in order to reset the melody range, the frequency of each candidate in potential melody candidates is regarded as the fundamental frequency of the melody, and is applied to the harmonic component of the fundamental frequency among the upper K peak values extracted through the first process (S141). If there is a peak value around that frequency, add that peak value to the fundamental frequency peak value.

For example, if there is a peak at 100 Hz, set it as the fundamental frequency of the melody, and around 200 Hz, 300 Hz, 400 Hz, 500 Hz, ..., 3200 Hz where the harmonic component of the melody should be located. Check if the peak value exists at. In this case, if there is a peak near the harmonic component frequency, then the magnitude of the peak is added to the 100 Hz peak. If no peak value exists near the frequency at which the harmonic component is to be located, it is assumed that there is no peak of the harmonic component. As described above, the harmonic components are investigated and all peaks are added to all peaks existing within the melody range initially set.

That is, the third process S143 converts the frequency of each candidate from the potential melody candidates into the base frequency of the melody in order to reset the melody range based on the potential melody candidates selected through the second process S142. If there is a peak value around a frequency corresponding to the harmonic component of the fundamental frequency, the harmonic component is examined for all peak values existing within the initially set melody range by adding the peak value to the fundamental frequency peak value. The process of adding the peak values is performed.

On the other hand, there is a problem in the method of adding the melody peak values as described above before resetting the melody range. That is, the low frequency fundamental frequency has a large number of harmonic components, and the high frequency fundamental frequency has a small number of harmonic components.

For example, if the sampling frequency is 32 kHz and the fundamental frequency is 200 Hz, the harmonics appear at 400 Hz, 600 Hz, 800 Hz, ..., 32 kHz, so there may be a total of 159 harmonics. On the other hand, when the fundamental frequency is 1 kHz, only 31 harmonics may exist, such that the harmonics are 2 kHz, 3 kHz, 4 kHz, ..., 32 kHz.

Therefore, in order to compensate for the above problem, in the fourth process S144, the peak value added through the third process S143 is multiplied by a smaller value as the fundamental frequency is lower, and the value is obtained from the multiplication result. N candidates are found in this large order.

This is because the greater the sum of the peaks, the more likely the frequency of the peak is the melody fundamental frequency. Accordingly, the N peak frequency values selected based on the peak sum result can be melody pitch candidates.

Finally, a method of finding N melody pitch candidates by applying the fundamental frequency and its harmonic components from the spectral peak is applied for each frame. Therefore, it is possible to find N melody pitch candidates in each frame, and use the N melody pitch candidates extracted in each frame to find a range in which a melody pitch exists in the entire frame.

That is, in the fifth process S145, the first processes S141 to the fourth process S144 are applied to each frame to find a range in which the melody pitch exists in the entire frame. The newly found melody pitch range may have a narrower range than the melody range initially set.

On the other hand, after resetting the melody pitch range, it is necessary to analyze the dynamic change section in which the melody suddenly changes. In order to analyze the melody dynamic change region, the autocorrelation coefficient between frames is determined from the short-term Fourier transform (SFTF) result obtained in the second step S120.

In this case, in order to obtain the autocorrelation coefficient, it is not necessary to consider all frequency bins between frames, but only an area where the harmonic component of the melody appears best. Here, the lowest melody pitch value in the given music section may be known through the fourth step S140, that is, the melody pitch range resetting step.

Meanwhile, the highest frequency of the section in which the harmonic component of the melody is well represented is usually 2 to 3 kHz. In general, bins in the frequency region between 200 Hz and 2 kHz are used to obtain autocorrelation coefficients.

That is, in the fifth step S150, the dynamic change information of the melody pitch is extracted by extracting the autocorrelation coefficient between the frames from the result obtained by converting the signal to the frequency domain reference signal through the second step S120. The correlation coefficient can be obtained by Equation 2 below.

Where X _l (k) is the k-th coefficient of the l-th frame of the Discrete Fourier Transform (DFT).

As described above, the autocorrelation coefficients between frames have a constant melody pitch as the autocorrelation coefficient is close to 1, and on the contrary, the dynamic change is severe as the autocorrelation coefficient is closer to 0.

Next, in the sixth step (S160), the information on the range in which the melody pitch of each frame reset through the fourth step (S140) exists and the dynamic change of the melody pitch extracted through the fifth step (S150). The length of the analysis window corresponding to each frame is set using the information.

In this case, the length of the analysis window is determined to use a long analysis window as the melody pitch value is low and constant and to use a short analysis window as the melody pitch value is high and the dynamic change is severe. If the pitch is high even if the melody is constant, a short analysis window is used. When the melody pitch is high, even if the analysis window is short, it can contain a sufficient period (more than 5 cycles) of the signal, so that the pitch can be accurately predicted. In addition, when the dynamic change of the melody is severe, it is difficult to predict the exact pitch when using the long analysis window because the melody pitch of the signal located at the beginning and the end of the analysis window is different. Therefore, even in this case, a short analysis window is used. In general, the analysis window preferably uses an analysis window between 32 ms and 92 ms, but the present invention is not limited thereto.

Next, in the seventh step S170, an audio signal having several notes is passed through as it is and a signal component in a range including the fundamental frequency and harmonic information of the melody pitch is left as it is, so that the signals in the lower frequency region can be removed. A pass filter (HPF) is used to remove the base and low frequency signals from the audio signal.

That is, the seventh step (S170) uses the more detailed melody pitch range obtained in the fourth step (S140), before using the analysis window having the variable length obtained in the sixth step (S160), The base signal and the low frequency region signals are removed from the input audio signal.

Next, in the eighth step S180, an analysis window corresponding to each frame whose length is set in the sixth step S160 of the audio signal passing through the high frequency pass filter HPF through the seventh step S170 is opened. Short-term Fourier transform (STFT).

Finally, in the ninth step 190, a multi-rate filter bank is used to increase the resolution of the frequency domain of the short-term Fourier transform (STFT) audio signal through the eighth step (S180). Pass it through (same as the M.Goto method).

On the other hand, the configuration of the multi-rate filter bank (Multi-Rate Filter Bank) is as shown in FIG. The reason for passing the multi-rate filter bank will be described in more detail as follows.

In general, the melody pitch that a human recognizes is a log scale. For example, between 440 Hz and 880 Hz one octave, and between 880 Hz and 1720 Hz one octave. There is also an octave between 1720 Hz and 3440 Hz. In this manner, as the melody becomes higher, the interval between the frequencies occupied by a piano keyboard increases. However, when the Discrete Fourier Transform (DFT) having N bins in general is performed, the frequency interval indicated by one bin is constant at the sampling frequency / N Hz (Fs / N Hz).

This results in a problem that a small number of frequency bins are allocated between low frequency sounds and a large number of frequency bins are allocated between high frequency sounds, so that the frequency resolution according to the pitch is different. . In order to solve this problem, a multi-rate filter bank as shown in FIG. 3 is used.

As described above, the frequency analysis step (S100) is the frequency of the audio signal having various sounds input through each detailed step including the first step (S110) to the ninth step (S190) as shown in FIG. Can be analyzed.

Meanwhile, the melody pitch candidate extraction step S200 is a step of extracting melody pitch candidates using a harmonic structure model from an audio signal having various sounds analyzed through the frequency analysis step S100. A flowchart of each detailed step of the melody pitch candidate extraction step S200 is illustrated in FIG. 4.

First, a first step S210 is a step of performing a short time Fourier transform (SFTF) of an input audio signal.

In the second step S220, the unit of the audio signal short-time Fourier transformed (STFT) through the first step (S210) is converted from hertz (Hz) to cents (Cent). Is performed by.

The reason for converting a unit of an audio signal from hertz (Hz) to cents is that a cent unit is a unit that can linearly represent the musical scale on a log scale. to be.

For example, one Hertz (Hz) difference is 2 ^2/12 difference, one octave is 2 times frequency difference, while in Cents one piano key is always 100 Cent difference. In addition, one octave is always 1200 cents apart, and mapping in cents increases the resolution of the low-frequency bands, making it easier to observe harmonics that were invisible due to harmonic interference in hertz. There is an advantage that it can.

In the third step S230, a local maximum value, that is, a peak point, is extracted for each frame of the audio signal converted in cent units through the second step S220. This is because there is a high probability that the harmonic pitch is located at the point where the peak is formed.

On the other hand, in the cent unit, in contrast to the hertz unit, a cent spread phenomenon occurs in which low frequency components occupy more cents. This phenomenon causes interference in obtaining the weight of the harmonic component when the base signal of the low frequency region has a large energy value. Therefore, if only the local maximum value point is considered, only the portion where the actual harmonic component is located can be extracted, and the effect of harmonic interference caused by the base signal in the low frequency region can be reduced.

The fourth step (S240) measures the weight of the harmonic structure model included in the melody pitch candidates. In more detail, in the fourth step S240, a short time Fourier transform (STFT) result and a harmonic structure model in which the remaining parts except for the local maximum value extracted through the third step S230 are set to 0 are set. The inner product is used to measure the weight of how many preset melody pitch candidates contain the harmonic structure model.

Here, the reason for using the harmonic structure model as shown in FIG. 5 is that the size of the harmonic component of the melody pitch decreases as the frequency increases. The reason why the harmonic component becomes smaller at higher frequencies is due to the radiation effect of the signal.

However, in most musical instruments, the harmonic component is much larger than the melody harmonic component in that the harmonic size decreases at higher frequencies. As described above, in consideration of the difference between the musical instrument and the melody harmonic component, when setting a harmonic model in which the harmonic size decreases to a high frequency as shown in FIG. 5, a larger weight is assigned to the melody harmonic component. This occurs.

Generally, harmonic components are known to occur at frequencies corresponding to integer multiples of the fundamental frequency. However, there may be times when harmonic components do not exactly occur at integer multiples of the fundamental frequency due to changes in sound or effects of other harmonic components. Therefore, the harmonic structure model is set as shown in FIG. 5 so that the harmonic structure model can be found even when it is not an integer multiple of the fundamental frequency.

Meanwhile, the weight of the harmonic structure model included in the respective melody pitch candidates may be obtained through the fourth step S240. However, the weight information does not consider the difference in energy level according to frequency.

Therefore, in the fifth step S250, the difference in energy level according to frequency is compensated for in the specific gravity information measured in the fourth step S240. To this end, in the fifth step S250, a difference in energy level according to frequency is compensated by multiplying a small value in a low frequency region by a larger value in a high frequency region.

This is because the signal in the low frequency region of the spectrum generally has a higher energy level than the signal in the high frequency region to eliminate this effect. By multiplying the relative value according to the frequency, the method according to the present invention has the effect of reducing the influence of the weight and energy of the bass signal having a low frequency. You can be stronger.

The sixth step S260 is to extract N melody pitch candidates in each frame. That is, in the sixth step S260, N melody pitch candidates are extracted in each frame based on the result of weight information compensated through the fifth step S250.

In most cases, the pitch value with the largest weight is often the melody pitch, but this is not always the case. If the harmonic component of another instrument is much larger than the melody harmonic component, or the size of the component of the melody pitch harmonic becomes smaller due to the interference of the surrounding instrument harmonics, Weight can be assigned.

Accordingly, in the present invention, in order to reduce such a problem, N melody pitch candidates are selected for each frame, and a melody line can be determined among them.

Finally, the melody line determination step (S300) is a step of determining the melody line by connecting the melody pitch candidates extracted in the melody pitch candidate extraction step (S200). That is, in the melody line determination step (S300), the melody pitch candidate extracted in the melody pitch candidate extraction step (S200) and the pitch candidates are connected more easily and accurately based on weight information of each candidate, so that a reliable melody Determine the line.

A flowchart of each detailed step of the melody line determination step S300 is illustrated in FIG. 6. As illustrated in FIG. 6, the melody line determination step S300 includes each step including the first step S310 to the seventh step S370.

First, in a first step (S310), N melody pitch candidates and information on the weight of the pitch candidates are received in each frame.

Next, in the second step S320, the start frame of the melody line is set. That is, in the second step S320, the first frame for determining the melody line is set as a start frame (start frame).

Meanwhile, in the present invention, pitch candidates of all frames are not connected to each other from the first frame to the last frame, and when the melody pitch candidates of adjacent frames do not satisfy all of the criteria determined according to the characteristics of the melody line, Determine that a change or discontinuity of the melody has occurred and disconnect the melody line. Subsequently, the next frame of the frame in which the melody line is broken is set as the start frame of the new melody line to continue another melody line. In this way, a bundle of frames connected by one melody line is determined as one section.

On the other hand, if the start frame of the new melody line is set in the second step (S320) it is possible to select the N melody pitch candidates received in the first step in the start frame, and in the third step (S330) The N melody pitch candidates are selected in the start frame set through the second step S320, and they are arranged in order of weight. In this case, the higher the weight, the more likely the melody pitch is, so that the priority is set when connecting the melody line.

Next, in a fourth step S340, it is determined whether two neighboring melody pitch candidates satisfy a predetermined melody line criterion. The melody line reference may be set based on characteristics of the melody line, and may be set differently according to the genre or characteristics of the music. At this time, the melody line characteristics used for the melody line reference may be as follows.

In the case of Vibrato in music, the human voice typically ranges from ± 60 to 200 cents and the instrument sounds from ± 20 to 30 cents.

In music, the change in sound represents a change in melody. In general, the change in sound is limited to less than one octave.

In general, the human voice has a breathing time of 50 ms or more.

Here, the melody line reference set based on the three melody line characteristics is as follows. In this case, the higher the number, the higher the priority.

1) Connect the two melody pitches if the difference between the two frame melody pitch values is within 100 cents for voice and within 50 cents for instrument sound. If two or more melody pitch candidates satisfying such a condition exist, the candidate is connected with a large weight.

2) If the difference between the two frame melody pitch values is within 200 cents, connect the two melody pitches. Here, too, when there are two or more melody pitch candidates satisfying the condition as in 1), the candidate is connected with a large weight.

3) If the difference between two frame melody pitch values is more than one octave, and the difference persists for more than 50 ms, start a new melody line from the frame with more than one octave difference.

In this case, the difference between the two melody pitch values in the above standards may be set differently according to the genre or characteristics of the music, and the present invention is not limited by the above-described standards.

Next, in the fifth step S350, if it is determined through the fourth step S340 that two neighboring melody pitch candidates satisfy a predetermined melody line criterion, N melodies in each melody section are connected through the melody line connection. Select a line candidate. In the fifth step S350, first, a melody line is connected according to the melody line reference. At this time, if the melody lines of the section based on the first candidate are connected, the melody lines are connected based on the 2,3,4, ..., N-th melody pitch of the start frame in the same manner. Next, a new melody line is formed by repeating the second step S320 and the third step S330. Through this process, N melody line candidates can be selected for each melody period.

That is, in the fifth step S350, when the melody lines of the section based on the first candidate are connected, the melody lines are connected to the Nth melody pitch of the start frame in this manner, so that a total of N pieces in the melody line section are provided. N is a method of connecting a melody line, setting a next frame at which the melody line ends as another start frame, and repeating the processes of steps S330 to S340 to form a new melody line candidate. Melody line candidates are selected.

Next, in the sixth step S360, an optimal melody line is selected from the N melody line candidates selected through the fifth step S350. At this time, the optimal melody line is determined based on weight information of pitch values of the melody line.

Finally, in the seventh step S370, a smoothing process is performed to remove and correct the melody pitch value or the wrong melody pitch value protruding from the optimal melody line selected through the sixth step S360. After the smoothing process, the melody line of the final next audio signal can be extracted.

As described above with reference to the drawings illustrating a melody extraction method according to the present invention, the present invention is not limited by the embodiments and drawings disclosed herein, but by those skilled in the art within the technical scope of the present invention Of course, various modifications may be made.

Claims (24)

A frequency analysis step of analyzing a frequency using an analysis window having a flexible length when an audio signal having several sounds is input;
A melody pitch candidate extraction step of extracting melody pitch candidates using a harmonic structure model from the audio signal having various sounds analyzed through the frequency analysis step; And
A melody line determination step of determining a melody line by connecting melody pitch candidates extracted in the melody pitch candidate extraction step;
The frequency analyzing step may include a first step of resampling an audio signal having various input sounds;
Converting an audio signal input based on a time domain to a signal based on a frequency domain;
A third step of extracting a peak value of an amplitude from the spectrum of each frame obtained through the second step and a frequency value at a position where the peak value appears;
A fourth step of resetting a range in which a melody pitch of each frame exists based on the values extracted through the third step;
A fifth step of extracting dynamic change information of a melody pitch by a method of determining autocorrelation coefficients between frames in a result of conversion to a signal based on a frequency domain through the second step;
Setting the length of the analysis window corresponding to each frame using information on the range in which the melody pitch of each frame reset through the fourth step exists and dynamic change information of the melody pitch extracted through the fifth step; Step 6;
A seventh step of removing a base signal and a low frequency signal from the audio signal using a high frequency pass filter (HPF);
An eighth step of performing a short time Fourier transform (STFT) on the audio signal passing through the high frequency pass filter (HPF) through the seventh step using an analysis window corresponding to each frame whose length is set in the sixth step; And
And a ninth step of passing a short-time Fourier transform (STFT) audio signal through a multi-rate filter bank to increase resolution in a frequency domain through the eighth step. Melodic extraction method using harmonic structure model and flexible length analysis window in multi-tone audio signal.
delete The method according to claim 1,
In the second step, a short time Fourier transform (STFT) using a fixed length analysis window is applied to convert an audio signal input based on a time domain into a signal based on a frequency domain. Melody Extraction Method Using Harmonic Structure Model and Analysis Window with Flexible Length in Sound Audio Signal.
The method according to claim 1,
The frequency value of the position where the peak value appears in the third step is extracted using the number of frequency bins, the number of bins of the discrete Fourier transform (DFT), and the resampling frequency. Melody extraction method using structural model and analysis window with flexible length.
The method of claim 4,
The method of extracting a melody using a harmonic structure model and an analysis window having a fluid length in an audio signal having various sounds, characterized in that the frequency value of the position where the peak value appears in the third step is obtained by Equation 1 below. .
[Equation 1]

The method according to claim 1,
The fourth step may include: extracting a top K melody peak value and a frequency value at which the peak value appears;
A second step of finding all peak values existing within the melody range initially set among the peak values extracted through the first process and the frequency values at the positions where the peak values appear, and selecting them as potential melody candidates;
In order to reset the melody range based on the potential melody candidates selected through the second process, the frequency of each candidate in the potential melody candidate is regarded as the fundamental frequency of the melody, and the peak value extracted through the first process. If there is a peak value around a frequency corresponding to the harmonic component of the fundamental frequency, the harmonic component is examined for all peak values existing within the first set melody range by adding the peak value to the fundamental frequency peak value. Adding a third process;
The peak value added to the fundamental frequency through the third process is multiplied by a compensation value. When one fundamental frequency is smaller than the other fundamental frequency, the compensation value smaller than the compensation value for the other fundamental frequency is multiplied. A fourth step of multiplying a compensation value that is larger than a compensation value for another fundamental frequency when the frequency is greater than another fundamental frequency, and finding N candidates in order of increasing values from the multiplication result; And
A fifth process of applying the first to fourth processes for each frame to find a range in which a melody pitch exists in the entire frame;
Melody extraction method using a harmonic structure model and an analysis window having a fluid length in the audio signal having a plurality of sounds, comprising a.
The method according to claim 1,
The fifth step extracts the dynamic change information of the melody pitch as a method of determining the autocorrelation coefficient between frames in the result of conversion to the signal based on the frequency domain through the second step, and the autocorrelation coefficient is Method for extracting melody using harmonic structure model and analysis window with flexible length in audio signal with multiple sound, characterized by 2).
[Equation 2]

The method according to claim 1,
The sixth step sets the length of the analysis window corresponding to each frame, and the length of the analysis window is set according to a melody pitch value. How to extract melody using window.
The method according to claim 1,
The melody pitch candidate extracting step may include: a first step of performing a short time Fourier transform (STFT) on the input audio signal when an audio signal having various sounds analyzed through the frequency analysis step is input;
A second step of converting a unit of an audio signal having a short time Fourier transform (STFT) from hertz (Hz) to cents through the first step;
A third step of extracting a peak point for each frame of the audio signal converted in cent units through the second step;
A fourth step of measuring a weight of the harmonic structure model included in the melody pitch candidates;
A fifth step of compensating for a difference in energy level according to frequency in weight information measured in the fourth step; And
Extracting N melody pitch candidates in each frame;
Melody extraction method using a harmonic structure model and an analysis window having a fluid length in the audio signal having a plurality of sounds, comprising a.
The method according to claim 9,
In the second step, the unit of the short-time Fourier transform (STFT) audio signal is converted from hertz (Hz) to cents (Cent) through the first step, and the conversion is based on Equation 3 below. Melodic extraction method using harmonic structure model and fluidized analysis window in multi-sound audio signal.
[Equation 3]

The method according to claim 9,
The third step extracts a peak point for each frame of the audio signal converted into cents through the second step, and the peak point is a local maximum value of each frame. Melody extraction method using harmonic structure model and fluidized analysis window in multi-tone audio signal.
The method according to claim 9,
The fourth step is performed by inner product of the short-term Fourier transform (STFT) result and harmonic structure model which is set to 0 except for the local maximum value extracted through the third step. A method of extracting a melody using an analysis window having a harmonic structure model and a fluid length in an audio signal having multiple notes, wherein the weight of the melody pitch candidates includes a harmonic structure model.
The method according to claim 9,
In the fifth step, in the weight information measured in the fourth step, a difference in energy level according to frequency is compensated for, and the compensation is performed by multiplying one value in one frequency region and another value in another frequency region. However, when the one frequency region is higher than the other frequency region, the work value is greater than the other value. On the contrary, when the one frequency region is lower than the other frequency region, the work value is smaller than the other value. Melody Extraction Method Using Harmonic Structure Model and Analysis Window with Flexible Length in Excitation Audio Signal.
The method according to claim 9,
The sixth step extracts N melody pitch candidates from each frame based on the result of the weight information compensated through the fifth step. Melody Extraction Method Using Analysis Window with Flexible Length.
The method according to claim 1,
The determining of the melody line may include: a first step of receiving N melody pitch candidates and information on the weight of the pitch candidates in each frame;
Setting a start frame of the melody line;
A third step of selecting N melody pitch candidates received in the first step from the start frame set through the second step and sorting the melody pitch candidates in ascending order of weight;
A fourth step of determining whether two neighboring melody pitch candidates satisfy a predetermined melody line criterion;
A fifth step of selecting N melody line candidates for each melody section through melody line connection when it is determined through the fourth step that two neighboring melody pitch candidates satisfy a predetermined melody line criterion;
A sixth step of selecting an optimal melody line from the N melody line candidates selected in the fifth step; And
A seventh step of removing and correcting a melody pitch value or an incorrect melody pitch value protruding from the selected melody line through the sixth step;
Melody extraction method using a harmonic structure model and an analysis window having a flexible length in the audio signal having a multi-tone.
The method according to claim 15,
The second step is a method of extracting a melody using a harmonic structure model and an analysis window having a flexible length in an audio signal having several sounds, wherein the first frame for determining a melody line is set as a start frame.
The method according to claim 15,
The melody line criterion of the fourth step is set based on the characteristics of the melody line, and is set differently according to the genre or characteristics of the music. Melody extraction method using the analysis window having.
The method according to claim 15,
In the fifth step, if it is determined through the fourth step that two neighboring melody pitch candidates satisfy a predetermined melody line criterion, the melody line is connected to the next frame pitch candidates from the first candidate. Melody Extraction Method Using Harmonic Structure Model and Analysis Window with Flexible Length in Sound Audio Signal.
The method according to claim 18,
In the fifth step, when the melody lines of the section based on the first candidate are connected, the melody lines are connected to the Nth melody pitch of the start frame in this way to connect a total of N melody lines in one melody line section. N melody line candidates are selected by setting a next frame at which the melody line ends as another start frame and repeating the process of steps 3 to 4 to form a new melody line candidate. Melodic extraction method using harmonic structure model and flexible length analysis window in multi-tone audio signal.
The method of claim 19,
The sixth step selects an optimal melody line from among the N melody line candidates selected through the fifth step, and determines the optimal melody line based on weight information of pitch values of the melody line. Melody extraction method using harmonic structure model and fluidized analysis window in multi-tone audio signal.
A computer-readable recording medium having recorded thereon a computer program relating to a method of extracting a melody using a harmonic structure model according to any one of claims 1 to 3 and an analysis window having a fluid length.
A melody extraction system using a harmonic structure model according to any one of claims 1 and 3 to 20 and a melody extraction method using an analysis window having a fluid length.
A system for automatically extracting music content information using a harmonic structure model according to any one of claims 1 and 3 to 20 and a melody extraction method using an analysis window having a flexible length.
Content-based music retrieval system using a harmonic structure model according to any one of claims 1, 3 to 20 and a melody extraction method using an analysis window having a flexible length.

Images