KR100512143B1 - Method and apparatus for searching of musical data based on melody - Google Patents

Abstract

The present invention relates to a method for retrieving music data from a music database based on extracted melody feature information and retrieving the melody feature information.

The present invention extracts a melody feature from music data of a music database, extracts a melody feature from music data input by a query, and extracts the music data melody feature information and the input query music melody feature information of the extracted music database. A melody-based music data retrieval apparatus and a melody-based music data retrieval method characterized by comparing and outputting a search result of music data according to the similarity.

Description

METHOD AND APPARATUS FOR SEARCHING OF MUSICAL DATA BASED ON MELODY}

The present invention relates to a method for retrieving music data from a music database based on extracted melody feature information and retrieving the melody feature information.

Searching for multimedia data such as movies, photos, and music is performed by searching for bibliographic information such as creator information, title, creation date, etc., and searching based on multimedia contents. have.

However, the former search method is often unable to fully reflect the user's search intent. For example, if you want to find a picture with a blue sky on a white beach, you can't search if there is no bibliographic information describing the landscape. However, the latter retrieval method describes color information, which is information that can be automatically extracted by the image processing technique from the photo itself, as feature information for content-based retrieval of the photo. The desired picture can be retrieved by comparing the feature information extracted from the query image with the feature information of the search target image.

When a user searches for multimedia in this way, if the user searches using the contents of the multimedia, the user can conveniently search more precisely according to the user's search intention. In fact, the user may try to search by using information such as the title and creator of the multimedia, but in the case of video, movement, color information, color, shape information in the case of music, music speed, mood information, etc. You may be more interested. Content information can be directly input by multimedia producers or automatically extracted from the information to build a database for searching.

In the case of content-based multimedia search in the above-described content-based multimedia search, if a content query other than bibliographic information (title, player, composer, publication year, etc.) can be made, a search more suited to the search intention can be performed. In the case of music, after listening to some music, the image, the melody, and the speed of the song are memorized, which is easier to remember and longer than the bibliographic information of the music. Therefore, when searching for music, it is efficient to search for music using this content information. Content-based music search can be divided into four types according to the type of query.

1. Input music search of melody query

In this method, the user inputs the melody of the music through a microphone and finds the music containing the melody.

2. Music Style Search

This method is to find the music that corresponds to the style of music by inputting a sample of music of the desired style or by adjusting several variables that define the music style.

3. Fast search of songs

In this method, a user inputs a desired speed as a numerical value and searches for a corresponding speed music.

4. Sample input search

In this method, a stored music file is inputted or a music being played is input into a microphone to find a music in the database.

A music search method using a melody query input will be described in the music data search method. Among the various elements of music, melody can be easily remembered by the listener. Music heard through audio media is easy to remember the melody of an important part, if not the whole part. The search method using this characteristic is a melody query input music search method. The melody of the user's memory is input to the melody input from the music of the database by inputting the melody of the part memorized through the microphone or through an input device such as a keyboard. The music containing the corresponding part is returned as a search result.

The feature of this melody query input music search technique is that it targets a database where the melody characteristics of music data are manually indexed. That is, score information, pitch, length, and the like of the original music are manually indexed in advance. The humming query analyzes the signal through the microphone and automatically extracts and retrieves the pitch, length, and so on. Therefore, such melody matching becomes a problem of string matching such as text matching, and methods for efficiently searching for this problem have been published.

As a method for melody matching, in the case of string matching between a melody query represented by a symbol and music, a method of matching using the semantic information of a string used in a general text search is introduced, and the musical semantic information of the melody symbol string is introduced. There is a way to extract and search.

Alternative methods include string matching based on the MIDI format, a standard form of computer music representation, or calculating similarities between two MIDI files based on melody, melody strings, rhythm strings, and chord strings from MIDI files. There is a method of extracting and string matching. In addition, a method of expressing a MIDI file for melody matching may be introduced.

In addition, for the melody matching, a method of extracting a melody element from a sound input or a matching method used when matching a MIDI file and a humming has been introduced, and a method of expressing a melody as a difference from successive notes has been introduced. A method of expressing and matching is also introduced.

The existing melody search methods introduced above are methods for searching a database where musical information (pitch, length) has already been extracted. Since this is not a form of music signal that users usually create, transmit, acquire, and consume, the search requires manual indexing of musical information about all music. In this case, since manual indexing is impossible, it cannot be applied to applications such as music search and personal music library management through the Internet.

In the melody-based music data retrieval, a method for retrieving music data based on melody by automatically extracting musical information of a database to be searched from music, and automatically extracting musical information from a melody input query It is intended to provide.

The present invention is a method and a search apparatus for searching music stored in a database, the melody being a musical feature that can be easily remembered by a person without searching using textual information such as the title, singer, album name of the music, etc. It is an object of the present invention to provide a method and a search apparatus for searching for music including the melody by inputting a signal corresponding to the melody.

In particular, in the present invention, in the music search through the melody input, instead of targeting the music database where the music information has been manually extracted in advance, information corresponding to the music information is automatically extracted from the normal music clip and used for the search. Accordingly, an object of the present invention is to provide a method and a retrieval apparatus for melody-based music data retrieval without using other additional data, ie, manually indexed musical information, to music data commonly encountered by users.

The melody-based music retrieval method of the present invention for achieving the above object, by automatically extracting the melody feature of the music data from the music to form a melody feature database, and automatically extracts the melody feature of the music input by the query The music data is searched by comparing the similarity between the extracted query music melody feature and the melody feature of the melody feature database.

In addition, the melody-based music retrieval apparatus of the present invention for achieving the above object, the music data melody feature extraction means for extracting the melody feature from the music data to be searched, and the query melody to extract the melody feature of the data input by the query And feature similarity measuring means for measuring similarity between the quality melody feature and the melody feature of the music melody feature database.

1 is a diagram illustrating an embodiment of a melody-based music retrieval apparatus of the present invention, in which a melody feature database is automatically extracted from the music data of a music database, and a melody feature database is generated. In the case of an input, the melody feature is automatically extracted from the hum and the music is searched for the melody feature database using the extracted query melody feature information.

The melody-based music retrieval apparatus of FIG. 1 includes a music database 1 for storing music data to be searched for, a music data melody feature extracting unit 2 for extracting a melody feature from the music data of the music database; A melody feature database 3 storing the extracted music data melody feature information, a query input melody feature extractor 4 for automatically extracting a melody feature from the query input data, and a melody feature extracted from the music data And a searcher 5 for comparing the melody features extracted from the query melody and measuring the similarity and outputting a search result.

The music database 1 stores music data to be searched for. The melody feature information is extracted from the music data melody feature extracting unit 2 and the extracted melody feature information is stored in the melody feature database 3 in the music data provided from the music database 1. As the music data input by the query, feature information about the query melody is extracted by the query melody feature extractor 4. The search unit 5 compares the feature information extracted from the query melody with the feature information stored in the melody feature database 3 to calculate the similarity between the two and outputs the search results in the order of high similarity. Search for music data similar to the input data.

The music database 1 or the melody feature database 3 may be stored in a storage device including one computer storage device, a plurality of distributed and connected computer storage devices, a storage device added to a music player, and the like. The melody query input is input as a melody input by a user through a audio signal input means such as a microphone, a melody input by hum or a song, a melody input by selecting another music file, or a melody string through a keyboard or other symbol input means. Melody, a file in which a symbol string representing a melody is stored, and the like can be used.

The melody feature of the music data is automatically extracted from all types of compressed or uncompressed digital music data using a computer algorithm, a signal processing processor, etc. to build a melody feature database, and search for music. Here, the melody query can be represented as all types of digital music data, compressed or uncompressed, and the melody query is automatically converted to an audio signal by converting a melody query represented by a symbol into an audio signal to search for a melody feature database. can do. In addition, the feature database converts the music data represented by the symbol into an audio signal to automatically extract the melody feature, constructs a melody feature of the music data into a database, and searches for the same.

Meanwhile, the apparatus for extracting the music melody feature, that is, the music data melody feature extractor 2 and the query melody feature extractor, that is, the query melody feature extractor 4, have a spectrogram structure, partial sound enhancement, harmonic summation, Frame-by-frame note energy vector calculation, note boundary segmentation, note fragment construction means and / or process. Figure 2 shows such a melody feature extraction.

Referring to FIG. 2, the apparatus for extracting melody features according to the present invention comprises a first feature information extractor 6 for extracting melody feature information, and a sound (system) segmentation using information extracted by the first feature information extractor. a third feature for extracting final melody feature information using information extracted by the second feature information extractor 7 for performing segmentation, the first feature information extractor and the second feature information extractor It includes the information extraction section (8). As an input for extracting melody features, the sampling rate of 16 kHz, 8-bit signal value resolution, and short channel PCM audio format can reduce the amount of information of the original music signal without losing the audio signal characteristics required for music searching.

The first feature information extractor 6 extracts feature information for each frame, that is, a spectrogram component / or process 601, a partial sound improving unit / or process 602, and a harmonic adder / or process 603. And a note energy vector calculation unit / or process 604 for each frame, and the second feature information extractor 7 calculates / or processes energy for each frame using the partial sound enhancement information for segmentation. 701, an extraction unit / or process 702 of the minimum peak point is calculated from the calculated energy for each frame, and the third feature information extraction unit 8 extracts feature information for each segment, that is, for each frame. A calculation unit / or process 801 of a segment note energy vector using the note energy vector and the minimum peak point, and a component / or process 802 of the note fragment from the segment note energy vector.

A process of extracting melody feature information will be described with reference to FIG. 2 and FIGS. 3 to 8 to be described below. First, when the music data is extracted from the music CD and the music melody feature is automatically extracted, the audio format is converted and input into the feature extraction apparatus. Extract features by converting digital signals with 44 KHz sampling rate, stereo, and 16-bit resolution, a lossless audio format from music CDs, into 16 KHz, mono, and 8-bit do. This is a sampling rate that can range from C0 (16.532 Hz) notes to B8 (7902.1 Hz) notes, and 8 for B5 notes (987.7 Hz), the upper limit of the band at which notes considered to be melodies in music. Partial parts can be extracted.

When the query input is a human hum through the microphone, the query input is converted into the same format as that of the music data input to the melody feature extraction apparatus.

1. spectrogram construction

In order to extract the melody feature, spectrogram configuration is performed to interpret the frequency characteristics of the signal in time, and the spectrogram is constructed by the spectrogram component / or process 601 as shown in Equation 1 below. Fast Fourier Transform (FFT) is used. A sample sampled at a sampling rate of 16 kHz consists of a spectrogram with an FFT size of 1024 and frame overlap of 512. That is, the FFT size of the FFT of the spectrogram is 1024 samples, has 512 samples of overlap, and is processed by using a Hamming window before conversion.

Where T is the frame size of the audio clip and N is the FFT size (1024).

The energy spectrum of the signal thus converted is obtained as shown in Equation 2 below.

2. Partial enhancing

The partial sound enhancement is performed by the partial sound enhancement unit / or process 602 as a processing operation for extracting a predominant sound from a normal music clip composed of multiple sounds. . Music is composed of various elemental sounds, of which the most important sound is determined by the elemental sound with great energy and clear partials. In the case of music in which various sounds are mixed, one element sound is lost or clarity due to the partial sounds of other element sounds. Therefore, in order to extract a clear (clear) partial sound, partial sound enhancement processing is performed as shown in Equation 3 below.

delete

In Equation 3, the value of W can be determined in proportion to the size of the main lobe of the window applied to the signal before the FFT, and used as a value of 4 or 8. This process improves the partial sound by averaging the difference between the energy values of the current FFT index and the surrounding FFT index in order to extract the distinct partial sound. To determine the number of surrounding indexes to average with the size of the main lobe of the window applied to the signal before performing the FFT, consider the energy values of the surrounding eight indexes or the energy values of the surrounding 16 indexes. do.

In this process, even if the absolute energy of the partial sound is large, the partial sound that is relatively smaller than the surrounding energy has a smaller value. Even if the absolute energy of the partial sound is small, the partial sound that is relatively larger than the surrounding energy is the value. Will become large. That is, as shown in the example shown in FIG. 3, even if the absolute energy of the partial sound is large, the partial sound having a relatively large surrounding energy becomes larger when viewed on the improved partial sound spectrum. FIG. 3 (a) shows an energy spectrum distribution by frequency before partial sound improvement, and FIG. 3 (b) shows an energy spectrum distribution by frequency after partial sound improvement. As can be seen in FIG. 3, when the partial sound improvement 602 is performed, the partial sound relatively smaller than the surrounding energy is hard to be heard by the human ear, and the relatively large partial sound is easily heard by the human ear. This has the effect of enhancing the energy of the partial sound of the sound that is easily heard by a person. The result of the partial sound enhancement is provided for the harmonic adder / or process 603 at the next stage, while also for the frame-by-frame energy calculator / or process 701 for segmentation.

3. Harmonic sum

The harmonic summation is performed by the harmonic adder / or process 603 by inputting the improved partial sound as described above. In the harmonic summing process, the improved partial sound is summed at equal intervals in the frequency domain to extract pitch values for each frame. In addition, the summation value is normalized by the sum of the summed partial tones, and the summation consideration is set to less than half of the FFT size.

The most important factor in the recognition of sound is the harmonic nature of the sound. The sound produced by human utterance or instrumental performance causes partial sounds to appear at regular intervals in the frequency domain due to the characteristics of the generator. Sound recognition is the process of recognizing how harmonic these parts sound. It has been reported that the pitch extraction method by the harmonic summation is more successful than any other method. In the present invention, the harmonic summation is performed by using the improved partial sound as shown in Equation 4 below.

Here, [x] represents an integer not exceeding x. By performing such a harmonic summation process, the loudness corresponding to the fundamental frequency p can be known. 4 shows the effect of harmonic summing. As shown in FIG. 4, when the harmonic summation is performed based on the improved partial sound spectrum, the loudness corresponding to the fundamental frequency p, that is, the energy for each frequency can be known.

4. note energy calculation

The calculation of the note energy vector for each frame is to calculate a note energy vector for each frame by using the result of the harmonic summation, which is performed by the frame-by-frame note energy vector calculation unit / or process 604. In the calculation of the note energy vector for each frame, the harmonic sum is calculated for each standard scale band of music. That is, 108 standard scale bands are used to calculate note energy. The energy value at the note band boundary is interpolated using the energy value at the FFT index.

Since humans have some resolution in recognizing sound, the signal expressed by harmonic summation by fundamental frequency is expressed separately by band. Since music has a standard scale used in music, signals are represented by bands corresponding to the standard scale. The standard scale ranges from 16.532 Hz, which corresponds to C0, to 7902.1 Hz, which corresponds to B8. The upper and lower limits for each note are determined. Modern music is divided into twelve notes per octave, and the relationship between the notes expressed in the scale of this system is expressed by Equation 5 below.

In Equation 5, I is the difference between the notes expressed in the standard scale and R is the fundamental frequency ratio of the two notes. According to this equation, the ratio of two adjacent notes on the scale is 1.059463.

The band-specific energy for the harmonic summation is obtained by the standard scale and the band. This is called note energy, which is obtained as shown in Equation 6 below.

In Equation 6, M is 108, which represents the total magnitude of the standard scale, so the note energy of each frame is represented by a vector having 108 elements. When the sound is expressed by note energy, the minute time difference (vibration, etc.) of the sound can be expressed as a single sound, and the pitch shift process is simplified in the matching process. 5 shows the calculation process of note energy. Fig. 5A is a value interpolated using the harmonic summation value and the adjacent harmonic summation value in the FFT index, (b) shows an FFT index, and (c) shows a note index, respectively. As can be seen in FIG. 5, the sound is expressed by band-specific energy, that is, note energy, which is obtained using the periodic property of the audio signal, and based on the harmonic summation value as shown in FIG. As shown in c), the energy for notes C2, C # 2, D2, D # 2, E2, F2 was calculated.

The frame-specific note energy vector thus obtained is provided for calculation of the segment note energy vector.

5. Note segmentation

The note segmentation process, which is the segmentation process, is a process of grouping the frames of the same musical character into one segment. This process reduces the amount of data required for storage and matching by expressing the melody characteristics in segments. do. That is, the division of the note boundary is performed by dividing the note by selecting a point where the minimum point becomes the minimum point within a certain frame interval among the extracted minimum points of the energy of the frame of the improved partial sound as the boundary point of the segment. .

In order for a person or musical instrument to produce a single note, an elapsed sound is generated for a certain period of time, during which the energy value of the note is small. In order to obtain significant energy change in the mixed music, the energy is calculated using the improved partial tone spectrum. The energy obtained from this improved partial tone spectrum is changed in time. Among the minimum points of the frame-by-frame energy of the improved partial tone, the minimum point within a constant frame interval is set as the boundary point of the segment.

6 shows an example of a note division process. That is, the second feature information extractor 7 performs note division through the frame-specific energy calculator / or process 701 and the minimum peak point extractor / or process 702, which are improved through partial sound improvement. By inputting the partial sound, the frame energy of the improved partial sound is obtained, and among the obtained minimum points of the frame energy, the minimum points within the constant frame interval are set as the boundary points of the segments. Note splitting is performed in this manner, and information about the obtained boundary point of note splitting is provided for the segment note energy vector calculating unit / or process 801 of the third feature information extracting unit 8.

6. Note fragment construction

The note fragment configuration is a process of constructing a note fragment by representing segment note energy as an average of element value of frame note energy vectors included in one segment. The note fragment configuration is a process of obtaining a note energy vector representing a segment by averaging the note energy vectors within a note boundary point obtained in the note splitting process for each element and extracting a plurality of local maxima. In this case, the note energy vector representing the segment may be calculated by dividing each element value by an average of element values of the note energy vector. In addition, note fragments are constructed by extracting fewer maximum values in the case of query input than in the case of music data. For example, when the query input is a humming input by a user through a microphone, seven maximum values are extracted from the note energy vector of the music data, and three maximum values are extracted from the note energy vector of the humming query input.

The note fragment configuration is performed by the third feature information extraction unit 8. The note fragment configuration is segmented using the note splitting information obtained in the note segmentation process and the note energy vector for each frame obtained through the frame-specific note energy vector calculating unit / or process 604 of the second feature information extractor 6. In the calculation unit / or process 801 of each note energy vector, the average of the note energy vectors for each frame is obtained to obtain a segment note energy vector. From this, the note fragment configuration unit / or process 802 configures the note fragment.

The composition of the note fragment is obtained as shown in Equation 7 below.

Here, C value is the number of frames included in the segment S _l , l _s is the start frame index of the segment, l _e is the end frame index of the segment. Finally, the note energy vector is constructed by normalizing each element by the average of the element values of the vector.

In addition, in order to extract important notes from the segment, the value of the top 7 elements is extracted in the case of a music clip and the top 3 values in the case of a humming query input through a microphone according to the magnitude of the note energy.

In this way, the melody characteristic information was extracted. The melody feature information extracted through the melody feature information extractor / or process is for music data from a music database or for input query data.

Therefore, as shown in FIG. 1, the melody feature information extracted from the music database 1 is stored in the database 3, and the search unit 5 in the case of the melody feature information of the input query data. The search results are output through the similarity calculation of both.

7. Calculation of Similarity

The method of calculating similarity will be described. The similarity is calculated by considering the temporal difference between the music melody feature and the query melody feature, partial variation, and the overall pitch difference between the music and the query input.

When the query data is humming, the humming may be different from the music clip in overall length, partially different or different in length, and wrong sound information may be extracted in the process of pitch extraction. Therefore, we use DP Programming (Dynamic Programming Matching), which is a method of matching two signals that have different temporal lengths and partially change while allowing partial errors. In addition, since the overall pitch difference between the humming and the music clip, that is, the pitch shift occurs, the DP matching is performed while shifting the index of the segment note energy vector when calculating the similarity value of the matrix elements.

DP matching is performed as follows, see FIGS. 7 and 8.

If two time series R and Q are assumed and the lengths are NR and NQ, respectively, R and Q can be expressed as follows.

R = r ₀ , r ₁ , r ₂ , ..., r _i , ..., r _NR-1

Q = q ₀ , q ₁ , q ₂ , ..., q _j , ..., q _NQ-1

To match two series R, Q, an i-row × j-column matrix is formed. When the pitch shift is ps, the (i-th, j-th) elements of the matrix take on the similarity values d _ps (r _i , q _j ). Indicates. The similarity value is expressed as in Equation 8 below.

In Equation 8, the matrix elements (i, j) correspond to the matching of r _i and q _j respectively, and the matching path C _ps when the pitch shift is _ps is a set of continuous matrix elements that determine the matching of R and Q. Is defined. The kth element of the matching path C _ps when the pitch shift is ps is defined as c _{ps, k} = (i, j), thus representing the matching path C _ps as follows.

When there is a constant pitch shift ps, many matching paths C _ps may exist, but the path that minimizes the matching cost is selected as an optimal matching path as shown in Equation 9 below.

In Equation 9, K _ps of the denominator is intended to compensate for different matching paths having different lengths.

The similarity value between the music melody feature and the humming melody feature is determined as the minimum value of the minimum path cost calculated according to various pitch shift values as shown in Equation 10 below.

Here, the sum of the matching costs means the similarity. The smaller the matching cost, the smaller the difference between the melody characteristic information elements, and thus the similarity between the query data and the target music data is high.

Thus, by determining the similarity value between the music melody feature and the query melody, that is, the humming melody feature, and sorting the music data in the order of the highest similarity value, the user can obtain the desired music data search result.

In FIG. 7, the intersection points of the grids represent note fragments, the horizontal axis represents note fragments of query data, and the vertical axis represents note fragments of music data.

When determining the minimum path from the matched path candidate, a method of determining the minimum path from the 3-way matched path candidate or the 5-way matched path candidate may be performed.

The matching path at the time of determining the minimum path in the three-way matching path candidate is obtained by a matching equation as shown in Equation 11 below.

The matching path at the time of determining the minimum path in the 5-way matching path candidate is obtained by a matching equation as shown in Equation 12 below.

FIG. 8A shows an example of a three-way match path candidate, and FIG. 8B shows an example of a five-way match path candidate. FIG. 8 (a) shows matching path candidates in three directions from the current note fragment to adjacent note fragments in the horizontal, vertical, and diagonal directions. In FIG. We show the matching path candidates for five directions, including the diagonal direction (indicated by dashed lines) between the horizontal and diagonal lines.

The DP matching described so far may be applied to the similarity calculation in the present invention as follows.

First, in the similarity measurement device measuring the similarity in consideration of the temporal difference between the music melody feature and the query melody feature, partial variation, and the overall pitch difference between the music and the query input, the result of performing the melody feature extraction, In particular, DP matching is performed using the configured note fragment.

In addition, when using DP matching in measuring similarity, the elements of the DP matrix are calculated using the normalized Euclidean distance between the vector of the music melody feature and the query melody feature represented by the note energy vector representing the segment.

Here, when calculating the elements of the DP matrix, various minimum costs obtained by calculating the distance using the pitch shifted feature vector and performing DP matching based on the DP matrix calculated from various pitch shift values A pitch shift giving a minimum value in the matching path is determined, and the minimum value is used as the similarity between the music melody characteristic and the quality melody characteristic.

In the similarity calculation using the DP matching, when the user's humming through the microphone is used as a query input and DP matching is performed at various pitch shift values, the pitch of the music melody feature and the query input melody feature is calculated. The shift value can usually be retrieved by limiting the user to the maximum possible minimum pitch shift value.

In addition, in using the DP matching to measure the similarity, in order to obtain the minimum cost matching path to the current matching point, the minimum value of the three-way path or to obtain the minimum cost matching path to the current matching point DP matching can be used to find the minimum value of.

In addition, in using the DP matching in the similarity measurement, when the total minimum cost matching path is obtained, the cost may be normalized to the length of the cost path.

Alternatively, as described above, in obtaining the minimum cost matching path to the current matching point, a method of normalizing the candidate direction path cost to the matching length to the current matching point may be used.

Performance can be improved by various modifications to the DP matching method. In the present invention, the following modifications of DP matching are used.

That is, the matching speed may be improved by performing matching without matching the edges of the DP matrix using the matching window.

In addition, a similarity calculation method may be used that may reflect the degree to which the minimum cost matching path deviates from the diagonal matching path. To measure the extent to which the minimum cost matching path deviates from the diagonal matching path, the length of the minimum cost path is divided by the sum of the length of the music melody feature and the length of the query melody feature. The value for measuring the degree of deviation of the minimum cost matching path from the diagonal matching path may be used by adding to or multiplying the cost value of the minimum cost matching path calculated above.

On the other hand, a matching step pattern in five directions can be used to skip over and match elements with similarly misleading values. Alternatively, the vertical and horizontal matching step patterns do not appear repeatedly two or more times so as not to perform a registration that deviates greatly from the diagonal.

Melody based music search is performed by outputting a predetermined number or more as a search result starting from a song including the most similar segment using the similarity value obtained using the DP matching method described above.

The present invention does not target a music database in which musical information has been manually extracted in advance, and by automatically extracting information corresponding to the musical information from a normal music clip and using the same for searching, the music data commonly encountered by users. The present invention provides a method and a retrieval apparatus for melody-based music data retrieval without using other additional data, ie, manually indexed musical information.

1 is a block diagram of a melody-based music retrieval apparatus of the present invention.

2 is a view showing a melody feature extraction process in the present invention

3 is a view for explaining partial sound improvement in the present invention;

4 is a view for explaining the harmonic summation in the present invention;

5 is a view for explaining the note energy calculation in the present invention

6 is a diagram for explaining note division in the present invention.

7 is a view for explaining an example of the matching path in the present invention;

8 is a view for explaining a three-way / five-way matching path candidate in the present invention;

Claims (19)

Automatically extracting melody features of the music data from the music to construct a melody feature database ; Automatically extracting a melody characteristic of music input by a query ; Searching for music data by comparing the similarity between the extracted query music melody features and the melody features of the melody feature database ; In the melody-based music search method comprising a, The extracting of the melody feature may include configuring a spectrogram for data to extract feature information and improving a partial sound . The method of claim 1, wherein the extracting of the melody characteristic information further includes after the partial sound improving step, adding a harmonic sum, calculating a note energy vector for each frame, dividing a note boundary, and forming a note fragment. Melody-based music search method characterized by. The method of claim 1, wherein the melody query is input as a melody query input by a user by a hum or a song, a melody query input by selecting another music file, or a symbol string representing a melody through a keyboard or a symbol input means. Melody query, a melody-based music search method characterized in that the input is selected in the form of the input melody query by selecting a file that stores a symbol string representing the melody. The method of claim 1, wherein the extraction of the melody feature information includes a process of constructing a spectrogram of data to extract feature information, and when constructing the spectrogram, processing half of an audio frame size is superimposed over time using an FFT. Melody-based music search method characterized in that. The method of claim 1, wherein the extraction of the melody feature information includes a process of improving a partial sound of data for extracting feature information, and averaging the difference between energy values of the current FFT index and the surrounding FFT index to extract a distinctive partial sound. Melody-based music retrieval method characterized by improving the partial sound. 6. The method of claim 5, wherein when the partial sound is improved by averaging the difference between energy values of the current FFT index and the surrounding FFT index to extract the distinct partial sound, the average of the main lobe of the window applied before the FFT is performed. Melody-based music retrieval method characterized in that it determines the number of index to be around . 2. The method of claim 1, wherein the extraction of the melody feature information includes a harmonic summing process of data for extracting feature information, and for harmonic summing, an improved partial sound is summed at equal intervals in the frequency domain for each harmonic summation. Melody-based music retrieval method characterized in that the value is extracted, the summation value is normalized by the sum of the summed partial notes, or the subject to be summed is set smaller than half of the FFT size. The method of claim 1, wherein the extracting of the melody feature information includes calculating a note energy vector for each frame of data from which feature information is to be extracted, and calculating a note energy vector by calculating a harmonic sum for each standard scale band of music. Melody-based music search method characterized by. The method of claim 8, wherein the energy value at the note band boundary is interpolated using the energy value in the FFT index to calculate the note energy. The method of claim 1, wherein the extraction of the melody feature information includes a process of dividing a note boundary of data for extracting feature information, and for dividing the note boundary, a constant among the minimum points of the energy of each frame of the improved partial sound is fixed. Melody-based music retrieval method characterized in that for selecting the minimum point in the frame interval as the boundary point of the segment. The method of claim 1, wherein the extraction of the melody feature information comprises a process of constructing a note fragment of data to extract feature information, and for constructing the note fragment, averaging note energy vectors within a note boundary point for each element and extracting a plurality of local maximum values. Melody-based music retrieval method characterized by obtaining a note energy vector representing a segment. The method of claim 11, wherein when calculating a note energy vector representing a segment to construct the note fragment, the note energy vector is calculated by dividing each element value by an average of the element values of the note energy vector, or the note fragment is constructed. Melody-based music retrieval method characterized in that the extraction of fewer maximum values for the query input than the music data. The method of claim 1, wherein the similarity calculation between the melody feature and the query melody feature of the music data is performed by measuring similarity in consideration of temporal differences, partial variations, and overall pitch differences between music and query inputs. Melody-based music retrieval method characterized in that through the DP matching using a note fragment configured as the target melody characteristic information. 15. The method of claim 13, wherein the DP matching for measuring similarity is performed by calculating elements of a DP matrix using a normalized Euclidean distance between a music melody feature expressed as a segmental note energy vector and a vector of query melody feature. Melody-based music search method characterized by. 15. The method of claim 13, wherein the DP matching for measuring similarity is calculated by normalizing the cost to the length of the cost path when obtaining the total minimum cost matching path, or the candidate direction path cost to the matching length to the current matching point. Melody-based music retrieval method characterized in that the normalized calculation. The method of claim 13, wherein the similarity is calculated by reflecting the degree of deviation of the minimum cost matching path from the diagonal matching path when measuring the similarity, wherein the length of the minimum cost path is determined as the degree of deviation of the minimum cost matching path from the diagonal matching path. The sum of the length of the melody feature and the length of the query melody feature is used, and the value that measures how far the minimum cost matching path deviates from the diagonal matching path is added to or multiplied by the cost value of the minimum cost matching path calculated above. Melody based music search method characterized in that the use Music data melody feature extraction means for extracting melody features of music data to be searched from a music database; In the melody-based music search apparatus comprising a similarity measuring means for measuring the similarity of the melody characteristics of the base , In extracting the melody feature of the music data and the query melody feature, the melody-based music search comprises extracting feature information through a process of constructing a spectrogram for the data from which feature information is to be extracted and a partial sound enhancement process Device. 18. The apparatus of claim 17, further comprising a melody feature database for storing music melody feature information extracted by the music data melody feature extracting means, wherein the melody feature database is a computer storage device or a plurality of distributed and connected melody features. Melody-based music retrieval device comprising a computer storage device, a storage device including a storage device added to the music playback device, etc. to automatically extract the melody features of the music data to build a feature database. 18. The apparatus of claim 17, wherein the music melody feature extracting means and the query melody feature extracting means comprise: spectrogram constructing means for constructing a spectrogram of an input sound, partial sound improving means for performing partial sound enhancement from the spectrogram, Harmonic summation means for performing harmonic summation from an improved partial sound, note energy vector calculating means for calculating a note energy vector per frame using the summed harmonic summation information, and dividing a note boundary using the improved partial sound; And a note fragment constructing means for constructing a note fragment using the note segmentation means and the note boundary information and the frame-specific note energy vector.