KR20090083972A - Method for building music database for music search, method and apparatus for searching music based on humming query - Google Patents

Abstract

A method and an apparatus for improving searching speed of searching music by minimizing an access of database are provided to guarantee the accuracy of search when uncertain query such as humming is inputted. A music file is inputted(110). A feature data showing a melody is extracted(120). The extracted feature data as described above is transformed to the character string(130). The transformed character string as described above is stored as a first index in which it becomes the basis of the search(140). The character string showing the meaningful melody from the transformed character string as described above is extracted. The extracted character string as described above is stored as the based second index of search(150).

Description

Method for building music database for music search, method and apparatus for searching music based on humming query}

The present invention relates to a music search, and more particularly, to a music database construction method for music search, and a method and apparatus for searching music by inputting a humming query.

Currently, researches to find effective feature data according to query characteristics in content-based music information retrieval system are actively conducted. It is divided into those for processing exact queries such as original music with the exact height and length of notes and those for processing incorrect queries such as hum.

In the system proposed in [1], feature data commonly used to express melodies is the string 'UDS', which uses the heights of two adjacent notes, so that when the next note is higher than the previous note, the 'U (Up)', If it is low, it is expressed as 'D (Down)' and if it is equal to 'S (Same)'. However, this data is not efficient in terms of accuracy because it only uses a curve of pitch and ignores the length of pitch. Another humming query processing system is the search using the tempo of music [2]. However, since the tempo data does not have much feature information, the accuracy is inferior when the incorrect humming is processed.

Another research activity related to music information retrieval involves efficient index structures. This research is active in order to reduce the long search time and the large amount of comparative calculations caused by the sequential search of large music databases. To solve this problem, there is N-gram as a typical index method [3]. This method treats N notes as a unit and indexes them by moving them to the right one by one. This is similar to the sliding window of size N. Due to this index design method, N-gram has a large index size and poor performance in terms of search cost. Also, to improve the search speed, the feature data in numeric form is converted into a string, but this is a character string that does not take into account the characteristics of the incorrect melody Humming. As the query is incorrect, the accuracy drops sharply. Another indexing method is the use of a spatial access method [4]. This method uses height and length information of a note to express it as a point in two-dimensional space, and then index the location information of consecutive points. Representative method is to use R-Tree, which means that the search speed is fast but the selectivity is not good because many musics appear near the query in the minimum bounding rectangle including the query. Have.

The content-based indexing technique used in music information retrieval is a method of indexing a specific melody of music to reduce the search speed. Since the performance of the music search system depends on which part of the melody is indexed in a piece of music, it is important to select a meaningful melody that can improve performance. Typically, there is a method of improving the search speed by indexing the beginning of music [5]. This exploited the possibility that the user would frequently query the beginning of the music. However, this method does not guarantee fast search speed unless the user queries the beginning of the music. It also failed to provide sufficient evidence that the user frequently asked the beginning of the music. Another content-based indexing method is the study of indexing cyclically repeated melodies [6]. However, this method has the disadvantage that the repetitive melody extraction process is too complicated and many candidate frequent melodies are generated because the melody is searched without a certain unit, and the index size becomes large. In contrast to the previous two methods, there is a method of indexing a user's query without indexing music stored in a database [7]. In this method, when a user's query melody is input, it is converted into a 'UDS' string and the queries are compared with each other to index the frequently asked melody. This has the advantage of quickly searching for music that is frequently queried by the user. However, if a melody that is not frequently queried is input, the database has to be accessed directly. In addition, it shows low performance in terms of accuracy because the 'UDS' string method using only the curve of the pitch is not considered.

<References>

[1] N. Kosugi, Y. Nishihara, T. Sakata, M. Yamamuro and K. Kushima, "A Practical Query-By-Humming System for a Large Music Database", In Proc. ACM Multimedia, pp. 333-342, 2000.

[2] E. Scheirer, "Tempo and Beat Analysis of Acoustic Musical Signals", Acoustic Society of America, 103: 1 January, pp. 588-601, 1998.

[3] A. Pienimaki, "Indexing Music Databases Using Automatic Extraction of Frequent Phrases", In Proc. 3rd International Symposium on Music Information Retrieval (ISMIR), pp. 25-30, 2002.

[4] J. Reiss, J. Aucouturier and M. Sandler, "Efficient multidimensional searching routines for music information retrieval", In Proc. 2nd International Symposium on Music Information Retrieval (ISMIR) ,, pp. 163-171, 2001.

[5] S. Doraisamy, S. Ruger, "Robust polyphonic music retrieval with n-grams", Journal of Intelligent Information Systems, 21: 1, pp. 53-70, 2002.

[6] C. Liu, J. Hsu and A. L. P. Chen, "Efficient Theme and Non-trivial Repeating Pattern Discovering in Music Databases". In Proc. 15th International Conference on Data Engineering (ICDE '99), pp. 14-21, 1999.

[7] Seung-Min Roh, Dong-Moon Park, In-Jun Hwang, "Efficient Audio Indexing Technique Using User Query Pattern", Journal of Korean Information Science Society, Vol. 31, No. 1, pp.143-153, 2004.

The technical problem to be achieved by the present invention is to guarantee the accuracy of the search even if an incorrect query, such as humming, and to minimize the access to the database, music search, which can improve the speed of searching for music to find the user The present invention provides a method for building a music database.

Another technical problem to be solved by the present invention is to secure music by inputting a humming query, which can guarantee the accuracy of a search even when an incorrect query such as a humming is input, and improve the speed of the search by minimizing access to a database. To provide a method and apparatus for searching.

According to an aspect of the present invention, there is provided a music database construction method for music search, comprising: (a) receiving a music file and extracting feature data representing a melody; (b) converting the extracted feature data into a character string; (c) storing the converted string as a first index on which a search is based; And (d) extracting a character string representing a meaningful melody from the converted character string and storing the character string representing a meaningful melody as a second index which is a basis of a search.

Here, the feature data preferably includes an amount of change in height and a rate of change in length between consecutive notes. At this time, in the step (b), it is preferable to convert the feature data into a character string by matching the combination of the change amount of the height and the change rate of the length to a character according to a predetermined rule. In the step (b), the two-dimensional space having the change amount of the height and the change rate of the length as each axis is divided into a plurality of zones, and the combination of the change amount of the height and the change rate of the length belonging to the same zone is the same. It is preferable to correspond to letters.

In the step (c) and the step (d), the storage of the character string is preferably stored in the form of a suffix tree.

In addition, the step (d), (d1) is a character string representing the meaningful melody, and extracts a character string representing a frequent melody that is a melody frequently generated in one music to the 2-1 index that is the basis of the search Storing; Or (d2) extracting a string representing a melody divided by a comma of a predetermined length or more as a string representing the meaningful melody and storing the extracted string as a 2-2 index on which the search is based.

In this case, the step (d1) may include: (d11) analyzing a string pattern in units of measure of music, and obtaining a list indicative of the sequence of strings, measure positions, the number of occurrences, and the number of occurrences of the units of measure; (d12) generating groups including each node having the highest rank of the number of repetitions as a representative node; For each of the created groups, (d13) comparing the number of occurrences of nodes adjacent to the representative node with a predetermined first threshold value and joining the adjacent nodes to the group according to the result; And (d14) comparing the accumulated value of the rank of the occurrence number of nodes combined with the rank of the number of occurrences of the representative node with a predetermined second threshold value, and repeating or terminating the step (d13) according to the result. ; And (d15) extracting a character string included in each group obtained as a result of performing step (d13) and step (d14) as a character string representing the frequent melody.

In addition, when there is a string overlapping each of the obtained groups, the method may further include extracting a string included in the group in which the corresponding groups are combined as a string representing the frequent melody.

In order to solve the other technical problem, a method for searching for music by inputting a humming query according to the present invention includes: (a) receiving a humming input as a query; (b) extracting feature data representing a melody from the received humming and converting the extracted feature data into a character string; (c) storing, according to the converted character string, feature data representing melodies of various kinds of music, converted into a character string, and storing a first index stored in the character string and a second index extracted and stored from the converted character string. And searching for music using the music database.

Here, the step (b), the step of extracting the height and length of the notes constituting the received humming; And extracting, as the feature data, data including an amount of change in height and a rate of change in length from the height and the length of the extracted note. And converting the extracted feature data into a character string.

In addition, the second index is a character string representing the meaningful melody, and represents a melody divided by a boundary of a 2-1 index or a comma of a predetermined length or more, which is stored by extracting a character string indicating a frequent melody that is a frequently generated melody. It is preferable to include the 2-2 index stored by extracting the character string.

In addition, the step (d) may include searching for music from the 2-1 index and the 2-2 index according to the converted character string; And retrieving music from the first index according to the converted character string.

In addition, the music database further stores feature data representing melodies of the various pieces of music, and in step (d), music is stored from the feature data stored in the music database according to the feature data extracted in step (a). The method may further include searching.

In order to solve the another technical problem, an apparatus for searching for music by inputting a humming query according to the present invention includes receiving a humming input, extracting feature data representing a melody, and converting the extracted feature data into a character string. A query preprocessor; A music database storing, as an index for the search, a first index in which feature data representing melodies of various music is converted into a character string, and a second index in which a character string representing a meaningful melody is extracted from the converted character string; And a music searching unit searching for music from the music database according to the character string converted from the humming.

Here, the query preprocessor extracts the height and length of the notes constituting the received humming, and includes data including a change rate of the height and a change rate of the length between consecutive notes from the height and length of the extracted note. It is preferable to extract as.

In addition, the second index is a character string representing the meaningful melody, and represents a melody divided by a boundary of a 2-1 index or a comma of a predetermined length or more, which is stored by extracting a character string representing a frequent melody that is a frequently generated melody. It is preferable to include the 2-2 index stored by extracting the character string.

Further, it is preferable that the music searching unit first searches music from the 2-1 index and the 2-2 index, and then searches music from the first index.

The music database may further store feature data representing melodies of the various music pieces, and the music search unit may search for music from the first index, and then search for music from the first index according to the feature data extracted from the query preprocessor. Music can be retrieved from feature data stored in a database.

According to the present invention described above, even if an incorrect query such as a humming is input, the accuracy of the search can be guaranteed, and the search speed for searching for music to be searched by the user can be improved by minimizing the access of the database.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description and the accompanying drawings, the substantially identical components are represented by the same reference numerals, and thus redundant description will be omitted. In addition, in the following description of the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

1 is a flowchart of a music database construction method for music search according to an embodiment of the present invention, and FIG. 2 is a structural diagram of a music database according to an embodiment of the present invention.

First, in step 110 a music file is input. In this case, it is preferable to receive a music file in the form of MIDI as a music file.

Next, in step 120, feature data representing a melody is extracted from the input music file. In order to search for music using the user's query, it is necessary to compare the melody stored in the music database and the user's query melody to search for the desired music. Therefore, it is necessary to extract feature data representing the melody from the music file. . In the present embodiment, as the characteristic data representing the melody, the amount of change in height and the rate of change in length between successive notes or rests are used. The humming entered by the user is an incorrect melody, not the exact height and length of the note. In other words, unlike the original music, the user may sing the entire pitch higher or lower, and may sing faster or slower than the original music. In consideration of these features, in order to increase the accuracy of music retrieval, the change rate of the height and the change rate of the length between consecutive notes are used as feature data.

In order to extract the amount of change in height and length between successive notes or rests, first the height and length of the note, the length of the rest and the node boundary are extracted. For note height information, MIDI music uses Octave as a predefined value. Octave notation consists of alphabetic scales and numbers, with values ranging from 0 to 127. For example, a note with an octave 'F5' has a height value of '65'. In this embodiment, the numerical value is used to express the height of the note. Next, we extract the length of the note and the comma. In MIDI music, the length of a note is calculated based on the quarter note, or time base. However, because the length of one beat may vary between MIDI songs, the same note may have a different length for each song. Therefore, it is necessary to convert one beat length of all music to the same value. In this way, we extract the height and length of every note. Also, at this stage, the boundaries of nodes are extracted, and the reason for extracting them is to use nodes as units having a certain length when trying to find frequent melodies that occur frequently in a music. The boundary of a node can be calculated from the total time of the input MIDI music, that is, the time signature of several minutes, the length of one beat, and the length of notes and rests.

The change in height between notes means the difference between the height of the next note and the height of the previous note between two adjacent notes. In other words, it is an accurate numerical representation of how much the height of the note is increased, decreased or the same. The rate of change of length is calculated by dividing the length of the next note by the length of the previous note between two adjacent notes.

3 is a conceptual diagram illustrating a method for extracting a change rate of a height and a change rate of a length between consecutive notes according to an embodiment of the present invention. Referring to FIG. 3, feature data in the form of a two-dimensional vector having a change amount of height and a change rate of length between two adjacent notes is extracted, and this value is extracted as information on the preceding note. For example, if two adjacent notes are called N _i and N _{i + 1} , the height of each note is P _i , P _{i + 1} , and the length of each note is D _i , D _{i + 1} , the height of the characteristic data of N _i . Equation for extracting Pitch Interval and Duration Ratio of is as follows.

4 is a conceptual diagram illustrating a method of extracting a change rate of a height and a change rate of a length when a comma exists between consecutive notes. In the conventional research, the comma is mostly ignored, but in the present embodiment, the comma information is included in the feature data in consideration of the case where the user includes the comma. In general, the position of the comma should be treated as important because the user is more likely to query a melody that starts after a comma longer than a certain length. Commas, unlike notes, contain only length information. Therefore, as shown in FIG. 4, the height of the comma is stored as a null value and expressed as '#'. Next, the amount of change in height and the rate of change in length are calculated to extract the comma's characteristic data. At this time, depending on the position of the comma can be divided into two cases. The first case is a comma followed by a note. At this time, the feature data to be calculated is the data of the previous note, and the value is stored as '0.00', which means that the change in height is unchanged as shown in FIG. The information of the comma is included in the information of the preceding note by calculating the change rate of '0.5' using the length. If instead of a comma followed by notes with length and time signatures of 67 and 30, respectively, the characteristic data value of (0.00, 0.5) will be calculated as in the above case. This may be a problem that the information of the comma is unreliable, but if the user hums, the user may not keep the comma correctly and may recognize it as a note and continue to play the note. . Next, as shown in FIG. 4, if a note appears after the comma, the feature data to be calculated at this time becomes the data of the comma, which is ignored. The reason is that the neglected rate of change of length is a silent melody that does not express the melody. Through this method, all the feature data in the form of a continuous two-dimensional vector in one MIDI music are generated and stored in the feature data storage 210 shown in FIG. 2.

Next, in step 130, the feature data extracted in step 120 is converted into a character string. As described above, the feature data extracted in step 120 is a two-dimensional vector having a change rate of height and a change rate of length between consecutive notes, that is, a combination of a change rate of height and a change rate of length. In this embodiment, the feature data is converted into a character string by associating each combination of the change amount of height and the change rate of length with alphabet characters according to a predetermined rule.

In case of searching music by input of user's query, if searching the database storing feature data in the form of 2D vector, all the stored music should be compared and searched sequentially. At this time, the 2D vector stored in the database is a numeric value, and the comparison method with the query for retrieval generally uses the Euclidean distance calculation method. Therefore, using this search method, the search speed becomes slower as the size of the database increases. To compensate for this, the present embodiment uses an index stored by converting feature data into a string form. When the feature data is converted to a string, a music search can be performed using a string matching algorithm, and the computational complexity is much smaller than comparing two-dimensional vectors in the form of numbers.

FIG. 5 is a conceptual diagram illustrating a rule for converting a two-dimensional vector value having a change amount of height and a change rate of a length into character values according to an embodiment of the present invention. As shown, a character conversion criterion is applied by dividing a two-dimensional space having a change amount of height and a change rate of length into a plurality of sections, for example, 35 sections, and assigning each section to one character. Thus, similar two-dimensional vector values are replaced by the same letter. This reflects the characteristic that the humming is inaccurate, and the converted character is a value that allows the error of the incorrect humming. Therefore, by applying these character conversion rules, people without singing ability or people with language disabilities can effectively search for music.

Referring to FIG. 5, the horizontal axis represents the amount of change in height and the vertical axis represents the rate of change in length. On the right side of the horizontal axis, the amount of change in height is positive. This means that the height of the back note is higher than the height of the front note between two adjacent notes, and the difference becomes larger as it goes to the right. On the contrary, the change in height on the left side shows a negative value based on the zero value of the horizontal axis. This means that the height of the back note is lower than the height of the front note, and the difference increases gradually toward the left. Similarly, on the vertical axis, the upper part means that the length of the last note is longer than the length of the previous note, and the ratio difference of the length becomes larger as the upward direction increases. On the other hand, the lower part of the vertical axis means that the length of the last note is shorter than the length of the previous note, and the difference in length becomes larger as it goes down. Referring to FIG. 5, the amount of change in height between -4 and +4 is converted to the same character, which means that the error is allowed even if the user sings 4 or higher than the height of the actual sound when the user sings one note of the actual music. It means. Likewise, between 0.5 and 2 in the rate of change of the beat is converted to the same character. This means that when a user sings a note of the actual music, the error is called twice as long or as 0.5 times as long as the actual length.

FIG. 6 illustrates a result of converting a sequence of successive two-dimensional vector values into a string using the conversion criteria shown in FIG. 5.

Referring back to FIG. 1, in operation 140, the string converted in operation 130 is stored as the first index 220. In this case, it is preferable to store the data in the form of a suffix tree. The suffix tree is a data structure that can reveal the internal structure of a string and is mainly applied to an exact matching algorithm.

Next, in step 150, a string representing a meaningful melody is extracted from the string converted in step 130 and stored as a second index. A music consists of several melodies, which are the best expressions of musical expressions and human emotions, meaning a combination of notes and rests containing various pitch and length information. A meaningful melody refers to a melody that can represent the music among the melodies that make up a music, that is, a melody that a user is likely to inquire.

As the first meaningful melody in this embodiment, a melody with an arbitrary length that appears several times in a music is used. This is because it is easy to remember melody that appears frequently when the user listens to music. That is, in order to search for the music desired by the user, many melodies that can be easily memorized are often humming in the melodies of the music. In this embodiment, melody frequently generated is designated as a meaningful melody.

As the second meaningful melody in this embodiment, the melody between the commas of a predetermined length or more is used. Long commas longer than a certain length can be seen as boundaries that divide the melody. Analyzing the actual music data shows that the flow of melody is divided by long commas. In most music, long rests appear before the beginning, before the chorus, and before the beginning of verse 2. That is, the melody divided by a comma of a predetermined length or more is designated as a meaningful melody by applying a meaning of a comma on the assumption that the chorus part and the first part of the music will be easy for the user to remember.

First, a process of extracting a string representing frequent melody, which is the first meaningful melody, will be described. 7 is a flowchart illustrating a method of extracting frequent melodies according to an embodiment of the present invention.

In step 710, the string pattern is analyzed in units of nodes. In this embodiment, first, a string pattern is analyzed in units of nodes in order to find frequent melodies that are melodies repeated several times in a music. The reason for extracting frequent melodies using a node is that the node is a physical and objective unit having the same time signature, and thus it is possible to extract the frequent melody based on this and obtain a frequent melody accurately and concisely.

8 is a conceptual diagram illustrating a process of obtaining information by analyzing a string pattern in units of nodes. As shown in FIG. 8, in order to perform string pattern analysis, one piece of music represented by a string form is divided by a node boundary '/'. Through string pattern matching algorithm, strings divided by measure are analyzed by matching strings. Sequence in bar, node location information (hereafter, BarID), number of occurrences of each string, that is, The number (hereafter C) and the rank (hereafter R) sorting C in descending order are obtained. For example, if you receive music with the title 'Long time since', divide the string into units of measure and analyze the string pattern of units of measure, you can get the list as shown in Table 1 below. Referring to Table 1, it can be seen that the strings <j, x, k, j, i, x> appear at the 18th, 22, 35, and 39th nodes, and the number of occurrences C is 4. Also, this string is the most frequent occurrence in the above music, so the rank R is 1.

Sequence in Bar Bar Position (BarId) Count (C) Rank (R) <j, x, k, j, i, x> 18, 22, 35, 39 4 One <k, k, j, x, g> 19, 36 2 2 <j, x, j, k, k, k> 20, 37 2 2 <k, j, j, w, g> 21, 38 2 2 … … … … <j, x, g, k, E, x, i, x> 17, 49 One 3

Referring to FIG. 7 again, in step 720, a group including a node having the highest number of occurrences, that is, nodes having R = 1 as a representative bar (KB) is created. Here, the number of groups is C. FIG. 10 is a diagram illustrating a result of generating a group based on a list shown in Table 1. FIG. Referring to FIG. 10, since the C value is 4, it can be seen that four groups are generated. In FIG. 10, a vertical line means one piece of music, and each scale means one piece. In addition, the black scale indicates the representative node of each group, which is the starting node for frequent melody generation. As shown in FIG. 7, the 18th bar is the 18th bar, the second bar is the bar 22, the third bar is Bar35, and the fourth bar is Bar39.

In this embodiment, two factors are defined to extract frequent melodies. The first factor is a selection threshold for determining which nodes are frequently included in the melody. This is called a bar selection threshold (BST). If the BST value is 2, it means that frequent melodies are generated using only nodes having an occurrence count C value of 2 or more. The smaller the BST value, the longer the length of the frequent melody. Therefore, the second factor is a threshold for limiting the length of the frequent melody, which will be referred to as a frequent melody length threshold (hereinafter referred to as FMLT). FMLT creates frequent melodies, limiting the size of the variable (BSum) that is calculated each time a node is added. As will be described later, the BSum is calculated by calculating the difference between the R value of the node combined with the R value of the representative node and adding it to the previously calculated BSum. FMLT is a value for eliminating the upper limit of the BSum. In this embodiment, the melody is repeatedly generated until the BSum is less than or equal to the FMLT.

In step 730, by using the list obtained as a result of analyzing the string pattern in step 720, the group information generated in step 730, the BST value, and the FMLT value, the adjacent nodes are combined for each group based on the representative node. As you go out, you will often create melodies.

FIG. 9 is a flowchart illustrating step 730 of FIG. 7 in more detail.

First, in step 910, the symbol c representing each group (1 ≦ c ≦ C) is set to c = 1. This means that the steps described below are started in the first group of the generated groups.

In the following description, both nodes adjacent to the representative node will be referred to as KB _right and KB _left , respectively.

In step 915, BSum = 0 is set as an initial value of BSum. In step 920, if BSum is greater than FMLT, the process proceeds to step 950, otherwise, the process proceeds to step 925.

In step 925, if there is a node whose C value is greater than or equal to the BST value by comparing the C value of the current node KB _right and KB _left , and performing any one of steps 930 to 940 described later, otherwise, step 950 Proceed to

If there is one node whose C value is greater than or equal to the BST value, that is, one of KB _right and KB _left , in step 930, the corresponding node is added to the group.

If there are two nodes whose C values are greater than or equal to the BST value and each C value is different, in step 935, the nodes having the large C values are added to the group.

If there are two nodes whose C value is greater than or equal to the BST value and each C value is the same, both nodes are added to the group in step 940.

In step 945, the difference between the R value of the current node and the R value of the KB node added in any one of steps 930, 935, and 940 is added to BSum. In step 920, the process repeats steps 925 to 945 according to the comparison result of the BSum value and the FMLT value. If repeated, perform the steps above with the _right node, KB _{right + 1} , if the right node of the current node merged, or KB _left- , the next node to the left of the current node, if the left node of the current node is merged. _{View 1} as the current node and perform the above steps.

FIG. 11 is a diagram illustrating a process of generating frequent melodies for the first group of FIG. 10 based on a list shown in Table 1 in a situation where BST values and FMLT values are given as 2 and 6, respectively. First of all, the first group, KB, is designated as Bar18. Both nodes of the representative node, Bar17 and Bar19, are assigned KB _left and KB _right , respectively. Since the C value of Bar17 is 1, which is smaller than the BST value, the condition is not satisfied and is ignored. In other words, Bar17 cannot be combined with Bar18, which is representative because it is not a node that appears many times. On the other hand, the C value of Bar19 is 2, which is more than the BST value, so it is combined with Bar18. If the nodes are combined then the BSum of the group is calculated. The combined R value of Bar19 is 2, which is 1 when the difference from the R value of the representative bar Bar18 is obtained. This value is added to the existing BSum value and stored in the BSum again. Next, the above process is repeated by comparing the C value of Bar20 with the BST value. At this time, the left direction is neglected to generate melody frequently because the C value of Bar17 is not more than BST value. Increasing the number of nodes in the right direction will generate frequent melodies until BSum does not exceed FMLT, or until a value of C less than BST appears.

In order to calculate the BSum, the value of C, which is the number of nodes, may be used, but in the present embodiment, the rank R value is used instead of the C value. The reason for this is as follows. For example, consider two pieces of music with the sort order shown in Table 2 below.

Music 1 Music 2 Sequence in Bar BarId C R Sequence in Bar BarId C R c, c, a, d, d 6, 9, 20 3 One a, e, e, s, d 4, 9, 15, 27, 30, 34 6 One c, d, a, a, d 7, 21 2 2 c, e, s, a 5, 16, 28 3 2 a, d 8, 22 2 2 a, g, c, d 6, 10, 31 3 2 a, a, c, s 10, 23 2 2 e, s, c, c, d, s 3, 7, 11 3 2 b, e, h, z 11, 30 2 2 c, e, a 8, 12 3 2 b, b, k, u 12, 31 2 2 c, c, d 13, 14 3 2 c, c, a 13, 32 2 2 a, d, b, b 29, 31 3 2 ... ... ... ... ... ... ... ...

Let's assume that we use C instead of R to compute the BSum, and then apply frequent melody generation algorithms. If the BST value is set to 2 and the FMLT value is set to 6, the first group of music 1 will generate a frequent melody consisting of 8 bars from the 6th bar to the 13th bar. This is to create a frequent melody by combining the frequent nodes until BSum does not exceed FMLT, taking into consideration the most frequently occurring nodes in music 1. In addition, it can be seen that the same frequent melody is generated by calculating the BSum using the rank R value. On the other hand, if you generate the first group of frequent melodies in Music 2 using the C value to calculate BSum, you can get a frequent melody consisting of three nodes from the fourth to the sixth node. However, the seventh node was not treated as a frequent melody, although it was a frequent one in music 2. In other words, because it does not take into account the frequently occurring nodes as possible, inaccurate frequent melody is generated. On the other hand, if a frequent melody is generated using the R value, a frequent melody consisting of eight nodes from the third node to the tenth node can be generated. This may be referred to as a result of generating frequent melody in consideration of the most frequently occurring nodes in music 2. Since these two pieces of music have different characteristics, the number of frequent nodes appearing in each piece of music may be different. The most frequent number of nodes in music 1 is 3, while in music 2 it is 6. However, the rank R of these number of nodes is equal to one. In other words, in order to generate frequent melodies of music having different number of measures, the rank is applied to express their number of numbers as an objective number, and the melody is generated by calculating BSum using this value.

Referring again to FIG. 9, if the BSum becomes larger than the FMLT value, or there is no node whose C value is greater than or equal to the BST value in both adjacent nodes of the representative node or in the adjacent node of the combined node, frequent melody generation should be terminated for the group. do. In this case, as described above, the process proceeds to step 950 and c is increased by 1, and in step 955, the value of c and the number of groups C is compared. Steps 915 to 945 are performed on the group. If c is greater than the number of groups C in step 955, this means that steps 915 to 945 are performed for all groups. Then, in step 960, the strings included in each of the C groups are extracted as strings representing frequent melodies.

As described above, when the frequent melody generation is completed in all groups, a frequent melody consisting of nodes frequently generated in each group may be generated. However, as shown in FIG. 12, a situation in which frequent melodies belonging to two groups may overlap. At this time, if you store the frequent melody in each group separately, a lot of duplicate data will occur. Therefore, to prevent this, overlapping melodies can be treated as one melody by combining overlapping groups into one group. Therefore, if there is an overlapping string between each group, FIG. 9 may further include extracting a string included in a group in which the corresponding groups are combined into a string representing frequent melody. FIG. 13 illustrates a result of extracting a string representing frequent melodies finally based on the list shown in Table 1 according to this embodiment. The first and second groups were merged into one group because of the significant overlap, and the third and fourth groups were likewise merged into one group. Therefore, two groups are finally created, and the strings included in these groups are treated as strings representing frequent melodies.

The string representing the frequent melody generated through the above process is stored in the 2-1 index 230 shown in FIG. Here, the form of storing the character string is preferably stored in the form of a suffix tree, similarly to the first index 220.

Now, a process of extracting a string representing a melody divided by a boundary of a comma of a predetermined length or more, which is a second meaningful melody, will be described. To this end, in step 120 of FIG. 1, that is, in the process of extracting the feature data representing the melody, comma position information of a predetermined length or more is separately stored. And the music is divided by the position to create a melody divided into multiple commas. The comma of the predetermined length or more is a portion in which the continuous flow of the melody is broken, and the length of the melody divided by the comma as a boundary may vary depending on the value defined as the predetermined length. For example, if you divide a melody based on one beat, that is, a four-minute comma, divide the melody around where the comma that is longer than one beat appears.

14 shows the result of dividing a short melody by one or more beats. In the process of extracting feature data, a beat of a comma is used to generate feature data of the preceding note, and only the positional information is stored while the feature data of the comma is ignored. After converting it to character form, the melody is divided by the saved comma position. As shown in the figure, it can be seen that the strings [x, w, x] and [x] have been generated on the basis of the quarter comma.

The string representing the frequent melody generated through the above-described process is stored in the 2-2 index 240 shown in FIG. Here, the form of storing the character string is preferably stored in the form of a suffix tree, similarly to the first index 220 and the second-first index 230.

FIG. 15 is a conceptual diagram illustrating a process of extracting and storing a string representing all melodies, a string representing frequent melodies, and a string representing melodies divided based on a comma according to an embodiment of the present invention. As shown, first, a string representing the entire music is stored as a first index in the form of a suffix tree, and a string representing a frequent melody is extracted from the string representing the entire music, and stored as a 2-1 index. The melody divided based on the comma is extracted from the string representing the whole and stored as the 2-2 index. Using a suffix tree enables users to search from any part of the melody stored in the suffix tree. When music is searched through a music database constructed as shown in FIG. 15 with a humming query as an input, first, the 2-1 index and the 2-2 index are first searched, and then the first index is searched. This can reduce the time it takes for the user to search for the desired music.

16 is a block diagram of an apparatus for searching for music by inputting a humming query according to an embodiment of the present invention. Referring to FIG. 16, the music search apparatus according to the present embodiment includes a query preprocessor 1610, a searcher 1620, a display 1630, and a music database 1640.

The music database 1640 stores data as shown in FIG. 2 according to the above-described music database construction method. That is, the music database 1640 may include a feature data storage 210 that stores feature data representing melodies of various types of music, a first index 220 in which feature data is converted into a character string, and a frequent melody and a predetermined melody as a meaningful melody. It includes a 2-1 index 230 and a 2-2 index 240 for storing the melody divided by the comma or more boundaries than the length, respectively.

The query preprocessor 1610 receives a humming from the user as a query, extracts feature data representing a melody, and performs a query preprocessing process of converting the extracted feature data into a string. The query received is a humming humming through the microphone. The humming data has a wave form, unlike MIDI, which is a form input for the above-described music database 1640. Therefore, the query preprocessor 1610 first extracts the height and length of the sound and the height of the comma from the wave-shaped humming data like the information contained in the MIDI form. Then, the feature data is extracted from the extracted information in the same form as feature data of various music stored in the feature data storage 210. That is, the amount of change in the height of the note or the rest and the rate of change in the length are calculated. Next, the query preprocessor 1610 converts the feature data of the humming into a string form. At this time, of course, in the construction of the feature database, the same criteria as that of the feature data of various musics are converted into character strings are used.

The search unit 1620 may include the first index 220, the second-first index 220, and the second-second index of various kinds of music stored in the music database 1640 with the string extracted from the humming in the query preprocessor 1610. The music is retrieved from the index 240 or the feature data. In this case, the search unit 1620 may use an Exact Matching Algorithm as a method of comparing strings when searching for music from each index stored in a suffix tree form.

In addition, the display 1630 displays the search result of the searcher 1620 so that the user can know. At this time, for example, the song name, singer, and the like of the found music can be displayed as a search result.

17 is a flowchart illustrating a method for searching for music by inputting a humming query according to an embodiment of the present invention. The music searching method according to the present embodiment is performed in the music searching apparatus shown in FIG.

The music retrieval apparatus receives a humming as a query from the user (step 1710).

The query preprocessor 1610 performs the query preprocessing process as described above with respect to the received humming (step 1715).

The search unit 1620 searches for music from the 2-1 index according to the humming data converted through the query preprocessing process (step 1720), and searches music from the 2-2 index in parallel (step 1725). .

If the search is successful (step 1730), the search result is output via the display 1630 (step 1735). The user may stop the search through a user interface (not shown) provided in the music search apparatus when the desired music is searched based on the search result output. Therefore, when a search stop command is input from the user (step 1780), the search ends.

If the search fails in step 1730 or if the search stop is not requested from the user even though the search result is displayed, the search unit 1620 searches for music from the first index according to the converted humming data through the query preprocessing step (step 1740). ).

If the search succeeds (at step 1745), the search result is output via the display 1630 (step 1750). Similar to the above-described search from the 2-1 and 2-2 indexes, the user may look at the output search result and stop the search if the desired music is searched. Accordingly, if the search stop command is input from the user even during the search from the first index (step 1780), the search ends.

If the search result does not come out even from the first index or the search stop is not requested from the user, the search unit 1620 has the feature data in the form of numbers in which the humming data is changed in the query preprocessor 1610, and the music database 1640. The music is retrieved from the feature data stored in step 1755. Here, as a method of comparing the feature data of the humming and the feature data stored in the music database 1640, a method of obtaining music of a close distance by using the Euclidean distance function may be used. If the search is successful (step 1760), the search result is output through the display 1630 (step 1770). If the search fails, a message indicating the search failure is output through the display 1630 (step 1765). If a search stop command is input from the user even during the search from the feature data (step 1780), the search is terminated.

As described above, according to the present embodiment, first, a search is performed from a 2-1 index and a 2-2 index in which a character string representing frequent melody and a melody in a comma unit is stored (first step search). Therefore, if the user hums a melody frequently, or hums a melody starting after a predetermined length or more, the search is completed in the first stage search, and the search result can be delivered to the user in the shortest time. If the user does not obtain the desired search result in the first step search, the search is performed from the first index in which the character string representing the entire music is stored (second step search). By the second stage search, the search can be completed even if the user does not query the melody starting frequently or after the comma. Of course, the search result desired by the user may not come out even in the second step search, and finally, music is searched from the feature data stored in the form of numbers (third step search). Since the feature data in the form of numbers is the data that most accurately represents music, the most accurate search is performed in the third stage search. As described above, according to the present exemplary embodiment, a first stage search is first performed based on a query most likely to be input by the user. In addition, the first stage search and the second stage search which are relatively fast in search are performed first, and the third stage search is performed later. Therefore, it is possible to minimize the time required to search for music desired by the user, and to minimize access to feature data that takes a long time.

Meanwhile, the above-described embodiments of the present invention can be written as a program that can be executed in a computer, and can be implemented in a general-purpose digital computer that operates the program using a computer-readable recording medium. The computer-readable recording medium may be a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.), an optical reading medium (for example, a CD-ROM, DVD, etc.) and a carrier wave (for example, the Internet). Storage medium).

So far I looked at the center of the preferred embodiment for the present invention. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

1 is a flowchart illustrating a music database construction method for music search according to an embodiment of the present invention.

2 is a structural diagram of a music database according to an embodiment of the present invention.

3 is a conceptual diagram illustrating a method for extracting a change rate of a height and a change rate of a length between consecutive notes according to an embodiment of the present invention.

4 is a conceptual diagram illustrating a method of extracting a change rate of a height and a change rate of a length when a comma exists between consecutive notes.

FIG. 5 is a conceptual diagram illustrating a rule for converting a two-dimensional vector value having a change amount of height and a change rate of a length into character values according to an embodiment of the present invention.

FIG. 6 illustrates a result of converting a sequence of successive two-dimensional vector values into a string using the conversion criteria shown in FIG. 5.

7 is a flowchart illustrating a method of extracting frequent melodies according to an embodiment of the present invention.

8 is a conceptual diagram illustrating a process of obtaining information by analyzing a string pattern in units of nodes.

FIG. 9 is a flowchart illustrating step 730 of FIG. 7 in more detail.

FIG. 10 is a diagram illustrating a result of generating a group based on a list shown in Table 1. FIG.

FIG. 11 is a diagram illustrating a process of generating an empty melody for the first group of FIG. 10 based on the list shown in Table 1. FIG.

12 is a conceptual diagram illustrating a situation in which frequent melodies belonging to two groups overlap.

FIG. 13 illustrates a result of extracting a string representing frequent melodies finally based on the list shown in Table 1. FIG.

14 shows the result of dividing a short melody by one or more beats.

15 is a conceptual diagram illustrating a process of extracting and storing a string representing all melodies, a string representing frequent melodies, and a string representing melodies divided based on commas.

16 is a block diagram of an apparatus for searching for music by inputting a humming query according to an embodiment of the present invention.

17 is a flowchart illustrating a method for searching for music by inputting a humming query according to an embodiment of the present invention.

Claims (18)

In the music database construction method for music search, (a) receiving a music file and extracting feature data representing a melody; (b) converting the extracted feature data into a character string; (c) storing the converted string as a first index on which a search is based; And and (d) extracting a character string representing a meaningful melody from the converted character string and storing the character string representing a meaningful melody as a second index as a basis of a search. The method of claim 1, And the feature data includes a change rate of a height and a change rate of a length between consecutive notes. The method of claim 2, In the step (b), converting the feature data into a character string by associating a combination of the change amount of the height and the change rate of the length with a character according to a predetermined rule. The method of claim 3, In the step (b), the two-dimensional space having the change amount of the height and the change rate of the length as each axis is divided into a plurality of zones, and the combination of the change amount of the height and the change rate of the length belonging to the same zone is assigned to the same letter. And a music database construction method for matching. The method of claim 1, In the steps (c) and (d), the storage of the character string is stored in the form of a suffix tree. The method of claim 1, wherein step (d) (d1) extracting a string representing a frequent melody, which is a melody frequently generated in one piece of music, as a string representing the meaningful melody, and storing the string as a 2-1 index which is a search base; or (d2) a character string representing the meaningful melody, and extracting a character string representing a melody divided by a comma of a predetermined length or more into a boundary and storing the string as a 2-2 index which is the basis of the search. How to build a database. The method of claim 6, Step (d1), (d11) analyzing the character string pattern in units of measure of music, and obtaining a list indicative of the character string in units of units, the position of nodes, the number of occurrences, and the rank of the number of occurrences; (d12) generating groups including each node having the highest rank of the number of repetitions as a representative node; For each of the created groups, (d13) comparing the number of occurrences of nodes adjacent to the representative node with a predetermined first threshold value and joining the adjacent nodes to the group according to the result; And (d14) comparing the accumulated value of the rank of the occurrence number of nodes combined with the rank of the number of occurrences of the representative node with a predetermined second threshold value, and repeating or terminating the step (d13) according to the result. ; And (d15) extracting a character string included in each group obtained as a result of performing steps (d13) and (d14) as a character string representing the frequent melody. The music database of claim 7, further comprising extracting a character string included in the group in which the corresponding groups are combined as a character string representing the frequent melody when there is an overlapping character string between the obtained groups. How to build. In a method for searching music by inputting a humming query, (a) receiving a humming as a query; (b) extracting feature data representing a melody from the received humming and converting the extracted feature data into a character string; (c) storing, according to the converted character string, feature data representing melodies of various kinds of music, converted into a character string, and storing a first index stored in the character string and a second index extracted and stored from the converted character string. Searching for music using a music database. The method of claim 10, wherein step (b) comprises: Extracting a height and a length of a musical note constituting the received humming; And Extracting, as the feature data, data including an amount of change in height and a rate of change in length from the height and the length of the extracted note; And And converting the extracted feature data into a character string. The method of claim 10, The second index is a string representing the meaningful melody, and a string representing a melody divided by a boundary of a 2-1 index or a comma of a predetermined length or more, which is stored by extracting a string representing a frequent melody that is a frequently generated melody. And extracting the stored 2-2 index. The method of claim 11, In step (d), Searching for music from the 2-1 index and the 2-2 index according to the converted character string; And Retrieving music from the first index according to the converted character string. The method of claim 12, The music database further stores characteristic data representing melodies of the various music, The step (d) further comprises the step of searching for music from the feature data stored in the music database according to the feature data extracted in the step (a). In the device for searching music by inputting a humming query, A query preprocessor configured to receive a humming input, extract feature data representing a melody, and convert the extracted feature data into a character string; A music database storing, as an index for the search, a first index in which feature data representing melodies of various music is converted into a character string, and a second index in which a character string representing a meaningful melody is extracted from the converted character string; And And a music search unit for searching for music from the music database according to the string converted from the humming. 15. The method of claim 14, wherein the query preprocessor extracts a height and a length of a note constituting the received humming, and includes data including a rate of change of a height and a change rate of a length between consecutive notes from the height and length of the extracted note. Extracting the data as the feature data. The method of claim 14, The second index is a character string representing a meaningful melody, and is a string representing a melody divided by a boundary of a 2-1 index or a comma of a predetermined length or more, which is extracted by extracting a character string indicating a frequent melody that is a frequently generated melody. And a second-2 index extracted and stored. The method of claim 16, And the music searching unit first searches music from the 2-1 index and the 2-2 index, and then searches music from the first index. The method of claim 17, The music database further stores characteristic data representing melodies of the various music, And the music retrieval unit retrieves music from the feature data stored in the music database according to the feature data extracted by the query preprocessor after retrieving music from the first index.

Images