JP2000155759A - Retrieval device and storage device, and retrieving method and storing method for music information, and storage medium where programs thereof are recorded - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make the retrieval speed faster than that for an entire search by using indexes of feature quantities extracted from music information and to comply with a variety of retrieval requests by performing similar retrieval based upon feature quantities. SOLUTION: In the conversion and storage phase of a retrieval object, a conversion part 3 converts source data of music to music information, divides the music information into pieces of partial data, and extracts feature quantities from the pieces of partial data. A storage part 4 stores the extracted feature quantities of the retrieval object in a data base with indexes. In the retrieval phase of music information, the conversion part to music information 3 similarly extracts feature quantities from the source data of music as retrieval conditions and a music information retrieval part 5 retrieves similar source data from the data base in the storage part 4 by making use of indexes. The result is outputted to a display and sound source device 6.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an apparatus for searching for music information based on source data of a sound source from a humming or keyboard, and an apparatus for storing music information used for music information search.

[0002]

(References)

・ Tomoya Sonoda, Masataka Goto, Yoichi Muraoka: "Song Search System by Singing Voice", IPSJ 55th Annual Convention, 1J-6, 199
7.・ Tetsuya Kageyama, Younori Takashima: "Melody Search Based on Humming Singing", IEICE Transactions, Vol.J77-
D-II No.8 pp.1543-1551, 1994.

Further, as shown in, for example, Japanese Patent Application Laid-Open No. 07-121556 (music information retrieval apparatus), a musical score is regarded as a code (character), and is regarded as a character string and replaced with a conventionally known character string search. There is a music information search method depending on a search method.

FIG. 10 is a diagram for explaining DP matching. DP matching is a technique that uses dynamic programming to match patterns of different lengths. When the character strings to be collated are A = a1, a2,..., Ai,..., Am B = b1, b2,. Is represented by a path (a polygonal line) connecting grid points on a two-dimensional plane with the rectangular coordinates. With this path, a path that matches both character strings A and B is determined. as a result,
The one with the longest matching partial character string length has a similarity. In the example of FIG. 10, in the comparison of the pitch strings A and B, a path connecting the grid points of a, b, c, d, and a shown by circles is selected.

[0005]

In the above-mentioned conventional method, pattern matching is performed from the beginning to the end of the musical score data of the input data and the musical score data stored in the database. there were. (1) In the conventional search, the search speed is slow because the search does not use an index (exhaustive search). (2) In the conventional search, since the search is not performed using the characteristics of music information (such as those corresponding to the vocal height and modulation), various search requests (for example, “I want to select a song with a high vocal” , "I want to select a song that has been transposed")
I can't respond to

[0006] The present invention solves the above-mentioned problem. By using an index of a feature amount extracted from music information, the search speed can be made faster than the exhaustive search.
In addition, by performing similarity search using multiple features,
An object is to be able to respond to various search requests.

[0007]

SUMMARY OF THE INVENTION In order to solve the above-mentioned problems, the present invention comprises a means for converting source data into music information, a means for searching from a database, and a means for outputting a search result. It is characterized by using.

The means for converting to music information converts the source data of the music into music information that can be analyzed by a computer, and divides the entire music information into a plurality of partial data. Next, the music data is converted into music information for each frequency component by using a numerical conversion process so that a musical feature can be extracted from the partial data. Then, a feature amount is extracted from the music information for each frequency component and stored in a database. Also, an index (multidimensional index) is created from the accumulated feature amounts. Here, since the feature amount is a multidimensional vector, for example, an index such as R-tree convenient for searching a multidimensional vector space is created.

The difference from the prior art is that music information is divided into partial data, and a plurality of musical features such as "vocal height" are extracted from the music information.

[0010] The means for searching from the database compares the feature quantity extracted from the music information equivalent to the input key with the feature quantity stored in the database. At this time, a similarity calculation is performed by using a distance function to make a short distance a high similarity (similarity search). Also,
Search using the aforementioned index.

The difference from the prior art is that a search based on similarity is performed, and that a feature index is used.

[0012] The result display means makes the searched music available as a search result to a searcher using a display or a speaker.

The operation of the present invention is as follows. By converting the music information into music information capable of extracting musical features from the music by the means for converting to music information,
Source data (CD, score, live, etc.) from various media can be uniformly searched. Further, it is possible to extract a plurality of musical feature amounts. By combining many musical features, it becomes possible to respond to various search requests. By dividing the entire music information into partial data and creating an index, it is not necessary to perform an entire search.

By using an index (multi-dimensional index) based on feature amounts extracted from music information by means of searching from the database,
It is possible to search at high speed. Further, music similar to the input data can be searched by the similarity search using the distance function, and it is possible to respond to various inquiries.

The result display means makes it possible to visually or audibly notify the searcher of the search result.

A program for realizing each of the above processing means by a computer can be stored in an appropriate recording medium such as a computer-readable portable medium memory, a semiconductor memory, and a hard disk.

[0017]

FIG. 1 is a block diagram showing a configuration example of the present invention. The music input device 1 is a microphone for inputting humming, singing voice, etc., an input device for music data such as a musical score, a reproducing device such as a tape or a compact disk (CD), or a device for inputting music source data such as a music keyboard. is there. The same applies to the search condition input device 2, and the music input device 1 and the search condition input device 2 may be the same device.

The music information conversion unit 3 is a means for converting source data into music information, extracting a plurality of feature amounts from the music information, and storing the extracted feature amounts in a database of the storage unit 4. This music information conversion unit 3 differs from the prior art in that it divides the data into partial data, extracts feature amounts, and creates an index.

The storage unit 4 is a database having information on feature amounts obtained from music information to be searched and an index (multidimensional index) for the information.

The music information search unit 5 searches the database of the storage unit 4 for similar source data using the feature amount extracted from the music information of the source data serving as a search condition by the music information conversion unit 3. Means. The difference from the conventional music information search is that a similarity search that measures similarity by distance calculation such as a Euclidean distance or a Manhattan distance is performed instead of DP matching, and that an index is used.

The display / sound source device 6 is a device for outputting a result searched by the music information search unit 5, and is a display for displaying a song title or a score, or a sound source device such as MIDI and a speaker.

Here, the music source data means music data, humming, singing voice, data input from a music keyboard, etc., musical scores, and one or more of these. The music information that can be analyzed by the computer is information that represents the source data itself in a data format that can be easily processed by the computer, such as numerical data obtained by sampling music data. In particular, in the present embodiment, no matter what kind of input source data such as analog signals, MIDI data, and image data such as musical scores, etc., the music information conversion unit 3 internally unifies the source data. Is converted into music information in the format specified in.

Hereinafter, specific examples of each of the above sections will be described in more detail.

[Conversion Phase to Music Information] FIG. 2 is a flowchart of a conversion phase to music information. First, source data is input from the music input device 1 (step S10). Next, the source data input from the music input device 1 is converted into a monaural 8k
Sampling is performed at a low rate near Hz and digitized (step S11). Next, the numerical data, which is the entire music information that has been digitized, is divided into a plurality of partial data (step S12).

The numerical data is divided into partial data of several tens of seconds while shifting the numerical data by several seconds so as to be overlapped. FIG. 3 shows an example of division into partial data. The division of the music information is made into a sliding window as shown in FIG.

Next, an FFT with a window is applied to the divided partial data.
Is applied. The window uses a Hanning window, a Hamming window, or the like. By using this window, the joint portion due to the periodicity of the FFT is prevented from becoming discontinuous. In the FFT, a finite number (for example, 512 pieces) of numerical data is shifted little by little until data is exhausted, applied to each of them, and new music information for extracting a feature amount is created. FIG. 4A shows an example of input data signals for partial data, and FIG. 4B shows an example of music information obtained by performing FFT on the input data of FIG. 4A.

Next, the data is collected at logarithmic intervals, and a characteristic feature amount is extracted as a vector from a sound range (around 50 Hz to 20,000 Hz) only in a range where humans can hear. The extracted feature amount vector is stored in the storage unit 4, and an index is created as described later (step S13).

Here, as an example of the feature amount vector,
(1) the overall sound pressure component, (2) the frequency of the sound pressure peak of the frequency component, (3) the frequency of the sound pressure peak of the frequency component obtained by normalizing the frequency into a certain section,
(4) There is a time series difference of the high sound pressure frequency obtained from the above (3).

When the entire sound pressure component of (1) is used as a feature quantity vector, the number of data collected for each frequency is converted into a histogram for each unit, and is used as a several-dimensional vector. FIG.
FIG. 4 shows an example in which the entire sound pressure component is used as an n-dimensional feature amount vector. In FIG. 5, the first dimension is from 0 Hz to 100 Hz.
, Which corresponds to the area of the hatched portion in the figure.

In the case of (2), the number of data having the maximum value in the music information is converted into a histogram for each unit, and is set as a several-dimensional vector. FIG. 6 shows an example of the position of the peak of the sound pressure, which is used as the feature amount here. FIG. 6A shows peaks in the music information of music stored in the database, and FIG. 6B shows peaks obtained from music information of the search key. By matching the data, it can be determined that the music information is similar. The histogram is created for each unit to which the above-described FFT is performed in each partial data, and is created, for example, by counting the number of maximum peaks in each section obtained by dividing the frequency at logarithmic intervals.

In the case of (3), the music information is folded (collectively) in a certain section at a frequency, and the maximum number of data in the section is converted into a histogram for each unit to make a vector of several dimensions.

For example, when the music information is folded between 100 Hz and 200 Hz, when the music information a is larger than 200 (Hz): a = a ÷ 2 (repeated until the music information a becomes smaller than 200) When the music information a is smaller than 100 (Hz) : A = a × 2 (repeated until 100 or more)

In the case of (4), the difference between the data folded in a certain section at the frequency of the above (3) and the next data is grouped for each difference, and the number of the data is made into a histogram to be a several-dimensional vector. . FIG. 7 shows an example. As shown in FIG. 7, if the frequency of the focused data is 50 Hz and the next time (for example, after 2 seconds) transitions (moves) to 55 Hz, the difference (5) is assigned to the section to which the n-dimensional vector applies. (Count) and add (count). This section is a section in which the difference value is divided at logarithmic intervals. For example, in a section from 100 Hz to 200 Hz, the maximum value of the difference is 100, which is 0 to 2, 2 to 4, 4 to 8,
, A histogram is created for each section divided by a logarithmic interval, and this is used as an n-dimensional feature amount vector.

As described above, each feature amount is a multidimensional vector. The similarity calculation using this multidimensional vector uses the Euclidean distance and the Manhattan distance.
The converted feature amount is stored in the database of the storage unit 4.

Further, a separate index is created for each feature based on the features extracted from the main music information. The present invention can be implemented using any type of index. For example, if an index such as R-tree is used, it is suitable for searching a multidimensional vector space. In this method, vectors of feature amounts in a multidimensional space are grouped with similar ones, and this is repeated to create a tree structure and create an index. By tracing the hierarchical structure, access can be limited to only data that needs to be evaluated, so that the search response speed can be maintained even if the number of data increases. The following references are related to the R-tree. [References] A. Guttman, ^ R-tree: a dynamic index s
tructures for spatialsearching ", Proceedings of th
e ACM SIGMOD International Conference on the Manag
ement of Data, pp.47-57, Boston, 1984. [Retrieval phase of music information from database] Figure 8
9 is a flowchart of a search phase of music information from a database. First, key data (source data) serving as a search condition is input from the search condition input device 2 (step S20). This input data is converted into music information by the music information conversion unit 3, and a feature amount is extracted (step S21). Next, the music information search unit 5 compares the feature amount obtained from the key data with the feature amount stored in the database of the storage unit 4 by a similarity search using an index, and searches for source data (step). S2
2). The search result is displayed as a result on the display / sound source device 6 or reproduced using the sound source device.

In the similarity search in the music information search unit 5, the similarity search is performed by calculating the distance between the feature amount of the input data to be searched and the feature amount of the partial data stored in the database.
Here, the overall similarity is calculated by linearly combining the features of the music information with the intermediate results obtained from the indexes for each feature, weighted for each feature, and the similarity search is performed based on this. I do. At the time of retrieval, the retrieval is performed using the index created based on the feature amount.

FIG. 9 is a diagram for explaining an example of the similarity calculation method according to the present embodiment. As shown in FIG. 9A, various feature amounts are extracted from music information obtained from input keys, and are compared with feature amounts in a database extracted from search target source data. The feature amounts of the source data in the database are distributed in a multidimensional space as shown in FIG. 9 (C), and an index obtained by clustering these into a tree structure is used as shown in FIG. 9 (B). Have been created. Therefore, when searching for data (features) in the database, narrowing down is performed using this index, the distances of the data (features) there are compared, and the results are returned in order from the closest one.

In the similarity search using a plurality of feature amounts as described above, the similarity calculation is performed by assigning a weight according to the user's search purpose to each feature amount, thereby making it possible to search for a music desired by the user. It can be easily realized. For example, for a search request such as “I want to select a song with high vocal”, the weight of the feature amount described in (2) described above as an example of the feature amount vector is increased, that is, the value of the feature amount is multiplied. This can be dealt with by increasing the weight value. In addition, in response to a search request for a tune that has been modulated, examples of the feature vectors (3) and (4) have been described.
This can be handled by increasing the weight of the feature amount of.

[0039]

As described above, according to the present invention, even if a searcher does not have special musical knowledge or skills, music can be obtained from a database at high speed by music from a humming sound source, a keyboard, or the like. Can be searched. Further, it is possible to respond to various inquiries (e.g., a modulated song) by the similarity search.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of the present invention.

FIG. 2 is a flowchart of a conversion phase into music information.

FIG. 3 is a diagram illustrating an example of division of music information into partial data.

FIG. 4 is a diagram illustrating an example in which FFT is performed on divided partial data.

FIG. 5 is a diagram illustrating an example in which an entire sound pressure component is used as an n-dimensional feature amount vector.

FIG. 6 is a diagram showing an example of a peak position of a sound pressure as a feature amount.

FIG. 7 is a diagram illustrating generation of an n-dimensional feature amount vector focusing on data transition.

FIG. 8 is a flowchart of a search phase of music information from a database.

FIG. 9 is a diagram for describing an example of a similarity calculation method according to the present embodiment.

FIG. 10 is a diagram for explaining conventional DP matching.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Music input device 2 Search condition input device 3 Conversion part to music information 4 Storage part 5 Music information search part 6 Display / sound source device

 ──────────────────────────────────────────────────続 き Continued on the front page (72) Inventor Seiichi Konya 3-19-2 Nishi Shinjuku, Shinjuku-ku, Tokyo Japan Telegraph and Telephone Corporation (72) Inventor Nobuo Taniguchi 3-19 Nishishinjuku, Shinjuku-ku, Tokyo No. 2 Nippon Telegraph and Telephone Corporation (72) Inventor Masashi Yamamuro 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo F-Term within Nippon Telegraph and Telephone Corporation 5B075 ND14 NK43 NR03 NR06 PP07 PQ02 PQ04 PQ32 PR06 QM08 5D077 CA03 DC16 DC31 EA21 HC25 5D378 AG04 BB02 BB09 BB11 BB21 KK01 KK07 QQ06 QQ08 QQ24 QQ25 TT22 TT26 XX43

Claims (10)

[Claims] In a search device for searching for music, music source data is converted into music information that can be analyzed by a computer, the music information is divided into a plurality of partial data, and a feature amount is extracted from the partial data. Conversion means, storage means for storing, as a database, feature amounts extracted by the conversion means into music information for source data of music to be searched, and source data of music to be search conditions to the music information. Music information retrieval means for retrieving similar source data from the database of the storage means using the feature amount extracted by the conversion means of
A music information search device comprising: a search result output unit that outputs a search result. 2. The music information search device according to claim 1, wherein the music information search means is configured to determine a distance between a feature extracted from music information of input data designated as a search condition and the stored feature. A music information search apparatus characterized in that a similarity is measured by performing a calculation, thereby searching for source data similar to the input data. 3. The music information search device according to claim 1, wherein the converting unit for converting the music information into the music information includes a feature amount extracted from the music information when storing the feature amount in the database. An index is created, and the music information search means uses the index to search for similar source data from the database based on the feature quantity extracted from the music information of the source data. Information retrieval device. 4. The music information search device according to claim 3, wherein the index is a multidimensional index based on one or more feature amounts. 5. A music information storage device used in a search device for searching for music, converts source data of a music to be searched into music information that can be analyzed by a computer, and divides the music information into a plurality of partial data. A feature is extracted from the partial data and stored in a database, and the feature is converted into music information for creating an index from the feature, and the source data of a music to be searched is extracted by the above-described music information converting means. A music information storage device comprising: a storage unit configured to store the obtained feature amount as an indexed database. 6. A retrieval device for retrieving music, wherein a characteristic amount extracted by dividing the music information obtained by converting the source data of the music to be searched into partial data is extracted as an indexed database. A storage means for storing, and converting the source data of the music as a search condition into music information which can be analyzed by a computer, converting the music information into a plurality of partial data and extracting the characteristic amount from the partial data into music information Means, music information search means for searching for similar source data from the database using the index based on feature amounts extracted from music information of the source data, and search result output for outputting search results Music information retrieval apparatus comprising: 7. A method for accumulating music information used in a search device for searching for music, converting source data of the music to be searched into music information that can be analyzed by a computer, and dividing the music information into a plurality of partial data. And extracting a feature from the partial data, and storing the extracted feature in a database and creating an index from the feature. 8. A music piece obtained by converting source data of a music piece to be searched for by dividing the music information into partial data and extracting a feature amount as an indexed database using a storage means for storing the feature quantity. A method of converting source data of a song serving as a search condition into music information that can be analyzed by a computer, dividing the music information into a plurality of partial data, and extracting a feature amount from the partial data; A step of searching for similar source data from the database using the index based on a feature amount extracted from music information of the source data, and a step of outputting a search result. Music information search method. 9. A recording medium storing a program for storing music information used in a search device for searching for music, wherein source data of the music to be searched is converted into music information which can be analyzed by a computer, and A program for causing a computer to execute a process of dividing music information into a plurality of partial data and extracting a feature amount from the partial data, a process of storing the extracted feature amount in a database and creating an index from the feature amount A recording medium on which a music information storage program is recorded. 10. A music piece obtained by converting source data of a music piece to be searched for by dividing the music information into partial data and extracting a feature amount as an indexed database using a storage means. A recording medium on which a program for retrieving music is recorded, in which source data of a music piece serving as a search condition is converted into music information that can be analyzed by a computer, and the music information is divided into a plurality of partial data, and the characteristic data is obtained from the partial data A process of extracting a quantity, a process of searching for similar source data from the database using the index based on the feature extracted from the music information of the source data, and a process of outputting a search result. A program for causing a computer to execute the program, wherein a music information search program is recorded.