EP0944033B1

EP0944033B1 - Melody retrieval system and method

Info

Publication number: EP0944033B1
Application number: EP99302171A
Authority: EP
Inventors: Tomonari Sonoda
Original assignee: Individual
Current assignee: Individual
Priority date: 1998-03-19
Filing date: 1999-03-19
Publication date: 2003-05-28
Anticipated expiration: 2019-03-19
Also published as: DE69908226T2; DE69908226D1; US6121530A; EP0944033A1

Description

The present invention relates to a melody retrieval system and method, and more particularly to a WWW-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes from a music database.

One of the problems in building a melody retrieval system, where a melody is input by a user singing, is that the input melody can include various errors which are caused by the uncertainty of the user's memory or by the user's singing ability. Those errors are usually tolerated by using several thresholds and converting the input melody into approximate relative-value sequences of pitch change (the pitch difference between adjacent notes) and of span change (the inter-onset interval ratio of adjacent notes).

For instance, an approximate relative-pitch value is expressed by using three characters {U(p), D(own), and E(qual)}, and the sequence "do-re-mi-mi-re-do" is converted into "X-U-U-E-D-D-" (wherein "X" indicates the first note which does not have a relative value). These approximate sequences are then utilized in the matching process which compares the input with all songs in a database.

Prior melody retrieval systems [Kageyama et al. 1993; Ghias et al. 1995] utilized static heuristic thresholds to obtain these approximate sequences. Since it is difficult to determine the optimum thresholds for all songs, those systems did not take advantage of the span information which can potentially be useful, and mainly used pitch information as a search clue.

It was therefore difficult to improve the matching accuracy by using only the pitch information.

It was also difficult to reduce the number of answer candidates, because several songs in a large database tended to have the same patterns of the approximate relative-values of pitch and span sequences.

Moreover, the public use of music databases over computer networks such as the World Wide Web (WWW) has not been considered.

In conventional music retrieval systems, the pitch information is mainly used as a search clue while the span information is not used effectively, and it is difficult to improve the matching accuracy by using only the pitch information.

According to a first aspect of the invention, there is provided a melody retrieval method as claimed in claim 1.

According to a second aspect of the invention, there is provided a melody retrieval system as claimed in claim 4.

In its preferred aspects, the invention may provide:

a WWW-based melody retrieval system on which a user can input a melody by singing;

a WWW-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes from a music database accurately;

a WWW-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes extracted from a music database whereby an accurate song's title can be obtained quickly;

a WWW-based melody retrieval system with thresholds determined by using distribution of pitch and span of notes from a music database whereby an approximate relative-value sequences of pitch and span with the maximum quantity of information can be obtained dynamically from music database and user's input melody;

a WWW-based melody retrieval system which can convert relative-pitch or relative-span information of notes into approximate relative-values by categorisation whereby the approximate values of each category appear with equal frequency in a database;

a WWW-based melody retrieval system which can determine arbitrary number of categories for converting relative-pitch or relative-span information of notes into approximate relative-values;

a WWW-based melody retrieval system using the coarse-to-fine matching whereby the number of answer candidates can be reduced by considering the trade-off between coarse matching and fine matching;

a WWW-based melody retrieval system which utilizes a sung melody as a query and is reliable so that it is robust enough to deal with anonymous users's inputs;

a WWW-based melody retrieval device which can be mounted either in a desired position or article whereby a melody retrieval can be carried out easily from the remotely located music databases over a WWW network;

a WWW-based melody retrieval device which can be set in a car whereby a car driver can retrieve music by singing, thus enabling the car driver to keep safe driving since there is no need for him to use his hands for retrieving or selecting music;

a WWW-based melody retrieval device which can be set in a karaoke player whereby a user can retrieve songs by singing;

a WWW-based melody retrieval device which can be set in a record shop whereby a user can retrieve music by singing over the network even if the record shop does not have the user's desired music CDs;

a WWW-based melody retrieval device which can be set in either a cellular phone or a portable device whereby a user can retrieve music anywhere;

a WWW-based melody retrieval device on which a plurality of users can input their desired melodies by singing at the same time; and

a device for retrieving the song's title from a music database over a WWW network whereby the matching accuracy of a music retrieval is high enough for melody retrieval services over a WWW network.

In a further preferred aspect of the invention, there is provided a media retrieval system which can retrieve media with music by melody.

A preferred embodiment of the invention will now be described by way of example only and with reference to the accompanying drawings, in which:

FIG. 1 is a histogram of the relative-pitch values in a melody retrieval system with thresholds determined by using distribution of relative-pitch values of notes extracted from a database, wherein the distributed three categories of U(p), E(qual) and D(own) for pitch determining the thresholds are illustrated;

FIG. 2 is a histogram of the relative-span values in the melody retrieval system with thresholds determined by using distribution of relative-span values of notes extracted from a database, wherein the distributed three categories of L(onger), E(qual) and S(horter) for span determining the thresholds are illustrated;

FIG. 3 is a flow chart illustrating the operation of the melody retrieval system, and

FIG. 4 is a melody retrieval terminal device which may be used with the melody retrieval system.

Referring now to the accompanying drawings, in which like numerals designate like parts throughout the several views thereof, there is shown in FIG. 3, a first embodiment of a melody retrieval system.

In preparing for retrieval, sequences of relative-pitch difference and relative-span ratio are first obtained from sequences of pitch and span values of each piece of music in a database (S101), and a histogram of the relative-pitch values and a histogram of the relative-span values of the database are made by using the distribution of all notes in the database (S102).

In FIG. 1, the relative-pitch value is the normalized pitch difference between the adjacent notes in which a semitone difference is normalized to the value of 100.

In FIG. 2, the relative-span value is expressed by percentage.

In making these histograms, the total frequency of values of a histogram of pitch is expressed by Sum1, and that of a histogram of span is expressed by Sum2, respectively. The number of categories for the histogram of relative-pitch values is expressed by Category_Num1, and that of relative-span values is expressed by Category_Num2 respectively.

In this case, thresholds for relative-pitch values are determined so that each category, which will be divided by the thresholds, equally has M1 (=Sum1 / Category_Num1) values in it. Thresholds for relative-span values are similarly determined so that each category equally has M2 (=Sum2 / Category_Num2) values in it (S103). Subsequently the relative-pitch difference and relative-span ratio are respectively converted into the approximate relative-pitch values and relative-span values by categorizing those values in several parts like U, E, D in FIG. 1 or S, E, L in FIG. 2 which are separated by the thresholds previously calculated (S104).

Each number of category is 3 in FIGS. 1 and 2.

A song is inputted by a user singing, humming, whistling or similar into a microphone 10 as preparation for retrieval (S105). The song is subjected to an A/D conversion (S106), and the A/D converted signal is written into a memory 16, through which the A/D converted signal is processed by a processor 14 in accordance with a program of the memory 16 to detect a voiced sound from the A/D conversion signal (S107). A fundamental frequency of each frame is defined from the detected voiced sound (S108).

An onset time of the voiced sound is divided off as an onset time of each note, a time difference (number of frames) with an onset time of the next note is determined as the span of the note, and the maximum value among the fundamental frequencies of each note contained during its span is defined as the highest pitch values (S109).

The relative pitch and span values (S110) of each note in the input song are calculated from the obtained pitch and span from the previous note. The relative values are then transmitted to a song retrieval system A, and this transmittal may take place over the WWW network.

As shown in FIGS. 3 and 4, the relative-pitch values and relative-span values of the user's input melody are respectively converted into approximate relative-pitch values and approximate relative-span values by using the same thresholds of (S104) and categorizing relative-pitch values and relative-span values in several parts like U, E, D in FIG. 1 or S, E, L in FIG. 2 (Sill). These approximate values are compared with the value of each melody in the music database, a distance between the pitch and span of each melody in the music database and the input is calculated (S112), and the titles of pieces of music, which are sorted by the order of short distance previously calculated, are shown on a display 20 of an input device B (S113).

As particularly shown in FIG. 4, the input device B comprises a microphone 10, an A/D converter 12, a processor 14, a memory 16, an input and output unit 18, a display 20, a speaker 22 and a D/A converter 24.

It can be understood from FIG. 4 that the input and output unit 18 may be connected to a WWW network (not shown).

Received from the song retrieval system A is the retrieved result, which is shown in the display 20, and the desired melody among the retrieved songs can be heard from the speaker 22.

As described in the foregoing paragraphs, the title of the desired song can be easily and accurately retrieved by inputting the melody by singing into the microphone 10.

Claims

A melody retrieval method, comprising the steps of:

obtaining sequences of relative-pitch difference and relative-span ratio from sequences of pitch and span values of each piece of music in a database;

making a histogram of the relative-pitch values of the database by using the distribution of all notes in the entire database;

making a histogram of the relative-span values of the database by using the distribution of all notes in the entire database;

determining thresholds for relative pitch so that each category, which will be divided by the thresholds, equally having M1 (=Sum1 / Category_Num1) values in it, and thresholds for relative-span values are similarly determined so that each category equally has M2 (=Sum2 / Category_Num2) values in it;

converting the relative-pitch difference and relative-span ratio into the approximate relative-pitch values and approximate relative-span values respectively;

inputting a song by humming, singing or similar into a microphone as preparation for retrieval;

subjecting the inputted song to an A/D conversion;

detecting a voiced sound from a sound signal after A/D conversion;

defining a fundamental frequency of each frame from the detected voiced sound;

dividing off an onset time of the voiced sound as an onset time of each note;

determining a time difference (number of frames) as the span of the note;

defining the maximum value among the fundamental frequencies of each note contained during its span as the highest pitch value;

calculating the relative pitch and span values from the obtained pitch and span from the previous sound;

converting the relative pitch and span information of the user's input melody into approximate relative-pitch values and approximate relative-span values by using the thresholds previously obtained;

comparing the approximate relative-pitch values and approximate relative-span values with the approximate relative-pitch values and approximate relative-span values of each melody in the music database;

calculating a distance between the pitch and span of each melody in the music database and the input;

retrieving at least one piece of music based on the calculated distances; and

presenting information about the retrieved music.
A melody retrieval method as claimed in claim 1, wherein the steps concerning the database and the comparison of the coarse values are carried out at a first location, and said steps concerning the processing of the input song take place at a second location, the relative pitch and span values of the input song being transmitted from said second location to said first location, and the information about the retrieved music being transmitted from said first location to said second location, preferably via the World Wide Web.
A melody retrieval method as claimed in any preceding claim, further including a step of:

retrieving a piece of media with music by melody retrieval.
A melody retrieval system, comprising:

a database of pieces of music;

first means for obtaining sequences of relative-pitch difference and relative-span ratio from sequences of pitch and span values of each piece of music, making a histogram of the relative-pitch values of the database by using the distribution of all notes in the entire database, making a histogram of the relative-span values of the database by using the distribution of all notes in the entire database, determining thresholds for relative pitch so that each category, which will be divided by the thresholds, equally having M1 (=Sum1 / Category_Num1) values in it, and thresholds for relative-span values are similarly determined so that each category equally has M2 (=Sum2 / Category_Num2) values in it, and converting the relative-pitch difference and relative-span ratio into the approximate relative-pitch values and approximate relative-span values respectively;

input means for inputting a song;

processing means for subjecting the inputted song to an A/D conversion, detecting a voiced sound from a sound signal after A/D conversion, defining a fundamental frequency of each frame from the detected voiced sound, dividing off an onset time of the voiced sound as an onset time of each note, determining a time difference (number of frames) as the span of the note, defining the maximum value among the fundamental frequencies of each note contained during its span as the highest pitch value, and calculating the relative pitch and span values from the obtained pitch and span from the previous sound;

second means for converting the relative pitch and span information of the user's input melody into approximate relative-pitch values and approximate relative-span values by using the thresholds previously obtained, comparing the approximate relative-pitch values and approximate relative-span values with the approximate relative-pitch values and approximate relative-span values of each melody in the music database, calculating a distance between the pitch and span of each melody in the music database and the input, and retrieving at least one piece of music based on the calculated distances; and

presentation means for presenting information about the retrieved music.
A melody retrieval system as claimed in claim 4, wherein said first means and said processing means are in different locations, first means and said processing means being connected to allow transmission of information therebetween, preferably via the World Wide Web.
A melody retrieval system as claimed in claim 5 or claim 6, wherein said input means, said processing means and said presentation means are provided in an input device comprising a microphone, an A/D converter, an operating unit (operator) for treating a digital signal, an input and output unit for either transmitting or receiving the digital signal over the WWW network, a memory storing a program, a display showing the retrieved title of the database songs, a D/A converter, and an output unit for playing the retrieved song.
Computer software on a carrier allowing any of said first means, said processing means and said second means in claim 4 to carry out the actions specified in that claim.
A melody retrieval method comprising the steps of:

analysing each piece of music in a database of pieces of music and categorizing each piece of music according to the pitch and span of its notes;

analysing an input piece of music and categorizing it according to the pitch and span of its notes;

comparing the categorized input piece of music to the categorized pieces of music in the database; and

retrieving at least one piece of music from the database based on the results of the comparison.
A melody retrieval system comprising;

a database of pieces of music;

first means for analysing each piece of music in a database of pieces of music and categorizing each piece of music according to the pitch and span of its notes;

second means for analysing an input piece of music and categorizing it according to the pitch and span of its notes;

third means for comparing the categorized input piece of music to the categorized pieces of music in the database; and

fourth means for retrieving at least one piece of music from the database based on the results of the comparison.
Computer software on a carrier, allowing any of the said means in claim 9 to carry out the actions specified in that claim.