JP2002024232A - Device and method for music retrieval - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device and a method for music retrieval which can simplify user's operation and retrieval processing regarding music. SOLUTION: In a music database structuring part 1010, variation patterns of absolute scales of music are registered and a relative variation pattern as part of music is inputted as a query to a music retrieval device. A pattern retrieval processing part 1020 retrieves a range in the variation pattern of an absolute scale having the relative variation pattern by a continuous DP method.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a music search apparatus and a music search method for searching a music database for music having characteristics similar to those of a query voice.

[0002]

2. Description of the Related Art Continuous DP (for example, Ryuichi Oka, "Continuous D
Continuous Speech Recognition Using P ”, Acoustic Society of Japan, S7
8-20, pp, 145-152 (1978-06))
Is one of the basic pattern matching processing methods used for gesture recognition and voice recognition. Japanese Patent Application No. Hei 11-120 discloses a technique for searching for music based on voice requests.
No. 198. In this proposal, music is searched for by matching the difference between the scale indicated by the input voice and the scale of the music.

[0003] Also, "Melody Search Using Clues to Humming Singing", Kageyama, Takashima, IEICE (D-II, vol.
J77-D-II, no. 8, pp. 1543-155
1, 1994), "Song search system by singing voice on WWW", Sonoda, Goto, Muraoka, IEICE (D-II, vol.
J82-D-II, no. 4, pp. 721-731,1
999), a music search is performed by applying the continuous DP to the pitch and the pitch.

[0004] The continuous DP will be described below.

The time-series data of a T frame as a query is converted into N-dimensional feature vectors (z (τ, 1),..., Z (τ,
N))

[0006]

(Equation 1)

[0007] Similarly, the feature vector sequence of one music piece registered in the database is

[0008]

(Equation 2)

[0009] Also, (z (τ, 1),..., Z
The local distance between (τ, N)) and (u (t, 1),..., U (t, N)) is denoted as d (t, τ). An example of the calculation formula of the local distance is shown below.

[0010]

(Equation 3)

Here, the time axes of the database and the query are distinguished from each other by t and τ. Further, the point (t,
The cumulative distance of the feature vector between the query ending at τ) and the music on the database is represented as S (t, τ). In the continuous DP, S (t, τ) is updated by the following recurrence formula. Initial conditions (t = 0);

[0012]

(Equation 4)

[0013]

[Outside 1]

[0014]

(Equation 5)

[0015]

(Equation 6)

[0016]

(Equation 7)

In this recurrence formula, a value that minimizes the cumulative distance is selected from the three local paths shown in FIG.
Thus, the cumulative distance S (t, T) from the entire query is
This is the minimum cumulative distance when the expansion and contraction of the database in the time direction is 1/2 to 2 times. Output A (t) of continuous DP, ie, query and 1 on database
The cumulative distance between the feature vectors of two songs is normalized by the sum of weights 3T

[0018]

(Equation 8)

## EQU1 ## Such calculation is performed for the feature vectors of all songs on the database, and the minimum cumulative distance is also stored in the memory at the time of search.

[0020]

[Outside 2]

S (t, τ), S (t, 2,
τ), S (t−2, τ), d (t, τ), d (t−1,
τ), the music search device can search with only 5T of memory without depending on the number of frames in the database.

[0022]

In the conventional music search method, both the characteristics of the music registered in the music database and a part of the characteristics of the music used as a query are relative changes in musical scale, specifically, for example, The relative change was such that the musical scale went up two musical scales from the initial musical scale and then went down one musical scale. For this reason, conventionally, further processing is performed from music data indicating an absolute scale directly obtained from the waveform of the music,
A process for finding the relative scale must be prepared separately. There is a problem that this additional processing is not guaranteed to be performed well for general music.

SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide a music search apparatus and a music search method that can simplify a user's operation and search processing for music.

[0024]

According to the present invention, the feature of a music database is a temporal change of an absolute scale, and the temporal change of the absolute scale is compared with the temporal relative change of a query.
Search for songs. Based on such an idea, in order to achieve the above object, the invention of claim 1 provides a first feature of music registered in advance in a music database and a second characteristic of a part of music used as a query. In a music search device for comparing a feature with a second feature and searching for a range of a music most similar to the second feature, the first feature is represented by a power sequence of an absolute scale group of the music along a time axis direction, The second feature is represented by the relative scale of the music along the time axis direction, and the input means for inputting the second feature, and the first feature and the second feature input from the input means are continuously displayed. Music search means for searching for a range of music similar to the second feature by comparison using the DP method.

According to a second aspect of the present invention, in the music search apparatus according to the first aspect, the input means inputs a voice and analyzes the input voice to acquire the second feature. .

According to a third aspect of the present invention, in the music search apparatus according to the first aspect, a first feature relating to a plurality of songs is registered in the database, and the first feature of the plurality of songs and the first feature of the plurality of songs are registered. A feature is to compare the second feature.

According to a fourth aspect of the present invention, in the music search apparatus according to the first aspect, the music search apparatus further includes a reproduction output unit that reproduces and outputs the music whose range has been searched by the music search unit based on a first characteristic. It is characterized by.

According to a fifth aspect of the present invention, in the music search apparatus according to the first aspect, a plurality of sets of difference values of two scales having a predetermined time interval are acquired at different times on a time axis, and the acquired plurality of sets are acquired. The second feature is represented by a series in which a set of difference values are arranged.

According to a sixth aspect of the present invention, a first feature of a song registered in the song database in advance is compared with a second feature of a portion of the song used as a query, and the second feature is In a music search method for searching a range of similar music, the first feature is represented by a power sequence of an absolute scale group of the music along the time axis direction, and the second feature is expressed by a music sequence along the time axis direction. The second feature is represented by a relative scale, and the second feature is input to a music search device.
The first feature and the second input from the input means;
And searching for a range of music similar to the second feature by comparing the feature with the second feature.

According to a seventh aspect of the present invention, in the music search method according to the sixth aspect, a voice is input to the music search device,
The input voice is analyzed by the music search device to acquire the second feature.

According to an eighth aspect of the present invention, in the music search method according to the sixth aspect, a first feature relating to a plurality of music pieces is registered in the database, and the music search device stores the first feature of the plurality of music pieces. The first feature is compared with the second feature.

According to a ninth aspect of the present invention, in the music search method according to the sixth aspect, the music search device reproduces and outputs the music whose range has been searched based on the first characteristic.

According to a tenth aspect of the present invention, in the music search method according to the sixth aspect, a plurality of sets of difference values between two scales having a predetermined time interval are acquired at different times on the time axis. The second feature is represented by a series in which a set of difference values are arranged.

[0034]

Embodiments of the present invention will be described below in detail with reference to the drawings.

First, the principle of the music search method to which the present invention is applied will be described.

Registered in the query and database,
In the present embodiment, the feature pattern of a musical piece targeted for query comparison (distance calculation) is conventionally represented by a change pattern of a main melody, that is, a time series of scalar amounts (changes in relative scales). A new feature is that a feature pattern registered in the database uses a spatiotemporal pattern having a frequency (absolute scale) axis of a power spectrum whose horizontal axis is defined by a time axis and whose vertical axis is defined by a scale axis. In other words, in the present embodiment, a part that matches the musical scale change series of the query (part of the music to be searched) is found from the absolute musical scale change patterns that exist on the spatiotemporal pattern. Has a new feature. In order to execute such a process, this embodiment also uses the above-described continuous DP method. However, while the conventional two-dimensional cumulative distance is calculated, the present embodiment calculates the three-dimensional cumulative distance. This three-dimensional continuous DP will be described first.

Assuming that the total number of scales is N, the voice section length as a query is T, and the scale difference data of the frame is ｛q (τ) |
Let τ = 1,..., T｝. Here, the scale difference data is
After the main scale of the input voice is extracted, it is determined by the time difference. The time difference is a difference value between two scales having an interval of a predetermined time. A plurality of sets of difference values at different times on the time axis are obtained from the difference values, and the obtained plurality of sets of difference values are arranged. Represents the amount of change in the scale of the musical composition.

Therefore,

[0039]

[Outside 3]

May be used. On the other hand, a time-series pattern of a vector, which is an analysis feature of a set of feature vectors of a plurality of music pieces registered in advance (hereinafter sometimes simply referred to as a database), is used.

[0041]

(Equation 9)

It is assumed that Here, the time axes of the database and the query are distinguished from t and τ, respectively.

In the present embodiment, a continuous DP is used to find the most similar section in the target section time-series data that does not depend on the start and end of the section. Here, b (t, x) represents the power of t scales and scale x. Reference data (characteristics of query) is captured as a time-series pattern of scalar variation, and search data (characteristics of music)
Is extended so that is a spatio-temporal pattern. In this specification, such a continuous DP method is referred to as Model
driven path continuous DP (mp-continuous DP)
I will call it.

In the mp-continuous DP, a local distance to be accumulated is determined. Here, the local accumulated distance is expressed as d (x, t) with respect to time t and scale x in the database. An example of the calculation formula of the local distance is shown below.

The local distance d (x, t) includes x at time t
Power M (t) = max b (t, x)

[0046]

(Equation 10)

Can be used.

Further, the cumulative distance between the query ending at the point (x, τ, t) and the database is represented by S (x, τ, t).
Expressed by In the mp-continuous DP, S (x, τ, t) is updated by the following recurrence formula.

[0049]

S (x, 1, t) = 3d (x, t) (11)

[0050]

(Equation 12)

The boundary conditions are given below.

[0052]

(Equation 13)

In this recurrence formula, like the continuous DP, the cumulative distance S (x, τ, t) to the entire query is the minimum when the expansion and contraction of the database in the time direction is 1/2 to 2 times. It is a cumulative distance.

The output A (t, x) of the mp-continuous DP is normalized by the weight sum 3T.

[0055]

[Equation 14]

Is determined.

At this time, the detection result is the time of the database.
t ^* and scale x ^*, which are obtained by the following equations.

[0058]

(Equation 15)

Here, arg is a function that returns the argument (t, x) of S (x, τ, t).

When such a distance calculation is performed for one registered music piece, the portion having the minimum value of the distance calculation result is
This is the part that is most similar to the query in one song.
The same distance calculation is performed for all songs, and the song portion having the smallest value among the obtained minimum values is the final calculation result.

FIG. 1 shows a functional configuration of a music search apparatus for performing a music search by applying such a music search method.

In FIG. 1, reference numeral 1000 denotes a query acquisition unit for acquiring a characteristic pattern from a part of a music piece used as a query. The query acquisition unit performs a normal spectrum analysis such as FFT on an audio signal obtained from an audio device such as a microphone to obtain a fundamental frequency and a corresponding scale, and then generates a sequence of scale differences serving as a query. I do. The series of scale differences is obtained by acquiring a plurality of sets of difference values of two scales having a predetermined time interval at different times.

Reference numeral 1010 denotes a music database construction unit which stores feature patterns of a plurality of sets of music and stores them in the form of a database. The database construction unit performs spectrum analysis from music such as compact discs (CDs) and obtains power spectra for each scale.
These are stored as a spatiotemporal pattern on a recording medium such as a hard disk. Of course, the database can be expanded by repeating the input of music.

Reference numeral 1020 denotes a pattern search processing unit for calculating a distance between a feature pattern on a music database and a feature pattern of a query to search for a music part most similar to the query. For the distance calculation, the mp-
Continuous DP is used.

Voice (song, humming) as query input
Is represented as scale difference data, and is represented by a feature pattern of the music registered in the music database construction unit 1010, that is, a spatiotemporal pattern.

Reference numeral 1030 denotes a music performance section for detecting a music piece having a part most similar to the query from the search result of the pattern search processing section 1020 from the music database construction section 1010 and playing the music from the head of the detected music piece.

The pattern search processing unit 1020 performs the above-described mp-continuous DP
According to the calculation procedure of, the cumulative distance S (x, τ, t),
S (x, T, t) is calculated, and a process of finding the end (t ^* , x ^* ) of the part most similar to the sequence of scale differences from the spatiotemporal pattern is performed. At this time, if the time is traced back from the output time t ^* by the length of the query T, it is possible to find the beginning of the music portion suitable for the input voice. This start-up processing (detection of music and its head) is performed by a search music performance unit 1030
And perform (output).

An example of a music search apparatus for actually realizing such a functional configuration will be described below with reference to FIG.
FIG. 2 shows a system configuration of the music search device. Since a commercially available personal computer or workstation can be used as the music search device, the description will be briefly omitted.

In FIG. 2, the CPU 10 executes a music search program (described later) loaded into the system memory 50 to execute a music search process according to the present invention. The hard disk (HD) 20 stores a pattern database used in the music search program and the pattern search process, and in this embodiment, a database storing (storing) spatiotemporal patterns of a plurality of sets of music to be searched.

The I / O 80 detects the address position of the feature pattern on the database that coincides with the start time of the search result, reads the feature pattern of one entire music from the database on the hard disk 20 with the position as the start position, and reads out the speaker. Output to 30. The speaker 30 converts the output characteristic pattern into an audio signal and reproduces and outputs the audio signal.

The input interface (I / O) 40 inputs the voice of the search target (query) from the microphone 70. The system memory 50 temporarily stores a music search program executed by the CPU 10 and input / output data for the CPU 10. The input device 60 has a pointing device (hereinafter, a mask) such as a mouse and performs an operation instruction and information input to the CPU 10.

FIGS. 3 and 4 show the contents of the music search program. FIG. 3 shows the main processing procedure, and FIG. 4 shows details of the calculation processing of the local distance and the accumulated distance in FIG. The music search processing will be described with reference to FIGS.

In response to an instruction from the input device 60, the CPU 1
At 0, the system memory 50 loads the search program of FIG.

In FIG. 3, the user instructs to input a query from the input device 60, in this case, to input a voice of a query to be searched from the microphone 70. In response to this instruction,
The CPU 10 inputs a voice from a microphone via the input / output interface 40, performs processing as the query obtaining unit 1000 in FIG. 1, and writes the characteristic pattern obtained from the voice signal into the system memory 50.

Further, the CPU 10 performs initial setting of data used in the music search processing (step S10). As a result, the initial condition described in the equation (11) and the boundary condition described in the equation (13) are set. Next, the CPU 10 initializes t = 0 as the first frame number in the database in order to search the (song) database on the hard disk 20, and the starting position (t ₀ , x ₀ ) of the optimal path is ( ₀ , ₀ ). )
Set.

Next, the CPU 10 calculates the local distances and the cumulative distances of the plurality of musical scales (τ) inputted as the query and the spatiotemporal pattern of the database according to the equation (12) (step S30). The local distance calculation and the cumulative distance calculation will be described later in detail with reference to FIG.

Further, the CPU 10 performs a process of detecting the maximum value of the cumulative distance within a predetermined period for the calculated cumulative distance, and searches for the start position and the end position of the optimum path having the maximum value, in other words, the query Is output (stored) to the system memory 50 as a search result for a range of music similar to (a similar part in one music) (step S40).

Thereafter, the value of the variable t is incremented (updated one by one) until the variable t reaches the last frame number of the database in step S50 (step S60), and the local distance calculation and the cumulative distance calculation are performed. Is repeated, thereby detecting the range most similar to the query in one song. The above process is executed for a plurality of music pieces (feature pattern forms) on the database, and the one with the smallest cumulative distance is the search result. Eventually, the most similar location in the music on the database is detected, and the beginning of the music including that location is also detected.
At this time, the CPU 11 executes the pattern search processing unit 10 shown in FIG.
20 and functions as a music tip detection unit of the music performance unit 1030.

FIG. 4 shows details of the local distance calculation and the cumulative distance calculation in step S30 in FIG. In FIG. 4, the frame number τ on the query side is initially set to 0 (step S100) to calculate the local distance between the spatiotemporal pattern of t on the database and the first scale difference of the query, and further to 1 It is incremented (S110).

Next, for t, τ, and x, the local distance d
The calculation of (x, t) and S (x, τ, t) is performed by the above-described equations. The scale x reaches the total number of scales N (S17)
0), but is incremented (S120), but after reaching, together with the increment of τ (S110),
Initialized to scale number 1 (x = 1) (S12 following S115)
0). Subsequently, the increment of τ is continued until the end W of the database is reached. During the increment of x, τ, the accumulated distance S at the current time is always determined.
(X, τ, t) and the maximum value SMA of the cumulative distance up to that point
As the comparison of X is performed, the SMAX is also sequentially updated to be monotonically non-decreasing.

All procedures for calculating the cumulative distance are as follows: τ = T,
When x = N, the value of SMAX and scale x at that time
Is output as a return value, and the processing ends. When the process in FIG. 4 ends, the process returns to step S40 in FIG. Here, when the value of τ reaches the value T at the end of the query and the value of x reaches the total number of scales N, it is the end position of the optimal path at the current t. A data set (a set of three data) consisting of the start position of the optimal path and the value of the cumulative distance, which is associated with the cumulative distance of the value of t, is temporarily stored in the system memory 50.

In this embodiment, the cumulative calculation S (x, τ,
When calculating t), storing all of the plurality of data sets in the system memory 50 requires a large amount of memory area. Therefore, only the t-th (current) data set and the (t-1) -th data (previous) are stored. Prepare only the memory area to
Each time the value of t is updated, the data set obtained this time is moved to the previous storage area. For this reason, the data set stored so far is erased by overwriting with a new data set.

In the local maximum value detection processing of step S40 in FIG. 3, when the value of τ reaches the end position of the query by the above-described cumulative distance calculation, the data set of the start position, end position, and cumulative distance of the optimal path Is obtained. The value of t is incremented by the iterative calculation of FIG. 3, and the magnitude relationship between the temporary maximum value obtained so far and the value of the accumulated distance obtained this time is determined. If the value is larger than the value, the temporary maximum value is updated with the current accumulated value.

[0084]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A specific calculation example will be described with reference to FIG. After the query acquisition processing of Fig.
The query sequence is expressed as (q (1), q (2), q (3), q
(4)) = (2, 1, −1, 0). For example,
2) This corresponds to a case where a person sings according to the temperament of mi (+1) fa (-1) mi (+0) mi. The local distance with respect to the time-series pattern obtained from the music such as a CD through the database construction unit 1010 in FIG. 1 is expressed as a 5 × 10 matrix in FIG. Row index scale matrix (here, x _1, ..., x 5 scale of ₅₎ represents a column index t _1, ..., t ₁₀ represents a time or frame number in the database. Further, each element corresponds to a local distance. For example, element 1 at x ₃ row and t ₆ column is d (x ₃ , t ₆ )
= 1.

Calculation of Cumulative Distance According to equations (11) and (12), when t = t ₁ ,

[0086]

S (x ₁ , 1, t ₁ ) = S (x ₂ , 1, t ₁ ) =... = S (x ₁₀ , 1, t ₁ ) = 3 × 0.5 (16) In order to obtain the value of the expression (15), it is necessary to take the maximum value of the above cumulative distance.
5) So, take the first value x ₁ of the scale. In music search,
Since time t is important, there is no problem in taking the first value of the scale. The value of the output (14) of the mp-continuous DP is given by:

[0087]

A (x ₁ , 1, t ₁ ) = A (x ₂ , 1, t ₁ ) =... = A (x ₁₀ , 1, t ₁ ) = 1.5 / (3T) = 0.125 (17) Also at t ₁ ,..., t ₁₀ , cumulative distance calculation formula (11),
(12) calculated based on formula, when obtaining the output of mp- continuous DP, when t _7, the maximum value of the cumulative distance is, S (x _3,
4, taking the t ₇₎ = 12, t ₇ is the end of the desired music. Therefore, the starting point is t ₃ , which is four times back, and FIG.
It can be seen that the difference when marked with the arrow is consistent with the query. Therefore, the route determined by these arrows is from the beginning to the end of the desired music.

The following embodiment can be carried out in addition to the above embodiment. 1) In the above embodiment, the characteristic pattern of the music is registered in the form of a database. However, the acoustic signal of the music may be registered, and the characteristic pattern of the music may be extracted from the acoustic signal for each search. 2) The feature pattern itself of the music used as a query may be input to the music search device. 3) In the above-described embodiment, the performance of a music piece including voice is input by a microphone, Fourier analysis is performed, and a power spectrum component indicating a voice feature is taken out of the input, and the components are arranged in time, thereby obtaining a music database. An absolute scale (a first feature of the present invention) of a musical composition to be registered is created. This also has the effect of removing noise components such as performances. If you can get absolute scale without noise,
The absolute scale may be represented in other forms.

[0089]

As described above, the present invention is characterized by the temporal change of the absolute scale of a song such as a voice sung by a person. There is no need to take out the features and register them in the database. For this reason, the registration in the music database is simplified, and the search does not have to deal with both the absolute scale and the relative scale.

Further, the present invention can be applied to various applications, such as searching for a range of a music piece to be given as a query within a single music piece, searching for a music piece from a music piece part in a music database, and so on. Can be.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a functional configuration of an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a system configuration according to an embodiment of the present invention.

FIG. 3 is a flowchart illustrating a detection processing procedure according to the embodiment of the present invention.

FIG. 4 is a flowchart illustrating details of a calculation process of a local distance and a cumulative distance according to the embodiment of the present invention.

FIG. 5 is an explanatory diagram for explaining an embodiment of the present invention.

[Explanation of symbols]

 10 CPU 20 HD 30 Speaker 40 I / O 50 System memory 60 Input device 70 Microphone 80 I / O

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/12 G10L 3/00 531N 15/02 533A // G10L 101: 065 9/16 301A (72) Invention Person Hiroki Hashiguchi Ryukakusan 2F-12 Higashikanda, Chiyoda-ku, Tokyo Building 8th floor, Technology Research Association, New Information Processing Development Machine premises Inventor Ryuichi Oka 2-5-12 Higashikanda, Chiyoda-ku, Tokyo Building 8F Technology Research Association New Information Processing Development Facility F-Term (Reference) 5B075 ND14 NK06 PP03 PP07 PP12 PQ04 PR06 QM08 5D015 AA06 FF04 HH07

Claims (10)

[Claims] 1. A first feature of a song registered in a song database in advance and a second feature of a portion of a song used as a query are compared, and a range of a song most similar to the second feature is compared. The first feature is represented by a power sequence of an absolute scale group of musical scales along the time axis direction, and the second characteristic is represented by a relative musical scale along the time axis direction. And input means for inputting the second feature, and the second feature input from the first feature and the input means.
A music search device comprising: a music search unit that searches for a range of music that is similar to the second feature by comparing the feature with the second feature by a continuous DP technique. 2. The music search device according to claim 1, wherein said input means inputs a voice, and acquires the second feature by acoustically analyzing the input voice. 3. The music search device according to claim 1, wherein a first feature of the plurality of songs is registered in the database, and the first feature and the second feature of the plurality of songs are registered. A music search device characterized by comparison. 4. The music search device according to claim 1, further comprising: playback output means for playing back and outputting the music whose range has been searched by said music search means based on a first characteristic. Search device. 5. The music search device according to claim 1, wherein a plurality of sets of difference values of two scales having a predetermined time interval are obtained at different times on a time axis, and the obtained plurality of sets of difference values are obtained. A music search device wherein the second feature is represented by an arranged sequence. 6. A first feature of a song registered in a song database in advance and a second feature of a portion of a song used as a query are compared, and a range of a song most similar to the second feature is compared. In the song search method, the first feature is represented by a power sequence of an absolute scale group of musical scales along the time axis direction, and the second feature is represented by a relative musical scale along the time axis direction. Expressing and inputting the second feature to be input to the music search device, wherein the music search device compares the first feature and the second feature input from the input means by a continuous DP technique. A music range similar to the second feature. 7. The music search method according to claim 6, wherein a voice is input to said music search device, and said input voice is acoustically analyzed by said music search device to acquire said second feature. A featured music search method. 8. The music search method according to claim 6, wherein a first feature related to a plurality of songs is registered in the database, and the song search device stores the first feature of the plurality of songs and the first feature of the plurality of songs. A music search method characterized by comparing a second characteristic. 9. The music search method according to claim 6, wherein the music search device reproduces and outputs the music whose range has been searched based on a first characteristic. 10. The music search method according to claim 6, wherein a plurality of sets of difference values of two scales having a predetermined time interval are acquired at different times on the time axis, and the acquired plurality of sets of difference values are obtained. A music search method, wherein the second feature is represented by an arranged sequence.