JP2001265779A - Acoustic retrieving method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform processing based on accelerated and simplified operation in acoustic data retrieval for retrieving acoustic information in a library with acoustic information as a key. SOLUTION: Power for each musical scale is found from the acoustic of data base and keys by acoustic and normalization, that value is quantized into three large, medium and small values, the feature data of acoustic to be retrieved and key acoustic are prepared (100) and the coincidence of the feature data of acoustic to be retrieved and key acoustic is decided while using only three large, medium and small values (113). Thus, only by designating a parameter to be used for frequency analysis and the time of acoustic to decide coincidence, high-speed acoustic retrieval is performed with the feature data of a little bits.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound search method, and more particularly, to a method for retrieving desired sound and music information by using key sound information from an acoustic database including sound information such as sound and music in a library. The present invention relates to a method for creating an acoustic database for retrieval and an acoustic information retrieval method using the same.

[0002]

2. Description of the Related Art With the development of information processing technology, a multimedia database system and its search method,
A music data sales system that searches for and sells corresponding music from a music library using music information recorded on a recording medium such as a mini disk (MD) or a cassette tape or input from an audio input device such as a microphone microphone as a key, or Music / video search system for searching for a corresponding music from a video library such as a promotion video or a live video using music information recorded on a recording medium such as an MD or a cassette tape or input from an audio input device such as a microphone as a key. Is being developed.

[0003] Searching such a multimedia database system including audio information requires a system user to be able to search for desired audio information and video information easily and at high speed. In response to such demands, the following high-speed sound search methods are being studied as a method for performing high-speed image / sound search. (Kashino et al., High-speed search of image and sound time series using multimodal active search, IEICE, PRMU98-80, pp.51-58, 1998/0
9).

(1) Acoustic information is applied to a frequency band filter bank of N channels for every M samples, and the output of each channel is normalized, and the obtained N values are used as N-dimensional feature vectors.

[0005] (2) Each of the components of the feature vector is quantized into b values. From the above, the feature vector is L =
One of bN kinds of values.

(3) The K temporally continuous feature vectors of the above-mentioned acoustic information are combined (divided window), and a histogram thereof is obtained.

(4) The sum of the smaller values of the histogram values in the divided windows corresponding to the to-be-retrieved sound and the key sound is determined as the similarity of the divided windows, and (5) J continuous divided windows are collected ( Window of interest), the minimum value of the similarity of the divided windows included in the window of interest is calculated as the similarity S of the window of interest, and (6) the similarity S is calculated from the average, variance, and parameter c given manually. If it is larger than the obtained threshold θ, the corresponding search target sound is determined to be a search target; otherwise, the similarity S
The divided windows of the sound to be searched are shifted by the number calculated from the value of, and the above processes (2) to (6) are performed.

[0008] The above-described search method enables high-speed sound search, but its signal processing includes M, N, b, K, J, and c.
And other parameters must be given appropriately, and the signal processing becomes complicated.

[0009]

SUMMARY OF THE INVENTION The main object of the present invention is to:
It is an object of the present invention to provide a sound search method and system that enables a sound searcher to search for high-speed sound information by giving very few parameters.

Another object of the present invention is to provide a method for generating characteristic data of a sound to be searched and a sound converted from a key sound so that the similarity between the sound to be searched and the key sound can be determined by a simple arithmetic processing. It is to realize.

[0011]

In order to achieve the above object, in the sound search method of the present invention, a sound as a key for searching from sound data of sound to be searched (abbreviated as sound data to be searched) in a database or the like. (Hereinafter abbreviated as key sound)
In the method of searching for the searched audio data having, in the same manner as described below, the searched audio data and the key sound, feature data is extracted, and the similarity between the searched audio data and the feature data of the key sound is compared and determined. Search or reproduce sound information having a part similar to the key sound from the database.

In order to generate characteristic data of the searched audio data and the key audio data, the audio data is subjected to frequency analysis at predetermined time intervals (a plurality of samples) to obtain power for a plurality of frequency bands, and the power for each of the frequency bands is normalized. Then, the normalized power is quantized into three values to obtain the feature data. In order to create the feature data, a user of the search system needs a fixed time T corresponding to the feature data of the key sound and a feature vector constituting the feature data, that is, a search target required for performing the frequency analysis. What is necessary is just to input the sound data or the parameter E which determines the number of samples of the key sound data.

In particular, when the acoustic data is a musical sound, the power for each frequency is the power corresponding to each pitch in music, and the power corresponding to each pitch in music corresponds to a note name in each octave. The sum of the powers of the pitches is obtained, and feature data is extracted based on the twelve powers corresponding to the pitch names.

[0014]

1 and 2 are a PAD diagram illustrating processing steps in an embodiment of a sound search method according to the present invention, and a configuration diagram of an embodiment of a sound search system for implementing the sound search method. Show.

First, when constructing a multimedia database, which is a database containing audio data to be searched (hereinafter simply referred to as audio data), a feature data extraction program 210 for executing the method for creating audio information feature data according to the present invention. , A multimedia database 2
The feature data of the sound data of No. 21 is created (100).

Step 100 for creating the characteristic data
Then, the comparison time T (second) corresponding to the time length of the key sound and the parameter E required for the frequency analysis are stored in the memory 20 using the input / output device 202 including a keyboard, a display, and the like.
3 is set (101). Feature data, comparison time T
(Seconds) and the parameter E will be described later in detail.

Next, the processor 201 uses the characteristic data extraction program 210 and the parameters T and E set in the memory 203 to process each of the plurality of acoustic data (multimedia data) 102 stored on the disk 221. , Extract the characteristic data of the acoustic data (10
3) Save the feature data in the feature data storage unit 220.

Inputting a key sound, the sound database 22
When performing a sound search from No. 1, the processor 201 executes a search by the following procedure using the sound data search program 211 (110).

First, key sound data is input to the input / output device 20.
2 is designated (111). As the input device 202 for inputting the key sound data, if the key sound data is already digital data, a device such as a CD, MD, MO, or floppy (registered trademark) in which the key sound data is stored is used. Use the corresponding drive. When searching the multimedia database 221 through a network,
Cards can be used. In the case of data stored in a portable acoustic storage device or the like, a connection device with the storage device is used. When audio is directly input or input from an analog device such as a cassette tape, an analog audio signal from a microphone / cassette deck is converted into digital data by an AD converter.

Next, the processor 201 uses the memory 203 to extract the feature vector of the key sound data by the same method as the method for extracting the feature data of the sound data, and stores it in the memory 203 (112). . A match is determined between the feature vector of the feature data of the key sound data and the feature vector on the feature data, and sound data having the matching feature data is obtained from the multimedia data (113).

The result of the match determination is output to the input / output device 202. That is, when there is matching audio data, the multimedia data name as the audio data is displayed on the display device constituting the input / output device 202, or the audio information itself of the multimedia data is reproduced as audio information. I do. Although FIG. 2 shows an example in which the feature data extraction processing 100 and the acoustic data search processing 110 are performed by the same processor 201, these may be processed by using separately provided processors. When processing is performed using a separately provided processor, multimedia data is extracted (100) after performing feature data extraction.
The data 221 and the feature data 220 are copied or moved to another system, and a search process (110) is performed. Therefore, the system that performs the feature data extraction (100) stores the feature data extraction program 210, and the system that performs the search process (110) stores the acoustic data search program 211.

FIG. 3 shows the feature data of the multimedia data stored in the feature data storage section 220. The feature data of the sound data includes two internal parameters (M and N) 300 required for feature vector extraction, multimedia
Data file information 301 ... 303, feature vector 3
10 ... 313. The file information 301... 303 is composed of a file name 321 and a pointer 322 indicating a feature vector start position in the memory where the feature vector of the file is stored. Feature vector 310 ...
Each of 313 is, for example, a set of 24-bit data 320 (referred to as feature element side), as described in detail below.

FIG. 4 is a diagram for explaining the feature vector and the feature data in detail.

FIG. 2A is a waveform diagram of a part of a musical tone as acoustic data. Each data x is sample data sampled at a sampling frequency S. The predetermined period T of the acoustic data is divided into continuous N (integer) blocks, the frequency characteristic of each block is obtained, and the frequency characteristic of each block is converted into a specific form, which will be described later, as the feature vector. Assuming that the number of samples in each block is M, a relationship of N = TS / M holds.

The number of samples the M, the frequency analyzing acoustic data (FFT), the number required to convert the frequency information, it can be expressed by M = 2 ^E. Here, the parameter E is a constant experimentally determined from the range of the sampling frequency of the acoustic data. The parameter E is a sampling frequency of 11, 25 for 025 Hz / second, 22, 22 and 050 Hz / sec, 10 and 4 for normal sound.
4. In the case of 100 Hz / sec, good search results can be obtained in 11 and multimedia data stored in the database
The parameter E is determined according to the sampling frequency of the data sound.

FIG. 2B is a frequency characteristic diagram of the acoustic data during the above-mentioned fixed period T, wherein the horizontal axis represents the logarithmic frequency and the vertical axis represents the power P. In the frequency analysis, M pieces of acoustic data are obtained by shifting each time, so that N frequency characteristics can be obtained for the period T.

(C) Each of the above-mentioned N frequency characteristics is represented by a specific frequency component, for example, m powers in several wavenumber bands f1, f2... Fm-1, fm corresponding to the pitch of music. Is done. Therefore, m × N power data are constituted by the entire N frequency characteristics.

(D) Each of the power data is 3
Quantized to a value. The method of quantizing into ternary values is based on the above mN
The average value μ and the standard deviation value σ of the power data are obtained, and the values of the individual power data are O in the range of μ−σ to μ + σ, the value bl lower than the value μ−σ, and the value higher than the value μ + σ. It is 2-bit data indicating whether the data is bh. The mN pieces of power data quantized into the three values are referred to as feature data, and each of the N 2m-bit feature elements constituting the feature data is referred to as a feature vector.

Returning to FIG. 1, the parameter input processing (10
1) and the process (103) of extracting characteristic data will be described. FIG. 5 shows a process (10) for extracting the above-described feature data.
It is a PAD figure showing 3). After inputting the parameter E for determining the parameter M and the number of seconds corresponding to the period T for performing a match determination in the search, the file names 301, 302,...
.. Are stored in the file name 321 storage section in the feature data 220 (501). Here, the number of the file is represented by i.

Next, the multimedia of file number i
A pointer p (i) 322 indicating the start position of a series of feature vectors storing the feature data of the data is set, and at the same time, the position is stored in a variable j (502).

Next, the number K (i) of all feature vector data of the feature vectors of the feature data of the file number i is obtained (503). The feature vector is, as described in FIG.
Since the sound data (sample data) is obtained using MN data while shifting the data by M pieces, the number of feature vector data K (i) is determined by the number of sound data ND (i) in the multimedia data of file number i. ) (Total number of samples)
From MN and divide by M.

Next, the number of the feature vector is k, and k =
The following processing is performed for 0 to K (i) -1 (504). K of audio data of multimedia data of file number i
The feature vector is extracted using the (k + MN−1) th from the (th) to (510), and each of the feature vectors is d (j
1) to d (jN) (313), and the characteristic data storage unit 22
0 is stored (511), and j is finally incremented (512).

FIG. 6 shows the above-described feature vector extraction processing (5).
It is a PAD figure explaining 10). Here, it is assumed that a series of sample values of the acoustic data to be subjected to the feature vector extraction processing are x (k) to x (k + MN-1).

The feature vector extracting process (112) for the key acoustic data includes the first MN key acoustic data as data for the key acoustic data, and the feature vector extracting process (510) for the acoustic data in the database.
Then, D (k) to D (k + MN-1) are the target data. Feature vector extraction processing for key sound (112)
Then, the extracted feature vectors are d (0) to d (N−
1) is stored in the memory 203 of FIG. In the feature vector extraction process (510) for the sound data in the database 221, the feature data is stored in the feature data storage unit 220 as d (j1) to d (jN).

First, the following processing is performed for i = 0 to N-1 (600). Acoustic data x (iM) to x (iM + M-
For 1), frequency analysis is performed using a technique such as FFT (601). As a method such as FFT, a conventionally known method can be used. From the obtained power for each frequency band, power is obtained for each frequency band corresponding to the pitch of music (602). Specifically, for example, pitch A3 (LA, 440
Hz), the power in the frequency band corresponding to [ ^{2-0.5 / 2} 4
440 Hz to ^{2 0.5 / 2} 4.440 Hz], that is, the sum of the frequency components in the range of [427 Hz to 453 Hz].

Next, the power is summarized for each scale (60
3). For example, as the power of the pitch A (la), the sum of the powers corresponding to the pitches such as A2, A3, and A4 for each octave is obtained. As a result, the frequency analysis result corresponding to the 12-step pitch has 12 powers P (0, i) to P (11,
i). Note that the power obtained in the processing steps 601 to 603 can also be obtained by applying a band-pass filter to the acoustic data. In addition, the method is such that the data in the database is searched up or down (ie, the frequency is doubled or halved) to retrieve the original data in order to summarize the power for each octave. Has features.

The twelve powers P (0, i) to P (11,
As a method for obtaining i), x (iM) to x (iM + M−
In contrast to 1), for example, when obtaining the power of the pitch A (la), a band-pass filter that passes only frequencies corresponding to pitches such as A2, A3, and A4 for each octave is provided.
A method using the output power of the filter can be adopted.

Next, as described with reference to FIG. 4D, the power P (j, i) (j = 0 to 1) obtained by the above-described processing is obtained.
1, the average value μ of i = 0 to N−1) and the standard deviation σ are obtained (611), and the lower limit L (= μ−σ) and the upper limit H (= μ + σ) of the intermediate value of the power are obtained (612). Power P (j,
When i) is not in the [L, H] range, the power will represent a salient feature of the sound.

Then, next, all the powers P (j, i)
The following processing is performed on (j = 0 to 11, i = 0 to N-1) (620). If P (j, i) is smaller than the lower limit L, the feature value b (j, i) is set as the feature amount bl indicating that the power is small (622). If P (j,
If i) is larger than H, b (j, i) is set as a feature amount bh indicating that the power is large (623). Otherwise, P (j, i) is an intermediate value and b (j, i)
Is a logical value “0” indicating that it is not a characteristic value (6
24). Here, b (j, i) = “0”, bl and bh are all represented by 2-bit values, bl = 1 and bh = 2 in the feature data of the sound data, and bl in the feature vector of the key sound. = 2, bh = 1. The power P (j,
j = 0 to 11, i = 0 to N-1 of i) and b (j, i)
4 correspond to (c) and (d) in FIG. 4, respectively. By exchanging the values of bl and bh with the database and the key sound, the feature vector match determination (113) can be performed at high speed only by a logical AND (AND) bit operation, as described later.

Finally, the following processing is performed for i = 0 to N-1 (630). First, d (i) = “0” (63
1). Next, when j = 0 to 11 (632), d (i) is 2
The process (633) of shifting the bit to the left and substituting the result of the OR operation with b (j, i) is performed. Step 631
In the processing of to 633, the characteristic value of each scale obtained in 621 to 624 is substituted into the 0 to 24 bits of d (i) by 2 bits. Therefore, d (i) is a 24-bit value, and N feature vectors d (i) are obtained by this processing.

Next, feature vector coincidence determination processing (11)
3) will be described with reference to FIG. Here, the feature vector of the key sound obtained in the feature vector extraction process (112) for the key sound and stored in the memory (203) is dk
(I) (i = 1 to N). D of each of the feature vectors (310 to 313) for the multimedia data in the database 221 stored in the feature data 220
The following processing is performed for (j, i) (700). The following is processing for the j-th feature vector.
First, (701) for i = 1 to N, dk (i) and d
The logical product of (j, n) is obtained (702). If the logical product is not "0", the subsequent processing is interrupted, and the processing shifts to the processing of the next feature vector j + 1 (703). If n = 1 to N
If the logical product of dk (i) and d (j, i) is "0" for all i, it is determined that the feature vectors match, and the file name including the corresponding feature vector is obtained (710). ). When the corresponding feature vector j is located between the pointer (322) p (i) and p (i + 1) in the file information (301 to 303), the file name i is the file name to be obtained. If you are asked for a file name,
The match determination processing ends (711).

FIG. 8 shows a true value table in the AND operation. As described above, when the feature element side of the feature vector for the key sound is larger than H (H = μ + σ), between H and L, and smaller than L (L = μ−σ), bh = “ 01, O = “00” and bl = “10”
And the feature vector of the searched audio data is
Bh = “10”, O = “00” and bl = “0” respectively
1 ". Therefore, the logical product operation of each feature element side becomes like a true value table, and the match determination process can be performed at high speed by a simple calculation process. The required memory capacity of the storage means can be reduced.

If the present invention is applied to a music database,
It is possible to realize a music distribution / sales service in which a user can purchase music from a part of music recorded by air check of a radio or television music program or a microphone. Similarly, if applied to a music video database,
It is possible to realize a music / video distribution / sales service in which a video can be purchased or viewed from a part of music not accompanied by video. Further, when the feature data is extracted from the music database, the file name (321) is obtained from the audio data provided by the user without the multimedia data (221), and the song name search for outputting the song name or the like based on the file name is performed. Services can also be realized. Further, audio information distributed via the Internet or broadcasting is given to the music database as a key, and when music included in the music database is searched, a charging process is performed, whereby the music included in the music database is processed. A system for monitoring copyright can be realized.

[0044]

According to the present invention, if the feature data is used,
Characteristic data can be represented by data of an extremely small number of bits, and high-speed processing can be realized by a simple logical operation processing for determining the coincidence between sound data to be searched and key sounds in sound search or the like. A high-speed sound search is realized only by the operator of the search system designating a parameter E used for frequency analysis and a sound time T for determining a match.

[Brief description of the drawings]

FIG. 1 shows a PA of an embodiment of an acoustic search method according to the present invention.
FIG.

FIG. 2 is a block diagram of an embodiment of an audio search system according to the present invention.

FIG. 3 is a diagram showing acoustic feature data in an embodiment of an acoustic search method according to the present invention.

FIG. 4 is a diagram illustrating a feature vector and feature data.

FIG. 5 is a PAD diagram showing a feature data extraction process for one piece of multimedia data in one embodiment of the sound search method according to the present invention.

FIG. 6 is a PAD diagram showing a feature vector extraction process in one embodiment of the sound search method according to the present invention.

FIG. 7 is a PAD diagram showing a feature vector match determination process in one embodiment of the sound search method according to the present invention.

FIG. 8 shows a true value table for explaining the logical operation of the feature vector match determination processing in one embodiment of the sound search method according to the present invention.

[Explanation of symbols]

201: processor, 202: input / output device, 203: memory, 210, 211 ... program, 220, 221 ...
Data, 300: parameter, 301 to 303 file information, 310 to 313: feature vector, 320: feature element side, 321: file name, 322: pointer to feature vector.

Claims (8)

[Claims] A first parameter having a value corresponding to a length of the key sound data from the sound data serving as a search key; and a sample of the sound data for performing a frequency analysis of the key sound data. A first step of setting a second parameter of a value for setting a value of a time interval, a second step of extracting characteristic data of key audio data using the first and second parameters, Second
A third step of extracting feature data of the searched sound information from the searched sound information of the sound data included in the database by the same method as in the step; and extracting the feature data of the searched sound information and the key sound data. A fourth step of comparing and judging the feature data and searching the database for acoustic information having a part similar to the key sound.
A sound search method comprising steps. 2. A first step of performing frequency analysis of acoustic data at predetermined time intervals to obtain power for each of a plurality of frequency bands, a second step of normalizing the power of each frequency, and a ternary value of the normalized power. And generating a characteristic data of the acoustic data. 3. The method according to claim 2, wherein said acoustic information is music sound, and said power for each frequency band is a power corresponding to each pitch in music. 4. The power corresponding to each pitch in the music sound is obtained by summing the power of the pitch corresponding to the pitch name in each octave, and feature data is extracted based on the 12 powers corresponding to the pitch name. 4. The method according to claim 3, wherein the characteristic data of the acoustic information is created. 5. The step of quantizing into the above three values obtains upper and lower limits of an intermediate value from the obtained power, and when each power value is larger than the upper limit value, smaller than the lower limit value, and 3 when the value is between the value and the lower limit
5. The method according to claim 2, wherein the quantized power is quantized to a value, and the quantized power is used as feature data as a 2-bit value. 6. The method according to claim 1, wherein the characteristic data of the searched audio data and the key audio data is created by the characteristic data generating method of any one of claims 2 to 6. Acoustic search method. 7. The method according to claim 1, wherein the characteristic data of the acoustic data and the acoustic data serving as the key are created by the method for creating characteristic data of acoustic information according to claim 5. Of the quantized feature values, the intermediate value is set to 0, and the feature data obtained from the database and the feature data obtained from the key are inverted with respect to the values representing the large and small values. And a logical product of the respective feature data, and using a portion where the value of all the lengths of the set feature data becomes 0 as a search result. 8. The method according to claim 2, wherein
A database comprising characteristic data created by two acoustic information characteristic data creating methods and storing characteristic data.