KR100893123B1 - Method and apparatus for generating audio fingerprint data and comparing audio data using the same - Google Patents

Abstract

The present invention relates to a method and apparatus for generating audio fingerprint data, and to a method and apparatus for comparing audio data using the same, the method comprising: generating audio fingerprint data from audio data, the method comprising: generating at least one audio data; Dividing into one frames; Extracting feature data for each of the divided first frames; Generating feature distribution data representing a distribution feature of the feature data for each of the divided first frames; Dividing the audio data into at least one second frame; Generating audio fingerprint data for each of the second frames using the feature distribution data generated for the first frames; And generating audio fingerprint data for the whole audio data by a set of audio fingerprint data for each of the second frames, and a method using the same and an apparatus using the same. Provide a method.

Audio data, audio fingerprints, histograms

Description

METHOD AND APPARATUS FOR GENERATING AUDIO FINGERPRINT DATA AND COMPARING AUDIO DATA USING THE SAME}

1 is a block diagram illustrating a coupling relationship between an audio fingerprint generation and audio data comparison apparatus 100 and other components for performing the audio fingerprint data generation method and the audio data comparison method using the same;

FIG. 2 is a diagram schematically illustrating an entire process of an audio fingerprint data generation method and an audio data comparison method using the same according to the present invention; FIG.

3 is a flowchart illustrating an entire process of a method for generating audio fingerprint data according to an embodiment of the present invention;

4 is a diagram for describing a method of dividing audio data into at least one first frame;

5 is a view for explaining a process of extracting a feature vector by the MFCC method;

6 is a view for explaining a process of generating a codebook;

7 is a diagram illustrating an example of a codebook configuration;

8 is a view for explaining the audio fingerprint generation process according to the present invention for reference;

9 illustrates an example of a method of configuring audio fingerprint data for each second frame from feature distribution data;

10 is a flowchart illustrating an embodiment of a method for comparing audio data using audio fingerprint data according to the present invention;

11 is a diagram illustrating a process of dividing audio data into at least one segment;

12 is an overall flowchart of a case where an audio database including at least one or more original audio data is compared with the object data to be compared;

13 is a view for explaining an example of a method for setting a threshold.

<Explanation of symbols for main parts of the drawings>

10: audio data input device

20: audio fingerprint data generation unit

30: audio data comparison server

40: audio database

The present invention relates to a method and apparatus for generating audio fingerprint data, and to a method and apparatus for comparing audio data using the same, and more particularly, to generate audio fingerprint data indicating characteristics of audio data from the audio data and to use the audio data. The present invention relates to a method and apparatus for quickly and easily determining whether or not there is an identicalness between them.

Audio finger print refers to data that can generally describe the characteristics of audio data. The audio finger print is generated by analyzing audio data by various methods such as frequency conversion and using the same. To determine whether an unauthorized use of audio data is illegal or to search for audio data by an audio fingerprint.

Conventional methods for generating such an audio fingerprint have been proposed, but the conventional audio fingerprint generation method has a large amount of audio data to be searched (about 10000 or more). There is a disadvantage in that the speed of fingerprint generation is significantly slowed down, which is not suitable when comparing a large amount of audio data.

In addition, Korean Patent Registration No. 10-0456408 discloses an audio gene extraction method using a binary feature, but this means that the energy of the spectrum is zero or every 32 frequency bands for each frame of each audio data in the database. Express this value as 1, add this value to the lookup table value as (audio signal ID, frame i), and extract this 32-bit pattern in the same way for the input audio in any number of seconds. As a method of searching, the number of (audio signal ID, frame index) registered in each entry of the search table is variable, so that a sufficient search speed cannot be guaranteed. Also, the binary feature vector extraction method is fixed so that the input signal is fixed to the input signal. It has the disadvantage of being relatively vulnerable to damage that has occurred.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a method and apparatus for generating audio fingerprint data, which has a markedly improved speed and relatively simple generation method compared to a conventional audio fingerprint generation method. do.

In addition, the present invention analyzes the audio data to generate the audio fingerprint data using the distribution characteristic data of the characteristic data of the audio data, so that the audio fingerprint that can accurately reflect the characteristics of the audio data compared to the conventional method Another object is to provide a data generating method and apparatus.

In addition, the present invention compares the audio data having the audio fingerprint data generated by the method and apparatus for generating audio fingerprint data as described above, it is possible to obtain a fast and accurate comparison result compared to the conventional method and whether or not the same Another object of the present invention is to provide a method and apparatus for comparing audio data, which can significantly reduce a determination error of an audio signal.

Further, according to the present invention, since the audio fingerprint data can be divided into segments and compared and judged for each segment, the present invention can be applied to the case where the original audio data and the comparison target audio data are partially the same, and when the portions are the same, the same position is also detected. Another object of the present invention is to provide a method and apparatus for comparing audio data.

In addition, the present invention extracts the audio signal from the video data including the video signal and the audio signal to generate and compare the audio fingerprint data for the audio signal to easily determine whether unauthorized copying and theft of the video data as well as the audio data. Another object of the present invention is to provide a method and apparatus for comparing audio data, which can be easily applied even when only a part of video data is edited and stolen.

According to an aspect of the present invention, there is provided a method of generating audio fingerprint data from audio data, the method comprising: dividing audio data into at least one first frame; Extracting feature data for each of the divided first frames; Generating feature distribution data representing a distribution feature of the feature data for each of the divided first frames; Dividing the audio data into at least one second frame; Generating audio fingerprint data for each of the second frames using the feature distribution data generated for the first frames; And generating audio fingerprint data for the entirety of the audio data by a set of audio fingerprint data for each of the second frames.

The audio data may be extracted from video data including a video signal and an audio signal.

The method may further include normalizing the audio data to a predetermined frequency band before dividing the audio data into at least one first frame.

The dividing of the audio data into at least one first frame may be performed such that one frame to be divided and a frame having a next time interval adjacent to each other have a portion in common with each other.

The dividing of the audio data into at least one second frame may be performed such that one frame to be divided and a frame having a next time interval adjacent to each other have a portion in common with each other.

In addition, the length of each of the second frames may be longer than the length of each of the first frames.

The extracting of the feature data for each of the divided first frames may include, for each of the divided first frames, a Mel Frequency Cepstral Coefficient (MFCC), a Perceptual Linear Prediction Coefficient (PLPC), or an LPC (Linear). Prediction Coefficient) can be configured to extract a feature vector using at least one combination.

In addition, generating the feature distribution data representing the distribution feature of the feature data for each of the divided first frames, grouping the feature vectors into at least one or more groups, and stores the index value for each group The feature distribution data may be generated by calculating an index value for each of the divided first frames with reference to the codebook.

Here, the codebook may have an average value of feature vectors for each group, and calculate an index value based on a distance between feature vectors for each of the divided first frames and an average value of feature vectors for each group. have.

Further, in the generating audio fingerprint data for each of the second frames using the feature distribution data generated for the first frames, the audio fingerprint data for each of the second frames is It may be generated by calculating the frequency of the feature distribution data for each of the first frame included in each of the second frame.

According to another aspect of the present invention, in an audio fingerprint data generating apparatus for generating audio fingerprint data from audio data, the audio data is divided into at least one or more first frames, and each of the divided first frames is provided. Extract feature data, generate feature distribution data representing a distribution feature of the feature data for each of the divided first frames, divide the audio data into at least one second frame, and Generate audio fingerprint data for each of the second frames using the feature distribution data generated for one frame, and generate the entire audio data by a set of audio fingerprint data for each of the second frames. Audio ping to generate audio fingerprint data for Generating print data that can provide audio fingerprint data generating apparatus comprising: a.

According to another aspect of the present invention, in the audio data comparison method for comparing whether or not the comparison target audio data having the audio fingerprint data generated by the above method and the original audio data, the original audio data is compared Dividing the data into at least one segment so as to correspond to the size of the target audio data; Calculating a distance for each segment between audio fingerprint data of the comparison target audio data and audio fingerprint data of each of the at least one segment; Determining a maximum value of the distances calculated for each segment; And comparing the maximum value with a magnitude of the threshold value, and when the maximum value is larger than the threshold value, determines that the compared audio data is equal to the original audio data, and when the maximum value is smaller than the threshold value, comparing the maximum value with a threshold value. And determining that the target audio data is not identical to the original audio data.

The dividing of the original audio data into at least one segment so as to correspond to the size of the comparison target audio data may be performed such that one segment to be divided and a segment of a next time interval adjacent to each other have a portion common to each other. Can be.

The calculating of the distance between the audio fingerprint data of the comparison target audio data and the audio fingerprint data of each of the at least one segment may be performed by segmenting the divided audio data of the comparison target audio data with respect to each segment. Sequentially calculating a distance between frame-by-frame audio fingerprint data for each of two frames and frame-by-frame audio fingerprint data for each of the divided second frames of each segment; And calculating the sum of the calculated distances for each segment.

In addition, frame-by-frame audio fingerprint data for each of the divided second frames of the comparison target audio data for each segment and frame-by-frame audio fingerprint data for each of the divided second frames of each segment The step of sequentially calculating the distance between them is generated by feature distribution data constituting frame-by-frame audio fingerprint data for each of the second frames of the comparison target audio data, for each segment. The corresponding frequency data and frequency data generated by the feature distribution data constituting the audio fingerprint data for each frame for each of the divided second frames of each segment are sequentially compared to obtain a minimum value for each second frame. , The sum of the minimum values obtained for each second frame It can be calculated per unit.

The method may further include dividing the calculated sum by the number of second frames.

In addition, when the maximum value is greater than the threshold value, the maximum value is determined to be equal to the original audio data when the maximum value is greater than the threshold value, and when the maximum value is smaller than the threshold value, the comparison value is determined. The determining that the comparison target audio data is not identical to the original audio data may include comparing the size of the original audio data with the size of the comparison audio data when the maximum value is greater than a threshold. Is equal to the size of the original audio data, the comparison target audio is determined to be exactly the same as the original audio data, and when the size of the comparison audio data is smaller than the size of the original audio data, the comparison target Audio remind It can be determined to be partly identical to the original audio data.

According to another aspect of the present invention, in the audio data comparison device for comparing whether or not the comparison target audio data having the audio fingerprint data generated by the above method and the original audio data, the original audio data is compared The segment is divided into at least one segment to correspond to the size of the target audio data, and the distance between the audio fingerprint data of the comparison target audio data and the audio fingerprint data of each of the at least one segment is calculated for each segment, and for each segment. The maximum value of the calculated distance is determined, and the maximum value is compared with the magnitude of the threshold value. When the maximum value is larger than the threshold value, the comparison target audio data is determined to have the same identity as the original audio data, and the maximum value is determined. value Is smaller than the threshold value, the comparison-object audio data may provide an audio data comparison device including the audio data compared to determine that there is no audio data and the original identity.

According to still another aspect of the present invention, there is provided an audio data comprising audio data composed of comparison target audio data having audio fingerprint data generated by the above method and original audio data having audio fingerprint data generated by the above method. An audio data comparison method for comparing equality, the method comprising: dividing each of original audio data included in the audio database into at least one segment so as to correspond to a size of the comparison target audio data; Calculating, for each of the original audio data, the distance between the audio fingerprint data of each segment of each original audio data and the audio fingerprint data of the comparison target audio data for each segment; For all segments of all the original audio data, determining a maximum value of the calculated distance; And comparing the maximum value with a magnitude of a threshold value, and when the maximum value is larger than a threshold value, determines that the comparison target audio data is identical to at least one of the original audio data of the audio database. And if less than a threshold, determining that the compared audio data is not equal to all original audio data of the audio database.

Here, the step of dividing each of the original audio data included in the audio database into at least one segment so as to correspond to the size of the comparison target audio data, the one segment and the segments of the next time interval adjacent to each other It can be divided to have a common part.

Further, for each of the original audio data, calculating the distance between the audio fingerprint data of each segment of the respective original audio data and the audio fingerprint data of the comparison target audio data for each segment may include: For each segment of original audio data, frame-by-frame audio fingerprint data for each of the divided second frames of the comparison audio data and frame-by-frame audio finger for each of the divided second frames of each segment Sequentially calculating corresponding distances between the print data; And calculating, for each of the original audio data, the sum of the calculated distances for each segment.

Further, for each segment of the respective original audio data, the audio fingerprint data for each frame for each of the divided second frames of the comparison target audio data and for each of the divided second frames of the segment Computing the distance between frame-by-frame audio fingerprint data with respect to each segment of the respective original audio data, for each frame, for each frame of each of the second frame of the comparison target audio data Frequency data generated by the feature distribution data constituting the audio fingerprint data and frequency generated by the feature distribution data constituting the audio fingerprint data per frame for each of the divided second frames of each segment Second frame by comparing the data sequentially The minimum value may be obtained for each segment, and the sum of the minimum values obtained for each second frame may be calculated for each segment.

Here, the method may further include dividing the calculated sum by the number of second frames.

The maximum value is compared with the magnitude of the threshold value, and when the maximum value is larger than the threshold value, it is determined that the comparison target audio data is identical to at least one of the original audio data of the audio database. If it is less than this threshold, determining that the comparison target audio data is not equal to all original audio data of the audio database may include determining the size of the original audio data having the maximum value when the maximum value is greater than the threshold value. Comparing the size of the compared audio data and determining that the compared audio is exactly the same as the original audio data having the maximum value when the size of the compared audio data is equal to the size of the original audio data having the maximum value; Doing When the size of the audio data to be compared is smaller than the size of the original audio data having the maximum value, the comparison audio may be determined to be partially the same as the original audio data having the maximum value.

According to still another aspect of the present invention, there is provided an audio data comprising audio data composed of comparison target audio data having audio fingerprint data generated by the above method and original audio data having audio fingerprint data generated by the above method. An audio data comparison device for comparing equality, the audio data comparison apparatus comprising: dividing each of the original audio data included in the audio database into at least one segment to correspond to the size of the comparison target audio data, and comparing the original audio data to the respective original audio data. For each segment, a distance between the audio fingerprint data of each segment of each original audio data and the audio fingerprint data of the comparison target audio data is calculated for each segment, and for all segments of all the original audio data, The maximum value of the calculated distance is determined, and the maximum value is compared with the magnitude of the threshold value. When the maximum value is larger than the threshold value, the comparison target audio data is equal to at least one of the original audio data of the audio database. And an audio data comparison unit for determining that the comparison target audio data is not identical to all original audio data of the audio database when the maximum value is smaller than a threshold.

Next, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram illustrating a coupling relationship between an audio fingerprint generation and audio data comparison apparatus 100 and other components for performing an audio fingerprint data generation method and an audio data comparison method using the same.

Referring to FIG. 1, an audio fingerprint data generation and audio data comparison apparatus 100 for performing an audio fingerprint data generation method and an audio data comparison method using the same may include an audio fingerprint data generation unit 20. , An audio data comparison server 30 and an audio database 40. The audio fingerprint generation and audio data comparison device 100 is combined with the audio data input device 10 and receives audio data from them.

The audio data input device 10 may be, for example, a web page of the Internet, a PC, and a mobile terminal. The audio data input device 10 may include audio data or video data including video data and audio data to the audio fingerprint generation and audio data comparison device 100. send. Here, the audio data input device 10 means all means capable of performing a function for transmitting audio data or video data to the audio fingerprint generation and audio data comparison device 100, and is limited to that shown in FIG. 1. It doesn't happen. For example, in the case of a web page of the Internet in the audio data input device 10, a robot for automatically extracting audio data or video data included in a web page on the Internet in the audio fingerprint generation and audio data comparison device 100 is used. The audio data or video data may be transmitted to the audio fingerprint generation and audio data comparison apparatus 100 by the robot.

The audio fingerprint data generator 20 receives audio data or video data from the audio data input apparatus 10 and generates audio fingerprint data indicating characteristics of the audio data therefrom. The audio fingerprint data generation unit 20 generates audio fingerprint data in advance with respect to the original audio data, transmits them to the audio database 40, and stores them. The audio fingerprint data generation unit 20 compares the original audio data with the audio data to be compared with the original audio data. For example, the audio fingerprint data is generated and transmitted to the audio data comparison server 30 to compare the audio fingerprint data of the original audio data stored in the audio database 40 with the audio data comparison server 30. Make comparisons. Here, when the input data is video data including video data and audio data, the audio fingerprint data generation unit 20 may include means for extracting only audio data from the video data. By extracting only the audio data from the video data by generating audio fingerprint data for the extracted audio data and transmits it to the audio database (40). A detailed method of generating audio fingerprint data will be described later.

The audio database 40 stores the audio fingerprint data for the original audio data generated by the audio fingerprint data generator 20. It also stores the original audio data itself if necessary.

The audio data comparison server 30 compares audio data having audio fingerprint data generated by the audio fingerprint data generation unit 20 with each other to determine whether they match. The audio database 40 storing the audio fingerprint data previously generated is compared with the audio fingerprint data of the comparison target audio data to determine whether they match.

FIG. 2 is a diagram schematically illustrating an entire process of an audio fingerprint data generation method and an audio data comparison method using the same according to the present invention.

Referring to FIG. 2, first, the audio fingerprint data for the original audio data is generated in advance by the audio fingerprint data generation unit 20 of FIG. 1 (S201), and the audio fingerprint of the generated original audio data is generated. Data are stored in the audio database 40 (S203).

Next, the audio fingerprint data generation unit 20 receives the comparison target audio data to be compared from the audio data input device 10 (S205), and generates audio fingerprint data of the comparison target audio data (S207). ), The generated comparison target audio fingerprint data is transmitted to the audio data comparison server 30.

Next, the audio data comparison server 30 compares the audio fingerprint data of the audio data to be compared with the audio fingerprint data of the original audio data included in the audio database 40 and compares the audio data with the audio database 40. In step S209, it is determined whether the audio data is identical to at least one of audio data included in the &quot; Here, comparing with the audio data included in the audio database 40 means that the audio data to be compared is included in the audio database 40 in comparison with all the audio data included in the audio database 40. Means for determining whether or not there is equality to at least one of the audio data, and for determining whether or not equality is compared with any one of the audio data included in the audio database 40. to be.

3 is a flowchart illustrating an entire process of a method for generating audio fingerprint data according to an embodiment of the present invention.

Referring to FIG. 3, first, when the target data for generating audio fingerprint data is video data including video data, only audio data is extracted therefrom (S301). This is a process that is not directly related to the present invention. Any technique for extracting audio data from moving image data may be used. Of course, if the target data for generating the audio fingerprint data is audio data, step S301 may be omitted.

Next, the extracted audio data is normalized to a predetermined frequency band (S303). Normalization can be achieved by sampling conversion by a predetermined sampling frequency, which means that the input audio data can use various types of frequencies, such as 8 kHz, 11 kHz, 16 kHz, 22 kHz, 44 kHz, etc. Because there is. Normalization can be normalized to 16 kHz, for example, and sampling frequencies above 16 kHz contain audio signal components corresponding to high frequencies, which increases the amount of signal to process, which can slow down the audio fingerprint data generation rate. Because.

Next, the normalized audio data is divided into at least one first frame (S305). The first frames refer to a plurality of frames having a length of time for audio data, and the term 'first' frame is used to distinguish that the first frames have a different time interval from the 'second' frame described later. . A method of dividing audio data into at least one first frame will be described with reference to FIG. 4. Referring to FIG. 4, it can be seen that all audio data is composed of one frame for every four sub frame time intervals, and one frame is moved for one sub frame time interval. In FIG. 4, f ₀ represents the first frame, f ₁ represents the second frame, f ₂ represents the third frame, and each frame includes four subframes. Further, f ₁ it is moved by a subframe time interval, and compared to f _0, f ₂ It can be seen that it is shifted by one sub-frame time period than the f _1. Of course, unlike in the case of FIG. 4, it is possible to divide one frame so that there is no overlapping portion, but as in the case of FIG. This is preferable because more accurate audio fingerprint data can be generated with respect to the audio data of.

Next, feature data representing a feature of audio data is extracted for each of the divided first frames as described above (S307). Extracting the feature data extracts feature data representing an inherent characteristic of the audio data itself from the audio data. For example, for each of the divided first frames, the MFCC (Mel Frequency Cepstral Coefficient) and PLPC (Perceptual) used in the prior art are used. A method of extracting a feature vector using at least one combination of a linear prediction coefficient (LPC) or a linear prediction coefficient (LPC) may be used.

For example, a process of extracting feature vectors by the conventional MFCC method will be described below. The analog audio signal is converted to the digital audio signal x (n) through A / D conversion. The digital speech signal is subjected to a digital pre-emphasis filter with a high pass characteristic. The reason for using this filter is to first apply high band filtering to model the frequency characteristics of the human ear / middle ear. To do this. This compensates for the 20 dB / decade attenuation by radiation on the human lips, so that only the vocal characteristics are obtained from the speech signal. Second, it compensates to some extent that the auditory system is sensitive to spectral regions above 1 kHz. The characteristic H (z) of the pre-emphasis filter is as follows, and a may be a value in the range of 0.95 to 0.98.

H (z) = 1-az ^-1

The signal pre-emphasized by the preemphasis filter is divided into frames in units of blocks covering a hamming window. Subsequent processing is performed in units of frames. The frame size is usually 20 ms and the frame shift is 10 ms. An audio signal of one frame is transformed into a frequency domain by using a fast fourier transform (FFT). Divide the frequency band into several filter banks and find the energy in each bank. Logging the band energy and performing a discrete cosine transform (DCT) yields the final MFCC. The shape of the filter bank and the method of setting the center frequency are determined in consideration of the acoustic characteristics of the ear (frequency characteristics in the snail tube). Referring to FIG. 5, a triangular filter is used in FIG. 5, and the center frequency is linearly located up to 1 kHz, and there are 20 banks distributed on a mel scale. For example, ₁₂ MFCC coefficients from c ₁ to c ₁₂ may be used, and separately obtained frame log energy may be used to obtain a 13 th order vector. In addition, a 39-dimensional feature vector can be obtained by combining the 13th order DELTA considering the difference from the previous MFCC frame and the 13rd order ACCELERATION considering the difference between the first order DELTA.

In this manner, when feature data (feature vector) is extracted for each first frame, feature distribution data indicating a distribution feature of feature data is generated using feature data extracted for each of the divided first frames (S309). ).

The feature distribution data is data representing a feature in which feature data is distributed. The feature distribution data may be generated by referring to a codebook generated in advance. Here, the codebook extracts feature vectors of a plurality of audio data in advance, distributes them in the vector space, groups the feature vectors in the vector space, calculates an average value of the feature vectors included in each group, and calculates an average value of the feature vectors in each group. Consists of data that stores index values.

FIG. 6 is a diagram for describing a process of generating such a codebook. For convenience of description, the codebook for generating a two-dimensional feature vector will be described as an example.

Referring to FIG. 6, feature vectors for a plurality of audio data acquired in advance in a two-dimensional vector space are distributed and grouped. As a criterion for grouping, a method of grouping feature vectors adjacent to each other in a vector space may be used. For example, a known k-Mean algorithm, a Linde-Buzo-Gray (LBB) method, and the like may be used. A total of seven groups are illustrated in FIG. 6, and an average value of feature vectors belonging to the group can be obtained for each group. When the average value is obtained for each group, a codebook of the form shown in FIG. 7 may be configured. Referring to FIG. 7, it can be seen that the codebook is composed of indexes and average values of each group. Here, the codebook may be, for example, 64th, 128th, or 256th order, and as the order increases, that is, as the number of groups increases, the codebook may be configured more precisely, and as a result, the distribution feature data of the feature vector may also be Can be obtained precisely.

Referring to the previously generated codebook, the index value of the group to which the feature data (feature vector) for each of the divided first frames may belong may be calculated. Finding a group to which the feature vector should belong may use a method of determining a group having a minimum value as a group to which the feature vector belongs by calculating a distance between the feature vector of each of the first frames and the average value of each group of the codebook. When the group to belong to is determined, the index value of the group is generated as the feature distribution data for the feature vector.

Next, the audio data is divided into at least one or more second frames (S311). Dividing the audio data into second frames may be divided by a method as described above in operation S305. However, there is a difference in that the length of each of the divided second frames has a larger value than that of the first frame. For example, when a frame length of one first frame is 20 ms and one first frame is composed of four sub frames, one sub frame length is 5 ms. In this case, when the total length of one second frame is 4s, 200 first frames are included in one second frame. The subframe of the second frame may be configured, for example, in units of 1s. In this case, the next frame adjacent to one second frame has an overlapping portion at intervals of 1s.

As described above, after the audio data is divided into at least one or more second frames, second fingerprint-specific audio fingerprint data is generated for each of the second frames (S313). Here, if the audio fingerprint data for the i-th second frame is defined as h _i , h _i may be defined as (i, o ₁ , o ₂ , o ₃ , ... o _L ). Here, i denotes the number (order) of the second frame, and L denotes the order of the codebook as described above. o ₁ , o ₂ , o ₃ , ... o _L indicates how many times among the feature distribution data of the first frames in which the indexes of the group of codebooks in the range of 1 to L are included in one second frame Refers to frequency data.

For example, when the audio data is divided into T first frames in step S305, the feature data for each first frame may be referred to as f ₁ , f ₂ , f ₃ ... f _T , The feature distribution data for may be c ₁ , c ₂ , c ₃ , ... c _T , respectively. Here, c ₁ , c ₂ , c ₃ , ... c _T are index values of the codebooks as described above, respectively, and have values ranging from 0 to L-1 in the case of L-order codebooks. In this case, o ₁ means frequency data indicating how many times a corresponding value of the first index value of the codebook has appeared among feature distribution data of the first frames included in one second frame. For example, when one second frame consists of ten first frames, the first of the codebook among the feature distribution data c ₁ , c ₂ ,... C ₁₀ of each of the ten first frames. We can get o ₁ by counting how many times the index is included, and we can get o ₂ by counting how many times the second index is included.

In this manner, o ₁ , o ₂ , o ₃ , ... o _L for the i-th second frame is obtained, and (t, o ₁ , o ₂ , o ₃ , ... o _L ) In this form, audio fingerprint data h _i for the i-th second frame may be obtained.

After the audio fingerprint data is obtained for each of the second frames, the audio fingerprint data for the entire audio data is generated by the audio fingerprint data of the entire second frames (S315). If the audio fingerprint data for the entire audio data is H, it may be defined as H = {h ₁ , h ₂ , h ₃ ... H _N }. Where N corresponds to the number of second frames.

FIG. 8 is a diagram for explaining an audio fingerprint generation process described with reference to FIGS. 3 to 7.

Referring to FIG. 8, first, the entire audio data is divided into at least one or more first frames (S501), and feature data is sequentially extracted from the first frame among the divided first frames by the above-described method ( Step S307 of Figure 3). When the feature data is extracted for each of the first frames, the feature distribution data for the feature data is generated with reference to the codebook (steps S309 and S803 of FIG. 3).

When feature distribution data for each of the first frames is generated, audio fingerprint data for each second frame is generated for each of the second frames including the at least one first frame (steps S313 and S805 of FIG. 3). . As described above, the audio fingerprint data for each second frame is composed of frequency data of the feature distribution data representing the distribution characteristic of the feature data, and they can be expressed in the form of a histogram as shown in FIG. 8. The audio fingerprint data generation method according to the present invention may be referred to as a histogram-based audio fingerprint data generation method.

FIG. 9 illustrates an example of a method of configuring audio fingerprint data for each second frame from feature distribution data in the above-described process of generating audio fingerprint data.

In FIG. 9, assuming that the number of first frames for audio data is 12, the feature distribution data for audio data may be defined as a set of feature distribution data of twelve first frames as shown in FIG. 9. . In the case of FIG. 9, it is assumed that the number of indices, that is, the number of groups grouped in the codebook, is four. In this case, each feature distribution data has a value of any one of numbers in the range of 1 to 4 as shown.

Assuming that one second frame includes four first frames and one second frame is slidably moved by a time interval of one first frame, a total of nine second frames (h ₁ to h ₉ ) can be configured. Audio fingerprint data h _i for each second frame is represented by five data of {i, o ₁ , o ₂ , o ₃ , o ₄ }, where the first i is the number of the second frame, o ₁ , o ₂ , o ₃ , o ₄ As described above, frequency data representing how many times the feature distribution data of the first frames included in the second frame is shown.

For example, in FIG. 9, h ₁ is represented by five data of {1,1,2,1,0}, where the first 1 is data indicating that the first second frame is a second frame number, and then 1 is data indicating that the value of index 1 appears one time among the feature distribution data 2, 3, 2, and 1 of the first four first frames C ₁ to C ₄ . Similarly, the third data of h ₁ , 2, indicates that the value of index 2 appears twice in the feature distribution data (2, 3, 2, ₁ ) of the first four first frames from c ₁ to c ₄ , beonjje data in one of four h ₁ indicates that the c ~ c was _first characterized in the distribution data (2,3,2,1) of the first frame _4. the first four to the value of the index number 3 1, ₁ h The fifth data of 0 is data indicating that the value of the index 4 appears 0 times among the feature distribution data (2, 3, 2, 1) of the first four first frames from c ₁ to c ₄ .

In this manner, audio fingerprint data of h ₁ can be obtained, and in the same manner, audio fingerprint data for each second frame for each of h ₂ to h ₉ can be obtained, and thus audio for the entire audio data can be obtained. Fingerprint data can be configured.

Hereinafter, a method of comparing the audio data using the audio fingerprint data generated by the method described above and determining whether or not the same will be described.

10 is a flowchart illustrating an embodiment of a method for comparing audio data using audio fingerprint data according to the present invention.

First, prior to comparing the audio data, audio fingerprint data for the audio data is generated by the method described with reference to FIGS. 1 to 9. The audio fingerprint data is generated in advance for the original audio data and stored in the audio database 40 (see FIG. 1), and the audio fingerprint data is also generated for the compared audio data to be compared with them. The audio data comparison server 30 transmits the audio fingerprint data generated for the audio data to be compared and the audio fingerprints of the original audio data stored in the audio database 40. The data are compared by the following procedure. Of course, it is also possible to compare the original audio data stored in the audio database 40, it is also possible to directly compare two or more audio data in the audio data comparison server 30 without storing separately in the audio database 40 Do.

Referring to FIG. 10, first, original audio data is divided into at least one segment to correspond to the size of audio data to be compared (S1001). Referring to FIG. 11, in the case of (a) in which the size of the comparison target audio data and the size of the original audio data are the same, there is no need to divide the original audio data into a plurality of segments, and in this case, only one segment exists. . However, as in the case of (b), when the size of the original audio data is larger than the audio data to be compared, as shown in FIG. 11, the original audio data is k ₁ such that the size of one segment corresponds to the size of the audio data to be compared. It can be split into multiple segments of, k ₂ , k ₃ .. At this time, segmentation, as described in the above-described first and second frame division, it is more complicated to divide so that one segment to be divided and the segment of the next time interval adjacent to each other have a common part with each other Is preferred since it is possible. As the number of segments increases, audio fingerprint data can be compared for each narrower section, thereby allowing more sophisticated comparison.

Next, the distance between the audio fingerprint data of the comparison target audio data and the audio fingerprint data of each of the at least one segment divided as described above is calculated for each segment (S1003). Calculating the distance between audio fingerprint data may be performed in the following manner.

As described above, the audio fingerprint data generated by the present invention is composed of a set H = {h ₁ , h ₂ , h ₃ .... h _N } of audio fingerprints per second frame (where N Is the number of second frames), and audio fingerprint data h _{i for} each second frame may be defined as h _{i =} (i, o ₁ , o ₂ , o ₃ ,... O _L ). Here, i denotes a sequence number of the second frame, L denotes the order of the codebook as described above, o ₁ , o ₂ , o ₃ , ... o _L are each the first frame included in the second frame It means frequency data indicating how many times the feature distribution data of the frames (that is, the index value of the codebook) appears in one second frame for each index.

In this case, the audio fingerprint data of the audio data to be compared is referred to as H = {h ₁ , h ₂ , h ₃ .... h _N }, and the audio fingerprint data of the kth segment of the original audio data is referred to as H ^k = If {h ^k ₁ , h ^k ₂ , h ^k ₃ .... h ^k _N }, the distance between them can be calculated by the following equation.

Equation 1.

This formula means that the distance between two audio fingerprint data to be compared is the first to last audio fingerprint data for each second frame constituting each of the two audio fingerprint data to be compared. The distances d between the audio fingerprint data for each second frame are calculated and corresponded sequentially. Here, the distance d between the audio fingerprint data for each second frame may be calculated by the following equation.

Equation 2

(여기서, D는 제2 프레임의 개수이며, L은 코드북의 차수)

Where D is the number of second frames and L is the order of the codebook.

The formula means that the frequency data o _l representing the frequency at which feature distribution data (index value of the codebook) constituting corresponding audio fingerprint data for each second frame appears in one second frame from the beginning. The minimum values are calculated by sequential correspondence until the end, and the sums are divided by the number of frames. Here, the process of dividing by the number of frames may be omitted.

For example, suppose h _i = {3,2,3,5,3,7,9,23} and h ^k _i = {3,4,5,2,23,56,3,2}. Since the previous data is the frame number, except for this, if the minimum value is obtained by sequentially matching the second value, 2,3,2,3,7,3,2 can be obtained, and the sum thereof is 22, which is the second value. By dividing by the number of frames, the distance between the original audio data and the audio fingerprint data of the i th second frames of the comparison target audio data may be obtained. In this way, after calculating the distance for each second frame and calculating the sum of the distances for each second frame according to Equation 1, the audio fingerprint data of the comparison target audio data and the k-th segment of the original audio data are compared. The distance between the print data can be obtained. By repeating this process for all segments, the distance between the audio fingerprint data of all segments of the original audio data and the audio fingerprint data of the comparison target audio data can be calculated for each segment.

In this way, if the distance is calculated for each segment for each segment, the maximum value among them is determined (S1005).

When the maximum value is determined, it is determined whether or not the comparison target audio data is identical to the original audio data by comparing the maximum value with a preset threshold value (S1007). The equality means that the audio data to be compared is exactly the same or contained (partially identical) with the original audio data.

If the maximum value is greater than the threshold as a result of the comparison, the original audio data is compared with the size of the comparison target audio data in order to determine whether they are completely identical or partially identical (S1009). As a result of the comparison, if the size of the compared audio data is equal to the size of the original audio data, the compared audio is determined to be exactly the same as the original audio data (S1011), and the size of the compared audio data is smaller than the size of the original audio data. In the case, the comparison target audio is determined to be partially identical to the original audio data (S1013).

On the other hand, if the maximum value is less than the threshold in step S1007, it is determined that the comparison target audio data is not the same as the original audio data (S1015).

Meanwhile, an embodiment of the method for comparing audio data using the audio fingerprint data according to the present invention described with reference to FIGS. 10 to 11 is almost the same when comparing with an audio database including at least one or more original audio data. You can apply it as it is. In comparison with the audio database as a whole, the process described with reference to FIGS. 10 to 11 is repeated for each audio data included in the audio database, which is illustrated in the flowchart of FIG. 12. 12 is different from the flowchart of FIG. 10 in that the processes of FIGS. 10 to 11 are repeated for each of the audio data included in the audio database.

Therefore, in the flowchart of FIG. 12, dividing original audio data into a plurality of segments is performed for each of all original audio data included in the audio database (S1201, see step S1001 of FIG. 10), and each original audio data The distance between the audio fingerprint data for each segment is calculated (S1203, see step S1003 of FIG. 10).

Further, determining the maximum value of the distance in step S1205 determines the maximum value from the distances obtained for all the audio data and for every segment. In addition, comparing the file size in step S1209, comparing the size of the original audio data having the determined maximum value with the comparison target audio data among the audio data included in the audio database, and if the size is the same The audio data is determined to be exactly the same as at least one of the audio data included in the audio database (S1213). When the size of the original audio data is larger than the compared audio data, the compared audio data is included in the audio database. It is determined to be partially identical to at least one of the data (S1211).

On the other hand, if the maximum value is less than the threshold in the step (S1207) it is determined that the comparison target audio data is not the same as all audio data included in the audio database (S1215).

Next, an example of a method for determining the threshold in FIGS. 10 and 12 will be described.

FIG. 13 is a distribution diagram of the maximum value of the distance between the original audio data and the audio fingerprint data of the comparison target audio data. The number of sets of the original audio data included in the audio database is referred to as Q. FIG. When there are P audio data included in the audio database among the target audio data and R audio data not included in the audio database among the compared audio data, P compared audio data and R compared audio data And distributions for the maximum value of the distance between the audio fingerprint data of the original audio data included in the audio database are shown as S1 and S2 of FIG. 13, respectively.

At this time, the threshold Th can be determined by the following equation.

Equation 3.

Th = μ ₁ -ασ ₁

Here, μ ₁ is an average value of the S1 curve, sigma ₁ is a standard deviation of the S1 curve, and α is a constant, and an appropriate value can be used depending on statistical characteristics. For example, α may use a real value between 1 and 7. The formula for determining the threshold is exemplary and may be determined using various other statistical methods according to the distribution characteristics of the audio database and the audio data to be compared.

In the above, the configuration of the present invention has been described with reference to the preferred embodiment of the present invention, of course, the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains It will be apparent that various modifications, changes and improvements can be made within the scope of the invention as seen with reference to the appended claims and the accompanying drawings. For example, although the present invention has been described with reference to an embodiment applied to audio data, as described above, the present invention may also be applied to video data including audio and video signals. That is, by extracting an audio signal from the video data to generate an audio fingerprint with respect to the audio signal and comparing the same, it is possible to easily determine whether the video data is identical.

According to the present invention, it is possible to provide a method and apparatus for generating audio fingerprint data, which is remarkably improved compared to the conventional audio fingerprint generation method and whose generation method is relatively simple.

In addition, according to the present invention, audio fingerprint data is generated using the distribution characteristic data of the characteristic data of the audio data by analyzing the audio data, so that audio characteristics capable of reflecting the characteristics of the audio data more precisely than in the conventional method are provided. A fingerprint data generating method and apparatus may be provided.

In addition, according to the present invention, when comparing the audio data having the audio fingerprint data generated by the method and apparatus for generating audio fingerprint data as described above, it is possible to obtain a quick and accurate comparison result compared to the conventional method It is possible to provide a method and apparatus for comparing audio data that can significantly reduce a determination error of identity.

Further, according to the present invention, since the audio fingerprint data can be divided into segments and compared and judged for each segment, the present invention can be applied even when the original audio data and the comparison target audio data are partially the same. It is possible to provide a method and apparatus for comparing audio data that can be detected.

In addition, according to the present invention, by extracting the audio signal from the video data including the video signal and the audio signal to generate and compare audio fingerprint data for the audio signal whether the unauthorized copying and theft of the video data as well as the audio data The present invention can provide a method and device for easily comparing audio data, which can be easily applied even when only a part of video data is edited and stolen.

Claims (25)

An audio fingerprint data generation method for generating audio fingerprint data from audio data, the method comprising: Dividing audio data into at least one first frame; Extracting feature data for each of the divided first frames; Generating feature distribution data representing a distribution feature of the feature data for each of the divided first frames; Dividing the audio data into at least one second frame; Generating audio fingerprint data for each of the second frames using the feature distribution data generated for the first frames; And Generating audio fingerprint data for the entirety of the audio data by a set of audio fingerprint data for each of the second frames Audio fingerprint data generation method comprising a. The method of claim 1, And the audio data is extracted from video data including a video signal and an audio signal. The method of claim 1, And prior to dividing the audio data into at least one first frame, normalizing the audio data to a predetermined frequency band. The method of claim 1, And dividing the audio data into at least one first frame comprises dividing the divided one frame and a frame having a next time interval adjacent to each other to have a portion in common with each other. The method of claim 1, And dividing the audio data into at least one second frame comprises dividing the divided one frame so that the frames having the next time interval adjacent to each other have a portion common to each other. The method of claim 1, The length of each of the second frame is longer than the length of each of the first frame. The method of claim 1, Extracting feature data for each of the divided first frames may include, for each of the divided first frames, a Mel Frequency Cepstral Coefficient (MFCC), a Perceptual Linear Prediction Coefficient (PLPC), or a Linear Prediction Coefficient (LPC). And extracting the feature vector using at least one combination thereof. The method of claim 7, wherein Generating feature distribution data representing a distribution feature of the feature data for each of the divided first frames comprises: grouping feature vectors into at least one group and storing an index value for each group The method of claim 1, wherein the feature distribution data is generated by calculating an index value for each of the divided first frames. The method of claim 8, The codebook has an average value of feature vectors for each group, and calculates an index value based on a distance between feature vectors for each of the divided first frames and an average value of feature vectors for each group. Fingerprint data generation method. The method of claim 8, Generating audio fingerprint data for each of the second frames by using the feature distribution data generated for the first frames; Audio fingerprint data generation for each of the second frames is generated by calculating a frequency of feature distribution data for each of the first frames included in each of the second frames. Way. An audio fingerprint data generation device for generating audio fingerprint data from audio data, the apparatus comprising: A feature distribution that divides audio data into one or more first frames, extracts feature data for each of the divided first frames, and distributes the feature of the feature data for each of the divided first frames. Generating data, dividing the audio data into at least one or more second frames, and using the feature distribution data generated for the first frames, to generate audio fingerprint data for each of the second frames And an audio fingerprint data generator configured to generate audio fingerprint data for the entire audio data by a set of audio fingerprint data for each of the second frames. Audio fingerprint data generation device comprising a. An audio data comparison method for comparing the equality of audio data to be compared with original audio data having audio fingerprint data generated by the method of any one of claims 1 to 10, Dividing the original audio data into at least one segment so as to correspond to a size of the comparison target audio data; Calculating a distance for each segment between audio fingerprint data of the comparison target audio data and audio fingerprint data of each of the at least one segment; Determining a maximum value of the distances calculated for each segment; And The maximum value is compared with the magnitude of the threshold value, and when the maximum value is larger than the threshold value, the comparison target audio data is determined to be identical to the original audio data, and when the maximum value is smaller than the threshold value, the comparison object is determined. Determining that audio data is not identical to the original audio data Audio data comparison method comprising a. The method of claim 12, The dividing of the original audio data into at least one segment so as to correspond to the size of the comparison target audio data may be performed such that one segment to be divided and a segment of a next time interval adjacent to each other have a portion in common with each other. How to compare audio data. The method of claim 12, Computing the distance between the audio fingerprint data of the comparison target audio data and the audio fingerprint data of each of the at least one segment for each segment, Between frame-specific audio fingerprint data for each of the divided second frames of the comparison target audio data for each segment, and frame-by-frame audio fingerprint data for each of the divided second frames of each segment. Sequentially calculating corresponding distances of s; And Calculating a sum of the calculated distances for each of the segments Audio data comparison method comprising a. The method of claim 14, Between frame-specific audio fingerprint data for each of the divided second frames of the comparison target audio data for each segment, and frame-by-frame audio fingerprint data for each of the divided second frames of each segment. Computing the distances of sequentially For each segment, frequency data generated by feature distribution data constituting frame-by-frame audio fingerprint data for each of the second frames of the comparison target audio data, and the divided second frame of each segment Comparing the frequency data generated by the feature distribution data constituting the audio fingerprint data for each frame with respect to each of them sequentially and obtaining a minimum value for each second frame, and calculating the sum of the minimum values obtained for each second frame for each segment. Audio data comparison method characterized in that. The method of claim 15, And dividing the calculated sum by the number of second frames. The method of claim 12, The maximum value is compared with the magnitude of the threshold value, and when the maximum value is larger than the threshold value, the comparison target audio data is determined to be identical to the original audio data, and when the maximum value is smaller than the threshold value, the comparison object is determined. Determining that the audio data is not the same as the original audio data, When the maximum value is larger than a threshold, the size of the original audio data is compared with the size of the comparison audio data. The audio data is determined to be exactly the same as the original audio data, and when the size of the comparison audio data is smaller than the size of the original audio data, the comparison audio is determined to be partially identical to the original audio data. Comparison method. An audio data comparison device for comparing the equality of audio data to be compared with original audio data having audio fingerprint data generated by the method of any one of claims 1 to 10, The original audio data is divided into at least one segment to correspond to the size of the comparison target audio data, and a distance between the audio fingerprint data of the comparison target audio data and the audio fingerprint data of each of the at least one segment is determined. Calculate the segment, determine the maximum value of the distance calculated for each segment, compare the maximum value with the magnitude of the threshold, and when the maximum value is larger than the threshold, the comparison target audio data is identical to the original audio data. Audio data comparison unit for determining that there is no equality with the original audio data when the maximum value is smaller than a threshold value. Audio data comparison device comprising a. An audio data comparison method for comparing the identity of an audio database composed of original audio data with a comparison target audio data having audio fingerprint data generated by the method of any one of claims 1 to 10, Dividing each of the original audio data included in the audio database into at least one segment so as to correspond to the size of the comparison target audio data; Calculating, for each of the original audio data, the distance between the audio fingerprint data of each segment of each original audio data and the audio fingerprint data of the comparison target audio data for each segment; For all segments of all the original audio data, determining a maximum value of the calculated distance; And The maximum value is compared with a magnitude of a threshold value, and when the maximum value is larger than a threshold value, it is determined that the comparison target audio data is equal to at least one of original audio data of the audio database, and the maximum value is a threshold value. If smaller, determining that the compared audio data is not identical to all original audio data in the audio database. Audio data comparison method comprising a. The method of claim 19, The step of dividing each of the original audio data included in the audio database into at least one segment so as to correspond to the size of the comparison target audio data may include a portion in which one segment and a segment of an adjacent next time interval are common to each other. The audio data comparison method, characterized in that divided to have. The method of claim 19, For each of the original audio data, calculating the distance between the audio fingerprint data of each segment of each original audio data and the audio fingerprint data of the comparison target audio data for each segment, For each segment of the respective original audio data, frame-by-frame audio fingerprint data for each of the divided second frames of the compared audio data and frame for each of the divided second frames of each segment Sequentially calculating corresponding distances between respective audio fingerprint data; And Calculating, for each of the original audio data, the sum of the calculated distances for each segment; Audio data comparison method comprising a. The method of claim 21, For each segment of the respective original audio data, frame-by-frame audio fingerprint data for each of the divided second frames of the compared audio data and frame for each of the divided second frames of each segment Computing the distance between the respective audio fingerprint data in sequence correspondingly, Frequency data generated by feature distribution data constituting frame-by-frame audio fingerprint data for each of the second frames of the comparison target audio data, for the respective segment of the respective original audio data; A minimum value is obtained for each second frame by sequentially comparing the frequency data generated by the feature distribution data constituting the audio fingerprint data per frame for each of the divided second frames of the segment, and obtaining the minimum value for each second frame. Audio data comparison method, characterized in that for calculating the sum of each segment. The method of claim 22, And dividing the calculated sum by the number of second frames. The method of claim 19, The maximum value is compared with a magnitude of a threshold value, and when the maximum value is larger than a threshold value, it is determined that the comparison target audio data is equal to at least one of original audio data of the audio database, and the maximum value is a threshold value. If it is smaller, determining that the compared audio data is not identical to all original audio data of the audio database, If the maximum value is larger than a threshold, the size of the original audio data having the maximum value is compared with the size of the comparison audio data so that the size of the comparison audio data is equal to the size of the original audio data having the maximum value. In this case, the comparison target audio is determined to be exactly the same as the original audio data having the maximum value, and when the size of the comparison audio data is smaller than the size of the original audio data having the maximum value, the comparison audio is the A method of comparing audio data, characterized in that it is determined to be partially identical to original audio data having a maximum value. An audio data comparison device for comparing the identity of a comparison target audio data having audio fingerprint data generated by the method of any one of claims 1 to 10 with an audio database composed of original audio data. Each of the original audio data included in the audio database is divided into at least one segment to correspond to the size of the comparison target audio data, and for each of the original audio data, audio of each segment of each original audio data The distance between the fingerprint data and the audio fingerprint data of the comparison target audio data is calculated for each segment, and for each segment of all the original audio data, the maximum value of the calculated distance is determined, and the maximum value and the threshold value are determined. When the maximum value is greater than a threshold, it is determined that the comparison target audio data is the same as at least one of the original audio data of the audio database, and when the maximum value is smaller than a threshold, Group compared with the audio data is the audio data to determine that there is no any original audio data and the identity of the audio database comparison unit Audio data comparison device comprising a.

Images