KR102447554B1 - Method and apparatus for identifying audio based on audio fingerprint matching - Google Patents

Abstract

An audio fingerprint-based sound source recognition method is disclosed. The method includes the steps of receiving a query sound source, normalizing the query sound source to a preset sampling rate, increasing the speed of the normalized query sound source by r times, a time-frequency histogram for the query sound source of the increased speed, 1/r-fold reduction, generating an audio fingerprint of the query sound source based on the reduced histogram, and comparing the audio fingerprint of the generated query sound source with the audio fingerprints of the pre-stored reference sound sources obtaining information.

Description

Sound source recognition method and device based on audio fingerprint matching {METHOD AND APPARATUS FOR IDENTIFYING AUDIO BASED ON AUDIO FINGERPRINT MATCHING}

The present disclosure relates to a method and apparatus for recognizing a sound source based on audio fingerprint matching, and more particularly, to a method and apparatus for recognizing a sound source capable of reducing the amount of computation during audio fingerprint matching.

Many sound sources are used in TV or radio broadcast content, movies, and the like, and the need for automatic sound recognition is increasing. For example, music copyright holders want to obtain evidence of how much their music is broadcast on radio or television and how much copyright fees can be claimed.

As a sound source recognition technology, an audio finger print comparison technology is used. The audio fingerprint refers to data that can describe the characteristics of audio data, reflects the inherent characteristics of the signal itself, and may be information indicating characteristics such as frequency and amplitude. This characteristic information is different from metadata in that it reflects the characteristics of the content signal itself, unlike text-based metadata.

As such, it is used in methods such as comparing audio fingerprints to determine whether a sound source has been used without permission or to search for a sound source.

However, in order to read the sound source, it is necessary to compare the fingerprint of millions of sound sources stored in the database with the fingerprint of the sound source to be compared, which is a task that takes a considerable load in terms of time and computational amount. to be. Accordingly, there has been a demand to reduce the speed of such operations.

The present disclosure is in accordance with the above-described necessity, and an object of the present disclosure is to provide a method and apparatus for recognizing a sound source capable of reducing the amount of computation when matching an audio fingerprint.

The present disclosure is intended to solve the above problems, and an audio fingerprint-based sound source recognition method according to an embodiment of the present disclosure includes the steps of receiving a query sound source, normalizing the query sound source to a preset sampling rate; Increasing the speed of the normalized query sound source by r times, reducing the time-frequency histogram for the query sound source of the increased speed by 1/r times, Audio finger of the query sound source based on the reduced histogram generating a print and comparing the generated audio fingerprint of the query sound source with audio fingerprints of pre-stored reference sound sources to obtain information about the query sound source.

In this case, the increasing may include downsampling with respect to the time axis of the time-frequency histogram of the query sound source.

On the other hand, the frequency range of the time-frequency histogram of the normalized query sound source is half of the preset sampling rate, and the increasing includes maintaining the frequency range of the time-frequency histogram of the normalized query sound source and increasing the speed can do it

In this case, the preset sampling rate may be 32 kHz, the frequency range of the time-frequency histogram of the normalized query sound source may be 16 kHz, and the frequency range of the histogram reduced in the reduction step may be 8 kHz.

Meanwhile, the audio fingerprints of the pre-stored reference sound sources may be generated in a manner corresponding to the generation of the audio fingerprints of the query sound source.

On the other hand, as a non-transitory computer-readable medium including computer-executable instructions according to an embodiment of the present disclosure, the computer-executable instructions, when executed by a processor, cause the processor to perform the steps of the method described above. can do it..

The problem to be solved by the present disclosure is not limited to the above-mentioned problems, and the problems not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the present specification and the accompanying drawings. .

According to the above-described microfluidic device, it is possible to reduce the number of matching operations when performing a comparison operation between audio fingerprints, and as a result, it is possible to improve the sound source recognition speed.

Effects of the present disclosure are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description.

1 is a view for explaining a sound source recognition system according to an embodiment of the present disclosure;
2 is a diagram for explaining a method for increasing the speed of audio fingerprint matching;
3 is a diagram for explaining a signal change when an OLA algorithm and a downsampling algorithm are applied to increase the sound source speed;
4 is a view for explaining a comparison between the existing method and the method according to the present disclosure;
5 is a flowchart illustrating a method for recognizing a sound source according to an embodiment of the present disclosure.

Hereinafter, specific embodiments of the present disclosure will be described in detail with reference to the drawings. However, the spirit of the present disclosure is not limited to the presented embodiment, and those skilled in the art who understand the spirit of the present disclosure may add, change, delete, etc. other components within the scope of the same spirit, through other degenerative embodiments or Other embodiments included within the scope of the present disclosure may be easily proposed, but this will also be included within the scope of the present disclosure.

Among the terms used in the present disclosure, terms defined in a general dictionary may be interpreted with the same or similar meaning as the meaning in the context of the related art, and unless explicitly defined in the present disclosure, ideal or excessively formal meanings is not interpreted as If there is no specific definition of the term, it may be interpreted based on the general content of the present specification and common technical knowledge in the art.

The same reference numbers or reference numerals in each of the accompanying drawings indicate parts or components that perform substantially the same functions. For convenience of description and understanding, the same reference numbers or reference numerals are used in different embodiments. That is, even though all components having the same reference number are illustrated in a plurality of drawings, the plurality of drawings do not mean one embodiment.

Also, in the present disclosure, terms including an ordinal number such as “first” and “second” may be used to distinguish between elements. This ordinal number is used to distinguish the same or similar elements from each other, and the meaning of the term should not be construed as limited due to the use of the ordinal number. As an example, the components combined with such an ordinal number should not be limited in the order of use or arrangement by the number. If necessary, each ordinal number may be used interchangeably.

In this disclosure, the singular expression includes the plural expression unless the context clearly dictates otherwise. In this application, terms such as "comprises" or "consisting of" are intended to designate the presence of a feature, number, step, action, component, part, or combination thereof, but one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

1 is a block diagram of a system for performing a method for recognizing a sound source based on audio fingerprint matching according to an embodiment of the present disclosure.

Referring to FIG. 1 , the system 1000 includes a content receiving unit 100 , an audio fingerprint generating unit 200 , a reference database 300 , an audio fingerprint comparing unit 400 , and a result output unit 500 . can do. Some of these components may be omitted or other components apparent to those skilled in the art may be included.

First, the content receiving unit 100 may receive reference sound sources. In addition, the audio fingerprint generation unit 200 may generate audio fingerprints from reference sound sources and store them in advance in the reference database 300 . In this case, the reference database 300 may be stored in association with, for example, a song title, a singer's name, etc. as information on each of the audio fingerprints.

In addition, the content receiving unit 100 may receive a query sound source (recognized sound source) and transmit it to the audio fingerprint generation unit 200, and the fingerprint generation unit 200 generates an audio fingerprint therefrom to generate an audio fingerprint. may be transmitted to the comparator 400 .

Thereafter, the audio fingerprint comparison unit 400 compares the fingerprint of the query sound source with the fingerprints of the reference database 300 , and searches for a fingerprint identical to the fingerprint of the query sound source from the reference database 300 . can do. When the search is performed, information on the sound source corresponding to the searched fingerprint may be transmitted to the result output unit 500 .

For example, in order to read the music from the broadcast, the audio fingerprint comparison unit 400 needs to search for the same as the fingerprint of the query sound source from millions of fingerprints stored in the reference database 300, which It is a task that takes a significant load in terms of time or computational amount. The present disclosure proposes a method for increasing the speed of such a search.

As a method for increasing the search speed, when the length of the sound source is shortened, that is, the reproduction speed is increased to extract the fingerprint, the amount of search operation can be reduced. Specifically, if the speed of the sound source is increased by r (r>1) times, the amount of matching operation is reduced to 1/r ² . This will be described with reference to FIG. 2 .

Referring to FIG. 2 , it is possible to determine whether matching by obtaining similarity images for two sound sources. The horizontal axis of the similarity image is a fingerprint (r ₁ , r ₂ ,...r _n ) generated from the reference sound source, and the vertical axis is When a fingerprint (q ₁ , q ₂ , ... q _n ) generated from a query sound source is called, a place with high similarity appears in a dark form, and when a straight section connecting these dark places is detected, that is, linear If there is a corresponding relationship related to , it can be determined that they match each other. For this, n X m similarity calculations are required. In this case, if the speed of the reference sound source and the query sound source is doubled, the similarity operation is performed n/2 X m/2 times, so that the number of matching operations is reduced to 1/4, and the operation speed can be much faster.

On the other hand, the algorithm for increasing the speed of the sound source can be largely divided into a method of recombining samples and a processing method for each frame.

A representative sample recombination method is a down-sampling method, and a representative frame-by-frame processing method is OLA (Overlap and Add) and SOLA (Synchronous Overlap and Add) algorithms. The basic processing process of SOLA is the same as that of OLA, but it is different in that the calculation formula for finding the processing position of OLA is compared in all positions to find it.

3 is a diagram for explaining a signal change when an OLA algorithm and a down-sampling algorithm are applied to increase the speed.

Referring to FIG. 3 , in the case of two identical sound sources, the similarity is high in the original speed, but when OLA is applied, the speed is changed while maintaining the spectrum. There is a problem to do. Therefore, it is difficult to apply OLA or SOLA algorithms to audio fingerprint extraction.

On the other hand, when (time axis) down-sampling is performed, the spectrum is expanded, so the frequency range of feature extraction is reduced. Accordingly, a method for solving the problem of using the downsampling method to increase the speed of the sound source and reducing the feature extraction frequency range will be described below with reference to FIG. 4 .

4 is a diagram for explaining a comparison between an existing method and a method according to the present disclosure. In the conventional method, a sample rate of a sound source is first normalized to a predetermined frequency band. Normalization can be done by sampling conversion by a predetermined sampling frequency, and since the input sound sources can use various types of frequencies, such as 8 kHz, 11 kHz, 16 kHz, 22 kHz, and 44 kHz, it is not necessary to normalize them to a specific frequency band. because there is Normalization can be normalized to 16 kHz or 11.5 kHz, but higher sampling frequencies contain audio signal components corresponding to high frequencies, which increases the amount of signals to be processed, which slows down the audio fingerprint data generation speed. because it can be

Thereafter, a fingerprint may be generated based on the histogram corresponding to the normalized sound source.

According to the present disclosure as compared with the conventional method, during normalization, normalization is performed r times larger than that of the conventional method. 4 is an example that is normalized to 32 kHz, which is twice as large as the conventional 16 kHz. Thereafter, the wave is downsampled to 1/r to increase the speed of the sound source by r times. Then, if the histogram is reduced to 1/r, it is possible to reduce the fingerprint data amount to 1/r while preserving the existing fingerprint extraction frequency range.

5 is a flowchart illustrating a method of generating a fingerprint according to an embodiment of the present disclosure.

Referring to FIG. 5 , first, a query sound source is received (S510). The query sound source can be any sound source to be recognized. Such a query sound source may be received from various external devices. Step S510 may be performed by the content receiving unit 100 . The content receiving unit 110 may be an input device capable of receiving a sound source from an external device. For example, it may be implemented as a communication port of various standards, a wireless communication device, and the like.

Thereafter, the query sound source is normalized to a preset sampling rate (S520). Here, the sampling rate is larger than the sampling rate applied in normalization for fingerprint extraction in the prior art. For example, the sampling rate may be 32 kHz.

Thereafter, the speed of the normalized query sound source is increased by r times (S530). As a method of increasing the speed, a downsampling algorithm with respect to the time axis in the time-frequency histogram of the normalized query sound source may be applied. And even if you increase the speed, the frequency range of the histogram is maintained. For example, if the sampling rate of the normalization is 32 kHz, the frequency range of the corresponding histogram is 16 kHz, which is half the value, and the frequency range is maintained at 16 kHz even if the speed is increased.

Thereafter, the time-frequency histogram for the query sound source of the increased speed is reduced to 1/r times (S540). In the above example, if the frequency range of the histogram is 16 kHz, the frequency range is reduced to 8 kHz in this step. In such a reduced histogram, the feature extraction range for generating a fingerprint is not reduced compared to the original sound source, but the speed is increased, so the data amount is reduced to 1/r.

Thereafter, an audio fingerprint of the query sound source is generated based on the reduced histogram (S550). Steps S520 to S550 may be performed by the audio fingerprint generator 200 .

Then, information on the query sound source is obtained by comparing the generated audio fingerprint of the query sound source with the audio fingerprints of the pre-stored reference sound sources (S560). Here, the audio fingerprints of the reference sound sources may be stored in the reference database 300 . Also, the audio fingerprints of the reference sound sources may be generated in the same manner as the fingerprint generation method for the query sound source. That is, it may be generated through the processes of normalization, speed increase, and histogram reduction. Therefore, since the audio fingerprints of the reference sound sources also have the same data amount reduced to 1/r, as a result, the operation amount of fingerprint matching is reduced to 1/r ² .

The information obtained in step S560 may be output from the result output unit 500 . Information output from the result output unit 500 may be output in various forms according to the purpose. For example, if the purpose of the Copyright Association is to calculate the copyright fee, it will be output in the form of information suitable for that purpose, and if the general user wants to know the title of the sound source, information about the song title, singer name, etc. will be output, or, If the reproduction of the original sound source corresponding to the query sound source is performed, the corresponding sound source may be output. The result output unit 500 may be configured as an output device such as a display or a speaker, or may be implemented as a communication device for transmitting information to another external output device.

According to the above-described embodiments, a sound source recognition speed based on fingerprint matching may be significantly improved compared to the related art.

As used in this document, a term such as "~ unit" is a term to refer to a component that performs at least one function or operation, and such a component may be implemented as hardware or software, or may be implemented as a combination of hardware and software. have. In addition, "~ parts" may be integrated into at least one module or chip and implemented by at least one processor, except when each needs to be implemented as individual specific hardware.

The various embodiments described above may be implemented by software, hardware, or a combination thereof. According to the hardware implementation, the embodiments described in the present disclosure are ASICs (Application Specific Integrated Circuits), DSPs (digital signal processors), DSPDs (digital signal processing devices), PLDs (programmable logic devices), FPGAs (field programmable gate arrays) ), processors, controllers, micro-controllers, microprocessors, and other electrical units for performing other functions may be implemented using at least one.

Various embodiments of the present disclosure may be implemented as software including instructions that may be stored in a machine-readable storage medium (eg, a computer). A machine is a device capable of calling a stored command from a storage medium and operating according to the called command.

When such an instruction is executed by a processor, the processor may perform a function corresponding to the instruction by using other components directly or under the control of the processor. Instructions may include code generated or executed by a compiler or interpreter.

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not include a signal and is tangible, and does not distinguish that data is semi-permanently or temporarily stored in the storage medium.

According to an embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. Computer program products are distributed in the form of device-readable storage media (eg compact disc read only memory (CD-ROM)) or online through application stores (eg Play Store??, App Store??). can be In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server.

In the above, preferred embodiments of the present disclosure have been illustrated and described, but the present disclosure is not limited to the specific embodiments described above, and it is common in the technical field to which the disclosure pertains without departing from the gist of the disclosure as claimed in the claims. Various modifications may be made by those having the knowledge of

100: content receiving unit
200: audio fingerprint generation unit
300: reference database
400: audio fingerprint comparison unit
500: result output unit

Claims (6)

In a sound source recognition method by an audio fingerprint-based sound source recognition system,
receiving a query sound source;
normalizing the query sound source to a sampling rate set to r times a specific sampling rate;
increasing the speed of the normalized query sound source by r times;
reducing the time-frequency histogram for the increased speed query sound source by 1/r times;
generating an audio fingerprint of the query sound source based on the reduced histogram; and
Comparing the generated audio fingerprint of the query sound source with the audio fingerprints of pre-stored reference sound sources to obtain information on the query sound source;
The increasing step is
Downsampling with respect to the time axis of the time-frequency histogram of the query sound source,
The frequency range of the time-frequency histogram of the normalized query sound source is half of the sampling rate set in the normalizing step,
The increasing step is
Method of increasing the speed while maintaining the frequency range of the time-frequency histogram of the normalized query sound source.
delete delete The method of claim 1,
The sampling rate set in the normalizing step is 32 kHz, the frequency range of the time-frequency histogram of the normalized query sound source is 16 kHz, and the frequency range of the histogram reduced in the reducing step is 8 kHz.
The method of claim 1,
The audio fingerprints of the pre-stored reference sound sources are generated in a manner corresponding to the generation of the audio fingerprints of the query sound source.
A non-transitory computer-readable medium containing computer-executable instructions, comprising:
The computer-executable instructions, when executed by a processor, cause the processor to perform the steps of the method as recited in claim 1 , 4 or 5 .

Images