KR101302567B1 - System of recognizing contents and method of extracting binary fingerprint using weighted voting - Google Patents

Abstract

A content recognition system using binary feature points through weighted voting is disclosed. The content recognition system normalizes a database and input content storing binary feature points for content-specific content retrieval, calculates feature values by applying different calculation methods to the normalized data, and calculates each of the calculated feature values. Extracting binary feature points consisting of a series of binary values compared to threshold values, reflecting a predetermined weight on each of the binary values constituting the extracted binary feature points, and extracting a final binary feature point from the weighted binary feature points. And a content recognizing unit for searching the database for metadata of the content having the same binary feature point as the final binary feature point.

Description

System of recognizing contents and method of extracting binary fingerprint using weighted voting}

The present invention relates to a content recognition technology, and more particularly, to a technology for recognizing content using a binary feature point.

Systems that recognize content (video or audio) are well known. This content recognition system extracts feature points, commonly referred to as fingerprints or hashes, from the input content and retrieves it from a previously configured database to recognize the input content. In this case, the feature points used may be mainly divided into real-type feature points or binary feature points. In particular, binary feature points are frequently used in content recognition systems. This is because binary feature points can reduce the capacity and search time taken up by the database.

As shown in FIG. 1, binary feature points are mainly generated by methods of normalization S10, calculation S20, and threshold application S30. The content recognition system performs a normalization process on the input content and converts the data into data having a certain format regardless of some distortion. The content recognition system extracts a feature that is robust to distortion through various mathematical calculations from the transformed data through normalization process, and extracts binary feature points by comparing this feature value with a threshold value.

In this way, the feature points extracted from the contents are used for searching. When a user wants to know information about an anonymous content, the anonymous content is entered into a query of a content recognition system, and the content recognition system extracts a feature point for the input content and stores it in a preconfigured database. Compared with the feature points, a content having the same feature point is searched for, and the retrieved content information is provided to the user.

On the other hand, in relation to the binary feature point extraction method of FIG. 1, the non-patent document [1] describes the normalization of audio according to a predetermined format, and the frequency of energy according to the frequency band after Fast Fourier Transformation (FFT). It is known to calculate the temporal difference and determine the feature point as 1 or 0 according to whether the calculated value is negative or positive. In the non-patent document [2], a method of normalizing a video, calculating a difference in brightness, and determining a feature point as 0 or 1 according to whether the value is positive or negative is known. However, in the case of FIG. 1, the feature point depends entirely on the result of one calculation. The feature point must be robust to various distortions. If the value obtained in the calculation process is weak to a specific distortion, the characteristic of the feature point itself becomes worse.

[1] J. Haitsma and T. Kalker, “A highly robust audio fingerprinting system,” Proc. Int. Conf. Music Information Retrieval, 2002. [2] J. Oostveen, T. Kalker, and J. Haitsma, "Feature extraction and a database strategy for video fingerprinting," Proc. Int. Conf. on Visual Information and Information Systems, pp. 117-128, 2002.

An object of the present invention is to provide a technical solution that can improve the content recognition performance.

According to an aspect of the present invention, a content recognition system using binary feature points through weighted voting normalizes a database and input content storing binary feature points for content search by content, and applies the normalized data to the normalized data. Apply different calculation schemes to calculate feature values, compare the calculated feature values with respective threshold values, extract binary feature points consisting of a series of binary values, and binary values constituting the extracted binary feature points. And a content recognizing unit for reflecting a predetermined weight to each, extracting a final binary feature point from the weighted binary feature points, and then searching the database for metadata of content having the same binary feature point as the final binary feature point.

Here, the weights are values set for respective positions of binary values, and the sum is 1. The content recognizing unit extracts the final binary feature point by determining a final binary bit value according to the total sum of binary values at the same positions of the binary feature points.

On the other hand, the binary feature point extraction method using weighted voting according to an aspect of the present invention for achieving the above technical problem is calculated by applying different calculation methods to the normalized input content, the calculated feature Extracting binary feature points consisting of a series of binary values by comparing the values with respective threshold values, reflecting a predetermined weight on each of the binary values constituting the extracted binary feature points, and applying the weighted binary feature point. Extracting the final binary feature points from them.

The content recognition system according to the present invention extracts binary feature points by using various calculation methods, creates a new final binary feature point through a weighted voting process, and searches a database based on this, thereby improving system performance. That is, the robustness of the feature point can be increased, and as a result, the performance of the system can be improved.

1 is a diagram illustrating a conventional binary feature point extraction process.
2 is a block diagram of a content recognition system using binary feature points through weighted voting according to an embodiment of the present invention.
3 is a diagram illustrating a binary feature point extraction process through weighted voting according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The foregoing and further aspects of the present invention will become more apparent from the following detailed description of preferred embodiments with reference to the accompanying drawings. Hereinafter, the present invention will be described in detail to enable those skilled in the art to easily understand and reproduce the present invention.

2 is a block diagram of a content recognition system using binary feature points through weighted voting according to an embodiment of the present invention, and FIG. 3 illustrates a process of extracting binary feature points through weighted voting according to an embodiment of the present invention.

As shown, the content recognition system includes a database 100 and a content recognition unit 210. The database 100 stores metadata for each content and binary feature points for content search. Here, metadata means specific information of content. The binary feature points for each content stored in the database 100 are final binary feature points extracted as illustrated in FIG. 3. The content recognizing unit 210 is a component that recognizes the input content by searching the database 100 using binary feature points of the input content. The content recognizing unit 210 may be a software module implemented in the digital signal processor 200 as shown.

As illustrated in FIG. 3, the content recognizing unit 210 extracts the final binary feature point through the normalization (S200), the calculation (S300), the threshold application (S400), and the weighted voting process (S500). . First, the content recognizing unit 210 performs a normalization process on the content input to the system (query) and converts the data into a data having a certain format regardless of some distortion (S200). This normalization process is the same as before. When the normalization process is completed, the content recognizing unit 210 calculates different calculation methods (calculation 1, calculation 2, calculation 3, ......, calculation N) on data having a predetermined format obtained through the normalization process. The feature values (feature 1, feature 2, feature 3, ..., feature N) of the shape which is robust to distortion are extracted by applying (S300). Here, the calculation methods include a method of calculating the frequency-time difference of energy according to the frequency band after FFT, a method of calculating the center value of each frequency band and the difference value of each band of the center values, the x-order center of gravity of each frequency band, or Well-known calculation methods such as the method of calculating the band-specific difference of the center of gravity, the spectral flatness of each frequency band, and the mel-frequency cepstral coefficients (MFCC) may be used.

When the calculation process is completed, the content recognition unit 210 compares the feature values extracted through the calculation process with the respective threshold values (threshold application 1, threshold application 2, threshold application 3, ..., threshold application). N) Binary feature points (binary feature point 1, binary feature point 2, binary feature point 3, ......, Binary feature point N) is extracted (S400). When the threshold application process is completed, the content recognizing unit 210 performs a weighted voting process for determining the final binary feature points from the binary feature points extracted through the threshold application process (S500). The weighted voting method of N binary feature points is multiplied by a binary feature point of 0 and 1 whose total sum is 1, and then assigns bit 1 to the final binary feature point if the total sum is greater than 0.5, otherwise 0 is assigned. To assign. That is, assuming that the k th bit of the final binary feature point is F _k , F _k is represented by Equation 1 below.

은 n번째 가중치,

은 특징점 n의 k번째 비트라고 할 수 있다. 이때,

이다. 가중 투표를 통해 최종 이진 특징점이 추출되면, 콘텐츠 인식부(210)는 그 결정된 최종 이진 특징점을 가지고 데이터베이스(100)에서 해당 콘텐츠를 검색하며, 검색된 콘텐츠의 메타데이터를 결과물로 출력한다.here

Is the nth weight,

Is the k-th bit of the feature point n. At this time,

to be. When the final binary feature point is extracted through weighted voting, the content recognizing unit 210 retrieves the corresponding content from the database 100 with the determined final binary feature point, and outputs the metadata of the retrieved content as a result.

의 쌍으로 정의한다. 그리고 i<=I_1까지는 동일한 콘텐츠에서 나온 이진 특징점으로 동일한 비트 값이 나와야 하고, 그 이외의 값은 서로 다른 비트 값을 가져야 한다.On the other hand, the weight used in the weighted voting is determined through the training (S100). Training is the process of finding the best performing weights based on the training data, which is the process performed by the system. The training process can be carried out in the following way. Once training requires training data. Training data exists in pairs. Since the binary feature points themselves are used in the form of judging whether the content is the same or different, training data also exists in pairs, and has information about whether two binary feature points are extracted from the same content or from different content. For convenience of explanation, the training data set is

It is defined as a pair of. And until i <= I_1, the same bit value should come out as a binary feature point from the same content, and other values should have different bit values.

This will be explained further. Normal training data consists of actual data + its correct answer. For example, if you create training data for face recognition algorithms for entertainers, (Jang Dong Gun face 1.jpg, “Jang Dong Gun”), (Jang Dong Gun face 2.jpg, “Jang Dong Gun”), ... (Jang Dong Gun face 100) .jpg, “Jang Dong Gun”), (Han Ye Seul Face 1.jpg, “Han Ye Seul”), ... (Han Ye Seul Face 100.jpg, “Han Ye Seul”), (Han Hyo Joo Face 1.jpg, “Han Hyo Joo”), ... (Han Hyo Joo Face 100.jpg, “Hyo Hyo Joo”) It will be like this. That is, it is configured as (data, correct answer value of data). However, the training data required during the training process is whether the two answers are the same or different. This structure is the method used in some papers. Reference paper information is as follows.

[Reference 1] Dalwon Jang, Chang D. Yoo, and Ton Kalker, “Distance metric learning for content identification,” IEEE Trans. Information Forensics and Security, vol. 5, no. 4, Dec. 2010.

[Reference 2] Dalwon Jang, Chang D. Yoo, Sunil Lee, Sungwoong Kim, and Ton Kalker, "Pairwise Boosted Audio Fingerprint," IEEE Trans. Information Forensics and Security, vol. 4, no. 4, pp. 995-1004, Dec. 2009.

Therefore, there must be one pair for the correct answer that the two are equal, and one pair for the correct answer that the two is different. Audio will be described as an example as follows. For a song A, noise is added or distortion after recording, delay, compression, etc. after speaker output is expressed as A ', A' ', A' '', or the like. And other songs B, C, and D. In this case, ((A, A '), "same song"), ((A, A' '), "same song"), ((A, A' ''), "same song"), (( A, B), “another song”), ((A, C), “another song”), ((A, D), “another song”).

라 할 수 있다. 여기서 설명의 편의를 위해서 i<=I_1까지는 “같은 노래”라고 되어 있는 쌍이다. 따라서 M개의 쌍이 “같은 노래”라고 되어 있고, 각 노래당 P개의 비트가 나온다면 I_1=M*N*P가 된다. 여기서 P는 이진 특징점을 어떻게 만드느냐에 따라 달라진다. 예를 들어, 0.1초에 8비트씩 나오게 하거나 32비트씩 나오게 하느냐에 따라 P 값이 달라진다. 이 같이 I_1개의 서로 같은 노래에서 나온 비트를 만들고, 반대로 서로 다른 노래에서 나온 비트들도 I_2개를 만들고, I_1+I_2=I라고 정의한다. 이렇게 하면 훈련 데이터 전체를

for i=1,2,..., I로 둘 수 있다. 그리고 훈련은 수학식 2를 통해서 이루어진다.With this training data, we can extract N kinds of binary feature points. And each bit of them

. For convenience of explanation, i <= I_1 is a pair of “same song”. Thus, if M pairs are called "same song" and there are P bits per song, then I_1 = M * N * P. Where P depends on how the binary feature points are made. For example, the value of P depends on whether 8-bits or 32-bits are output in 0.1 second. Likewise, I_1 beats are made from the same song and vice versa. I_2 beats are also made from different songs, and I_1 + I_2 = I is defined. This will allow the entire training data

for i = 1,2, ..., I And training is done through Equation 2.

subject to

only,

값(n=1,...,N)을 찾아야 한다.

값을 찾는 것은 semi-definite program이나 gradient descent 방법을 통해서 구할 수 있다.
That is, when i is smaller than I_1, the two bit values must be greater than 0.5 or less than 0.5 because the same bit values must be the same. So we have maximized the two products, and if it is greater than I_1 we should keep this value negative. Therefore, Equation 2 is created, and Equation 2 is optimized

Find the value (n = 1, ..., N).

Finding the value can be obtained through a semi-definite program or a gradient descent method.

So far I looked at the center of the preferred embodiment for the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: database 200: digital signal processor
210: content recognition unit

Claims (6)

A database storing binary feature points for content search by content; And
Normalize the input content, apply different calculation schemes to the normalized data, calculate feature values, compare the calculated feature values with respective threshold values, extract binary feature points consisting of a series of binary values, and And extracting a predetermined binary feature point from each of the binary values constituting the extracted binary feature points, extracting a final binary feature point from the weighted binary feature points, and extracting metadata of content having the same binary feature point as the final binary feature point. A content recognizing unit for searching in the database;
Content recognition system using a binary feature point through weighted voting characterized in that it comprises a. The method of claim 1,
The weight is a content recognition system using a binary feature point through weighted voting, characterized in that the sum total is 1. The method of claim 2,
And the content recognizing unit determines a final binary bit value of the final binary feature point according to Equation 1 below.
[Equation 1]

(here,

Is the nth weight,

Is the kth bit of feature point n) Database; And
A binary feature extraction method performed by a content recognition system including a content recognition unit which is a software module implemented in a digital signal processor,
Calculating, by the content recognizing unit, feature values by applying different calculation schemes to normalized input content;
Extracting, by the content recognizing unit, binary feature points consisting of a series of binary values by comparing the calculated feature values with respective threshold values;
Reflecting, by the content recognizing unit, a predetermined weight on each of the binary values constituting the extracted binary feature points; And
Extracting, by the content recognizing unit, a final binary feature point from the binary feature points in which the weight is reflected;
Binary feature point extraction method through weighted voting comprising a. 5. The method of claim 4,
The weighting method is a binary feature point extraction method using weighted voting, characterized in that the sum total. The method of claim 5,
And extracting the final binary feature point to determine a final binary bit value of the final binary feature point according to Equation 1 below.
[Equation 1]

(here,

Is the nth weight,

Is the kth bit of feature point n)

Images