CN102214218B - System and method for retrieving contents of audio/video - Google Patents

Abstract

The invention discloses a system and a method for retrieving contents of an audio/video. The method at least comprises the following steps of: receiving an audio/video segment; extracting the fingerprint characteristic of each frame of the audio/video segment; computing the anti-interference degree of a fingerprint of each frame by using an anti-interference degree computation model; sorting the frames according to the anti-interference degree of the fingerprint of each frame; and retrieving the fingerprints in a fingerprint database according to a frame sorting result. The fingerprint retrieval speed of the audio/video is greatly improved by computing the anti-interference degree of the fingerprint characteristic of each frame, sorting according to the anti-interference degree of the fingerprint characteristic of each frame and retrieving.

Description

Audio-video frequency content searching system and method thereof

Technical field

The present invention is about a kind of audio-video frequency content searching system and method thereof, particularly about a kind of audio-video frequency content searching system and method thereof based on audio/video fingerprint.

Background technology

Along with the fast development of network and multimedia technology, the quantity of audio-video frequency media is explosive increase, and it is very difficult that people manage digital audio/video frequency content accurately and effectively and visit becomes.In recent years, many new research and development directions have appearred in content-based audio frequency and video retrieval, and the audio/video fingerprint technology is arisen at the historic moment.

Audio-frequency fingerprint (audio fingerprinting) technology just is being suggested a long time ago; For example; Jaap Haitsma and Ton Kalke have delivered " a kind of audio fingerprint system of high reliability " (AHighly Robust Audio Fingerprinting System) in music searching progress in 2002 international conference; This system passes through method for processing signals; With the sound signal of (for example 11.6ms) at set intervals in the audio file, be converted into the fingerprint (fingerprint) of one 32 bit (bit) size, an audio file just can be converted into a file fingerprint by this method; System is after indexing to all audio-frequency fingerprint files, and just audio-frequency fingerprint has been retrieved fast.

Similar with the audio fingerprint techniques principle, video finger print (video fingerprinting) system is converted into very little fingerprint (for example 32 bits size) with each frame or several frames, retrieves then.For example international patent is that " the Method and system for fingerprinting digital video object based on multiresolution; multirat and temporal signatures " of WO2007/127590A2 discloses the video finger print disposal route that a kind of every frame with vision signal is converted into 84 bits or 132 bits, and a video file just can be converted into a very little file fingerprint by this method.

It is thus clear that existing audio/video fingerprint technology all is at first the audio-video frequency content of importing to be carried out fingerprint characteristic to extract usually, according to the frame preface of fingerprint characteristic, in fingerprint database, carries out the retrieval of inverted index in order then.Yet but there is following problem in this way: because code check, form and the noise of audio frequency and video can cause the fingerprint characteristic distortion, influence retrieval effectiveness.

In sum, can know that the audio frequency and video retrieval technique of prior art exists because audio frequency and video code check, form and noise cause fingerprint characteristic to be out of shape and then to influence the problem of retrieval effectiveness, therefore be necessary to propose improved technological means in fact, solve this problem.

Summary of the invention

For the audio frequency and video retrieval technique that overcomes above-mentioned prior art exists because audio frequency and video code check, form and noise cause fingerprint characteristic to be out of shape and then to influence the problem of retrieval effectiveness; Fundamental purpose of the present invention is to provide a kind of audio-video frequency content searching system and method thereof; It is through calculate the anti-interference degree of each frame fingerprint characteristic in advance; And, can improve retrieval rate greatly by the anti-interference degree of the fingerprint characteristic laggard line retrieval that sorts.

For reaching above-mentioned and other purpose, a kind of audio-video frequency content searching system of the present invention comprises at least:

Receive module, be used to receive an audio frequency and video segment;

Fingerprint characteristic extracts module, is used to extract the fingerprint characteristic of this each frame of audio frequency and video segment;

Anti-interference degree calculation module makes up an anti-interference degree computation model, calculates the anti-interference degree of every frame fingerprint according to this anti-interference degree computation model;

The ordering module carries out the frame ordering according to the anti-interference degree of every frame fingerprint; And

The retrieval module carries out the fingerprint retrieval according to the frame ranking results in fingerprint database.

Further, this fingerprint characteristic extracts module and assigns to obtain the fingerprint characteristic of each frame through calculating the energy difference of adjacent two sub belt energy difference and the same subband of adjacent two frames in the same frame.

Further, this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1)));$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, E (n, m) energy of m subband of expression n frame.

Further, the anti-interference degree of every frame fingerprint can the highest some dimensions be calculated acquisition through the absolute value of primary energy difference numerical.

Further, this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, and E_frame_sort (n, k)=sort _M=1,32(ABS (E (n, m)-E (n, m+1)-(E (n-1, m)-E (n-1, m+1))), (sort () representes ordering to E for n, the m) energy of m subband of expression n frame.

Further, this anti-interference degree calculation module also can be calculated the anti-interference degree of each frame fingerprint through spectrum value or color-values.

For reaching above-mentioned and other purposes, the present invention also provides a kind of audio-video frequency content search method, and this method comprises the steps: at least

Receive an audio frequency and video segment;

Extract the fingerprint characteristic of this each frame of audio frequency and video segment;

Utilize an anti-interference degree computation model to calculate the anti-interference degree of each frame fingerprint;

Carry out the frame ordering according to the anti-interference degree of every frame fingerprint; And

Result according to the frame ordering carries out the fingerprint retrieval in fingerprint database.

Further, each frame fingerprint characteristic is to assign to obtain through calculating in the same frame energy difference of adjacent two sub belt energy difference and the same subband of adjacent two frames.

Further, this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1)));$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, E (n, m) energy of m subband of expression n frame.

Further, this anti-interference degree computation model can also for:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, and E_frame_sort (n, k)=sort _M=1,32(ABS (E (n, m)-E (n, m+1)-(E (n-1, m)-E (n-1, m+1))), (sort () representes ordering to E for n, the m) energy of m subband of expression n frame.

Further, the anti-interference degree of each frame fingerprint also can be calculated acquisition through spectrum value or color-values.

Compared with prior art; A kind of audio-video frequency content searching system of the present invention and method are through the anti-interference degree of each frame fingerprint of the audio frequency and video of the needs of calculating in advance retrieval; And according to retrieval after the anti-interference degree ordering of fingerprint; The preferential retrieval that anti-interference degree is high, rather than press the retrieval of frame preface, can improve retrieval rate greatly.

Description of drawings

Fig. 1 is the system architecture diagram of a kind of audio-video frequency content searching system of the present invention;

Fig. 2 is the flow chart of steps of a kind of audio-video frequency content search method of the present invention.

Embodiment

Below through specific instantiation and accompanying drawings embodiment of the present invention, those skilled in the art can understand other advantage of the present invention and effect easily by the content that this instructions disclosed.The present invention also can implement or use through other different instantiation, and each item details in this instructions also can be based on different viewpoints and application, carries out various modifications and change under the spirit of the present invention not deviating from.

Fig. 1 is the Organization Chart of a kind of audio-video frequency content searching system of the present invention, below will with Fig. 1 System Operation of the present invention be described earlier.As shown in Figure 1, a kind of audio-video frequency content searching system of the present invention comprises reception module 101 at least, fingerprint characteristic extracts module 102, anti-interference degree calculation module 103, ordering module 104 and retrieval module 105.

Receive module 101 and be used to receive a files in stream media, comprise an audio frequency and video segment in this files in stream media at least, this audio frequency and video segment can be audio file, also can be video file.

Fingerprint characteristic extracts module 102 and is connected with reception module 101; Receiving the audio frequency and video sheet at reception module 101 has no progeny; Fingerprint characteristic extracts the fingerprint characteristic that 102 of modules are used for extracting each frame audio frequency of this audio frequency and video segment or each frame video, below specifies fingerprint characteristic and extracts the take the fingerprint course of work of characteristic of module 102.

Be characterized as example at this to extract audio-frequency fingerprint, at first, fingerprint characteristic extracts the monophonic audio of module 102 for fixed sample rate, moves by anchor-frame audio frequency to be divided into some milliseconds audio frame, and adds Hamming window (Hanning Window); Secondly, each frame audio frame is done Fourier transform, extract power spectrum, adopt logarithmic scale evenly to be divided into mutually disjoint 33 subbands on certain section frequency band (for example 300Hz-4000Hz), and calculate the sub belt energy of each frame; At last, calculate the energy difference of adjacent two sub belt energy difference and the same subband of adjacent two frames in the same frame, obtain fingerprint characteristic value, can obtain through following computing formula:

$F(n,m)=\left\{\begin{array}{c}1,E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1))>0\\ 0,E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1))\≤0\end{array}\right.$

Wherein use E (n, the m) energy of m subband of expression n frame, F (n, the m dimensional feature in the fingerprint characteristic that m) expression n frame is corresponding.

The characteristic of utilizing above-mentioned formulas Extraction is totally 32 bit (bit), the just length of what a long type data.Can see; Each dimension of each frame has been quantized into 0 or 1 by original energy difference fractional value; If the absolute value of energy difference fractional value is big more; This dimension of this frame is with regard to feeling bad noise more, if the absolute value of primary energy difference numerical is more little, this dimension of this frame just receives noise more easily.Though preferred embodiment of the present invention only obtains the fingerprint characteristic of every frame with the method for primary energy difference numerical, the present invention can certainly obtain the fingerprint characteristic of every frame through additive methods such as spectrum value, color-values not as limit, does not repeat them here.

Similar with the audio-frequency fingerprint feature extraction; For the video finger print Feature Extraction, both can take original energy difference fractional value method to obtain, also can obtain through additive methods such as spectrum value, color-values; The video finger print Feature Extraction is the prior art of convention, does not repeat them here.

After fingerprint characteristic extraction module 102 extracted the fingerprint characteristic of each frame audio frequency and each frame video, 103 of anti-interference degree calculation module made up the anti-interference degree that an anti-interference degree computation model calculates each frame fingerprint.Anti-interference degree for every frame fingerprint; Can calculate through the numerical value that adopts the primary energy difference; Also can adopt spectrum value or color-values to calculate, the present invention is not as limit, in preferred embodiment of the present invention; Then still be calculated as example with the numerical value that adopts the primary energy difference, anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1)));$

Here the anti-interference degree of representing n frame fingerprint, E (n, m) energy of m subband of expression n frame.

Preferable, the anti-interference degree of every frame fingerprint can also the highest some dimensions be calculated through the absolute value of primary energy difference numerical:

E_frame_sort (n, k)=sort _M=1,32(ABS (E (n, m)-E (n, m+1)-(E (n-1, m-E (n-1, m+1))), wherein, sort () representes ordering; Then anti-interference degree computation model can for:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Expression calculate every frame absolute value the highest the 1st tie up the 32nd dimension.

Ordering module 104 is used for carrying out the frame ordering according to the anti-interference degree Robust (n) of every frame fingerprint; Retrieval module 105 carries out the fingerprint retrieval according to the frame ranking results of anti-interference degree in fingerprint database.Specifically; During 105 retrievals of retrieval module is not to retrieve according to frame preface commonly used; But retrieve the preferential retrieval that Robust (n) is high more, the hysteresis retrieval that Robust (n) is low more according to the ranking results of the Robust (n) that representes the anti-interference degree of this frame fingerprint.

Fig. 2 is the flow chart of steps of a kind of audio-video frequency content search method of the present invention.As shown in Figure 2, a kind of audio-video frequency content search method of the present invention comprises the following steps: to receive an audio frequency and video segment (step 201); Extract each frame fingerprint characteristic (step 202) of this audio frequency and video segment; Utilize anti-interference degree computation model to calculate the anti-interference degree (step 203) of each frame fingerprint; Carry out frame ordering (step 204) according to the anti-interference degree of every frame fingerprint; And carry out fingerprint according to the result that frame sorts to fingerprint database and retrieve (step 205).

For step 203; The anti-interference degree of every frame fingerprint can be calculated through the numerical value that adopts the primary energy difference, also can adopt spectrum value or color-values to calculate; In preferred embodiment of the present invention; Then be calculated as example with the numerical value that adopts the primary energy difference, therefore, anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1)));$

Here the anti-interference degree of representing n frame fingerprint, E (n, m) energy of m subband of expression n frame.

Preferable, the anti-interference degree of every frame fingerprint, can also the highest some dimensions calculate through the absolute value of primary energy difference numerical:

E_frame_sort (n, k)=sort _M=1,32(AES (E (n, m)-E (n, m+1)-(E (n-1, m)-E (n-1, m+1))), wherein, sort () representes ordering; Then anti-interference degree computation model can for:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Expression calculate every frame absolute value the highest the 1st tie up the 32nd dimension.

In sum; A kind of audio-video frequency content searching system of the present invention and method are through the anti-interference degree of each frame fingerprint of the audio frequency and video of the needs of calculating in advance retrieval; And according to retrieval after the anti-interference degree ordering of fingerprint; The preferential retrieval that anti-interference degree is high, rather than press the retrieval of frame preface, improved the speed of audio/video fingerprint retrieval greatly.

The foregoing description is illustrative principle of the present invention and effect thereof only, but not is used to limit the present invention.Any those skilled in the art all can be under spirit of the present invention and category, and the foregoing description is modified and changed.Therefore, rights protection scope of the present invention should be listed like claims.

Claims (9)

1. audio-video frequency content searching system comprises at least:

Receive module, be used to receive an audio frequency and video segment;

Fingerprint characteristic extracts module, is used to extract the fingerprint characteristic of this each frame of audio frequency and video segment;

Anti-interference degree calculation module makes up an anti-interference degree computation model, and this anti-interference degree computation model of foundation calculates the anti-interference degree of every frame fingerprint, and this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1))),$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, E (n, m) energy of m subband of expression n frame;

The ordering module carries out the frame ordering according to the anti-interference degree of every frame fingerprint; And

The retrieval module carries out the fingerprint retrieval according to the frame ranking results in fingerprint database.

2. audio-video frequency content searching system as claimed in claim 1 is characterized in that: this fingerprint characteristic extracts module and assigns to obtain the fingerprint characteristic of each frame through calculating the energy difference of adjacent two sub belt energy difference and the same subband of adjacent two frames in the same frame.

3. audio-video frequency content searching system as claimed in claim 2 is characterized in that: the anti-interference degree of every frame fingerprint can the highest some dimensions be calculated acquisition through the absolute value of primary energy difference numerical.

4. audio-video frequency content searching system as claimed in claim 3 is characterized in that: this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, and E_frame_sort (n, k)=sort _M=1,32(ABS (E (n, m)-E (n, m+1)-(E (n-1, m)-E (n-1, m+1))), (sort () represent ordering to E for n, the m) energy of m subband of expression n frame, representes the numerical value label of participating in sorting with m before the ordering, representes to arrange the numerical value label of preface with k after sorting.

5. audio-video frequency content searching system as claimed in claim 1 is characterized in that: this anti-interference degree calculation module is calculated the anti-interference degree of each frame fingerprint through spectrum value or color-values.

6. an audio-video frequency content search method comprises the steps: at least

Receive an audio frequency and video segment;

Extract the fingerprint characteristic of this each frame of audio frequency and video segment;

Utilize an anti-interference degree computation model to calculate the anti-interference degree of each frame fingerprint, this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{m=1}^{m=32}\mathrm{ABS}(E(n,m)-E(n,m+1)-(E(n-1,m)-E(n-1,m+1)));$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, E (n, m) energy of m subband of expression n frame;

Carry out the frame ordering according to the anti-interference degree of every frame fingerprint; And

Result according to the frame ordering carries out the fingerprint retrieval in fingerprint database.

7. audio-video frequency content search method as claimed in claim 6 is characterized in that: each frame fingerprint characteristic is to assign to obtain through calculating in the same frame energy difference of adjacent two sub belt energy difference and the same subband of adjacent two frames.

8. audio-video frequency content search method as claimed in claim 7 is characterized in that, this anti-interference degree computation model is:

$\mathrm{Robust}\left(n\right)={\mathrm{\Σ}}_{k=1}^{k=32}E\_\mathrm{frame}\_\mathrm{sort}(n,k);$

Wherein Robust (n) representes the anti-interference degree of n frame fingerprint, and E_frame_sort (n, k)=sort _M=1,32(ABS (E (n, m)-E (n, m+1)-(E (n-1, m)-E (n-1, m+1))), (sort () represent ordering to E for n, the m) energy of m subband of expression n frame, representes the numerical value label of participating in sorting with m before the ordering, representes to arrange the numerical value label of preface with k after sorting.

9. audio-video frequency content search method as claimed in claim 6 is characterized in that: the anti-interference degree of each frame fingerprint can be calculated acquisition through spectrum value or color-values.

Images