CN102314681A - Adaptive KF (keyframe) extraction method based on sub-lens segmentation - Google Patents

Abstract

The invention discloses an adaptive KF (keyframe) extraction method based on sub-lens segmentation in the technical field of image processing. The method provided by the invention comprises the following steps: reading all frames of an image in a lens, and extracting the color feature vectors of each frame; combining a sliding window with a distance separability rule to segment the lens into sub-lenses; determining the number of KFs of each sub-lens according to the rate of change of interframe distances; and finally, selecting the set number of KFs according to the order from big to small of the interframe distances. The method provided by the invention has the advantages that a threshold is not required to be set, the number of the KFs can be confirmed adaptively, the segmentation precision of each sub-lens is improved, and the robustness is good.

Description

The self adaptation key frame method for distilling of cutting apart based on sub-camera lens

Technical field

The invention belongs to technical field of image processing, relate in particular to a kind of self adaptation key frame method for distilling of cutting apart based on sub-camera lens.

Background technology

Along with developing rapidly of multimedia technology and computer networking technology, the number of videos sharp increase, video frequency search system has received increasing concern.And key-frame extraction is basic as the early stage of video frequency searching, and its extraction effect will directly influence the performance of video frequency search system.Key frame of video extract to be exactly research and how to utilize minimum image to represent most effectively the main contents of video lens, and this just requires the selected key frame will be with the sequential property and the accuracy of the redundance reflecting video of minimum.

The key-frame extraction algorithm of cutting apart based on sub-camera lens as one of extraction method of key frame of classics, exactly from according to certain characteristic a camera lens being divided into the experimental process camera lens from time domain, extracts a frame as key frame then in each sub-camera lens.Document (Lei Pan; Xiaojun Wu; Xin Shu.Key Frame Extraction Based on Sub-shot Segmentation and Entropy) it is poor at first to calculate the histogram of present frame and preceding N frame; Adopt threshold method to carry out sub-camera lens to the difference result subsequently and cut apart, in each sub-camera lens, choose the maximum frame of information entropy at last as key frame.The defective of the method be to threshold value to choose dependence very big because the varying of video content, threshold method effect on sub-camera lens segmentation precision is undesirable; Document (Tianming Liu; Hong-Jiang Zhang; And Feihu Qi.A Novel Video Key-Frame-Extraction Algorithm Based on Perceived Motion Energy Model [J] .IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY; 13 (10): pp.1006-1013; 2003.) according to the motion of object in the camera lens camera lens is carried out time domain and cut apart, and choose the maximum frame of exercise intensity in the sub-camera lens after cutting apart as key frame; Document (Pascal Kelm; Sebastian Schmiedeke; And Thomas Sikora.FEATURE-BASED VIDEO KEY FRAME EXTRACTION FOR LOW QUALITY VIDEO SEQUENCES [J] .IN PROCEEDINGS OF 10TH INTERNATIONAL WORKSHOP ON IMAGE ANALYSIS FOR MULTIMEDIA INTERACTIVE SERVICES [C] .2009; Pp.25-28.) rely on the motion state of camera lens to carry out sub-camera lens and cut apart, on the basis of taking all factors into consideration factors such as object of which movement intensity, camera motion mode, choose can an attractive frame as key frame.

There is following several problem in above method:

1. the factor that causes the camera lens content change is diversified, and only rely on certain single characteristic carry out sub-camera lens cut apart obviously be have circumscribed.

2. above several method is only chosen a frame as key frame in each sub-camera lens; The multidate information of video content is beyond expression; Especially video content changes when very fast, and key frame can't the reflecting video content, and this has run counter to the principle that " rather lacks excessively " in the key-frame extraction.

Summary of the invention

Deficiencies such as segmentation precision is low to mentioning in the above-mentioned background technology in the existing sub-camera lens cutting procedure, poor robustness, key-frame extraction fixed number the present invention proposes a kind of self adaptation key frame method for distilling of cutting apart based on sub-camera lens.

Technical scheme of the present invention is based on the self adaptation key frame method for distilling that sub-camera lens is cut apart, to it is characterized in that this method may further comprise the steps:

Step 1: read all frames of image in the camera lens, extract the color feature vector of each frame;

Step 2: on the basis of step 1, with moving window with combine apart from the separability criterion, camera lens is divided into each sub-camera lens;

Step 3: the number of key frames of obtaining each sub-camera lens according to the interframe range rate;

Step 4: on the basis of step 3, from sorting from big to small, choose the key frame that specifies number according to frame pitch.

Said step 2 is specially:

Step 2.1: set up moving window, try to achieve the adjacent two types of samples in the sliding window within class scatter matrix and type between the dispersion matrix;

Step 2.2: on the basis of step 2.1, set up and pass judgment on function;

Step 2.3: the judge function curve of trying to achieve camera lens;

Step 2.4: to passing judgment on function curve filtering, and the frame that will pass judgment on each maximum point correspondence of function curve is regarded as the split position of sub-camera lens.

The computing formula of said judge function is:

$F=\frac{\det \left(S_b\right)}{\det (S_1+S_2)}$

Wherein:

F is for passing judgment on function curve;

S _bBe dispersion matrix between class;

S ₁For the first two/moving window in the within class scatter matrix of sub-camera lens frame;

S ₂Within class scatter matrix for sub-camera lens frame in 1/2nd moving windows of back;

Det (S _b) dispersion determinant of a matrix value between type of being;

Det (S ₁+ S ₂) be the determinant of within class scatter matrix.

Said S _bComputing formula be:

S _b＝(m ₁-m ₂)(m ₁-m ₂) ^T

Wherein:

m ₁For the first two/mean vector of sub-camera lens frame in the moving window;

m ₂Mean vector for the sub-camera lens frame in 1/2nd moving windows of back.

Said S ₁Computing formula be:

$S_1=\underset{H\∈L_1}{\mathrm{\Σ}}(H-m_1){(H-m_1)}^T$

Wherein:

L ₁For the first two/ set that the sample of sub-camera lens frame in the moving window is formed;

H is the proper vector of certain sample frame in the sample set.

Said S ₂Computing formula be:

${\mathrm{<figure-callout\; id="2"\; label="S"\; filenames="HSA00000534004000021.png"\; state="\{\{state\}\}">S</figure-callout>}}_2=\underset{H\&Element;L_2}{\mathrm{\&Sigma;}}(H-m_2){(H-m_2)}^T$

Wherein:

L ₂The set that the sample of the sub-camera lens frame in 1/2nd moving windows of back is formed.

The formula of said judge function curve filtering is:

$F^{\&prime;}=F-\stackrel{\&OverBar;}{F}$

Wherein:

F ' is filtered judge function curve;

is filter threshold.

The computing formula of said

is:

$\stackrel{\&OverBar;}{F}=F_{\mathrm{mean}}+F_{\mathrm{std}}$

Wherein:

F _MeanFor passing judgment on the average of function curve;

F _StdFor passing judgment on the standard deviation of function curve.

The computing formula of said number of key frames is:

$k=\frac{\mathrm{dist}(1,n)}{\frac{1}{n-1}\underset{i=1}{\overset{n-1}{\mathrm{\&Sigma;}}}\mathrm{dist}(i,i+1)}$

Wherein:

K is a number of key frames;

N is the totalframes of sub-camera lens;

Dist (1, n) be the Euclidean distance of sub-camera lens head and the tail two frames;

(i i+1) is the Euclidean distance of adjacent two frames in the moving window to dist.

The invention has the advantages that:

1. the present invention's artificial certain characteristic of extracting of having no backing is carried out sub-camera lens and is cut apart; But utilize the self-characteristic and the physical change of frame of video; A, type interior distance minimum principle maximum according between class distance is divided into the experimental process camera lens with camera lens, can greatly improve and cut apart accuracy.

2. discriminant function explicit physical meaning of the present invention, computing is simple.

3. the present invention has provided a formula that calculates the key-frame extraction number, and this formula can be confirmed the number of key frame adaptively according to each sub-camera lens content change speed, can reflect the dynamic perfromance of camera lens well.

4. the present invention need not be provided with threshold value, has overcome the low defective with poor robustness of document [1] thresholding method precision.

Description of drawings

Fig. 1 is a ROAD video F curve;

The extreme value that Fig. 2 tries to achieve for F ' curve;

The extreme value that Fig. 3 tries to achieve for the F curve.

Embodiment

Below in conjunction with accompanying drawing, preferred embodiment is elaborated.Should be emphasized that following explanation only is exemplary, rather than in order to limit scope of the present invention and application thereof.

The present invention utilize sample apart from the separability criteria construction discriminant function, the method through seeking this discriminant function maximum value is divided into several sub-camera lenses with camera lens.Subsequently, according to the content change speed of each sub-camera lens, confirm the key frame number of each sub-camera lens adaptively.

Step of the present invention is following:

Step 1: read all frames of image in the camera lens, extract the color feature vector of each frame;

Step 2: with moving window with combine apart from the separability criterion, camera lens is divided into each sub-camera lens;

Step 3: the number of key frames of obtaining each sub-camera lens according to the interframe range rate;

Step 4: from sorting from big to small, choose the key frame that specifies number according to frame pitch.

Particular content of the present invention is:

1. feature extraction

HSV (tone Hue, saturation degree Saturation, brightness Value) color space more meets human visual system than other color spaces, so this paper adopts the proper vector of hsv color histogram as frame of video.Because human eye is responsive to tone comparison saturation degree and brightness, so this paper is divided into 8 quantized levels with tone, saturation degree and brightness is divided into 2 quantized levels.Therefore, every image can be quantified as an one dimension histogram that comprises 32 handles (8 * 2 * 2), and promptly every frame can be by column vector H=[h ₁, h ₂..., h ₃₂] ^TExpression.

2. sub-camera lens is cut apart

(1) early-stage preparations

At first set up the moving window that length is the 2L frame length, and the preceding L frame in the moving window is regarded as sample set L ₁=(H _(i-L), H _(i-L+1)..., H _(i-1)), back L frame is regarded as sample set L ₂=(H _(i+1), H _(i+2)..., H _(i+L)), wherein: H _(j)(j ∈ [i-L, i+L]) represents the j frame in the moving window, can be by the column vector H=[h of one 32 dimension ₁, h ₂..., h ₃₂] ^TExpression.

The mean vector m of two types of sample sets before and after by formula calculate (1) _i, m ₁The preceding L frame of representative (is sample set L ₁) mean vector, m ₂Representing back L frame (is sample set L ₂) mean vector.

$m_i=\frac{1}{L}\underset{H\&Element;L_i}{\mathrm{\&Sigma;}}H,i=\mathrm{1,2}---\left(1\right)$

Wherein:

H is the proper vector of sample frame in the sample set;

L is the length of 1/2nd moving windows.

Calculate sample set L at last _iWithin class scatter matrix S _i, L ₁, L ₂The class between the dispersion matrix S _bDiscrete matrix S in type _iVery similar with covariance matrix in form, but covariance matrix is a kind of expectation value, and a type interior discrete matrix is represented the dispersion degree of limited sample in space distribution:

$S_i=\underset{H\&Element;L_i}{\mathrm{\&Sigma;}}(H-m_i){(H-m_i)}^T---\left(2\right)$

S _b＝(m ₁-m ₂)(m ₁-m ₂) ^T (3)

Wherein:

S _bBe dispersion matrix between class;

S _iBe within class scatter matrix;

m ₁For the first two/mean vector of sub-camera lens frame in the moving window;

m ₂Mean vector for the sub-camera lens frame in 1/2nd moving windows of back.

(2) make up the judge function

Maximum when two types of sample between class distances in the sliding window, type in apart from hour being sub-shot change part.By knowing apart from the separability criterion: between class distance is maximum, type interior distance is minimum, and promptly being equal to is det (S _b) maximum, det (S ₁+ S ₂) minimum.Therefore can pass judgment on function as follows based on constructing apart from the separability criterion:

$F=\frac{\det \left(S_b\right)}{\det (S_1+S_2)}---\left(4\right)$

Wherein:

F is for passing judgment on function curve;

S ₁Be the within class scatter matrix of preceding L frame (the first two/in the moving window sub-camera lens frame),

L ₁Set for preceding L frame sample composition;

S ₂Be the within class scatter matrix of back L frame (the sub-camera lens frames in 1/2nd moving windows of back),

L ₂Set for back L frame sample composition;

Det (S _b) dispersion determinant of a matrix value between type of being;

Det (S ₁+ S ₂) be the determinant of within class scatter matrix.

(3) the F value curve of calculating camera lens.

Move moving window backward by frame, and calculate the F value.When whole sliding window is in same camera lens, the F value is constant basically, and desirable situation F value levels off to zero; When after the F value increases to certain value gradually, reducing gradually again, explain that moving window has experienced a process of crossing over sub-camera lens:

When the F value increases gradually, explain that this moment, back L frame began to get into a back sub-camera lens;

When F obtains maximum value, explain that this moment, back L frame got into a back sub-camera lens fully, simultaneously preceding L frame is in last sub-camera lens fully;

The F value reduces gradually, and the L frame progresses into a back sub-camera lens before explaining;

F value curve tends to be steady, and explains that this moment, whole sliding window all got into next sub-camera lens.

Therefore, can utilize maximum point is corresponding in the characteristic curve of F frame number as sub-camera lens partitioning boundary.Video ROAD with in normal video storehouse (http://www.open-video.org/) is an example, and its F characteristic curve is as shown in Figure 1.

(4) sub-camera lens is cut apart

Visible by Fig. 1, except that two big extreme points, also there are two little sawtooth wave in the F characteristic curve.This is because utilize formula (4) to carry out F value when calculating, the noise that causes owing to reasons such as the flash of light in the camera lens, object of which movement and camera lens displacements, but not real sub-camera lens cut-point.

Before maximum value is extracted, at first utilize formula (5) that the F curve is carried out filtering, Fig. 2 is the filtered characteristic curve of Fig. 1 for this reason.

$F^{\&prime;}=F-\stackrel{\&OverBar;}{F}---\left(5\right)$

Wherein:

F ' is filtered judge function curve;

Be filter threshold,

F _Mean, F _StdAverage and the standard deviation of representing the F curve respectively.

After the filtering of F curve, suppose that the functional relation that newly obtains is F '=f (i), i is a frame number.We adopt the second difference point-score to carry out maximum value and extract, shown in formula (6).

sign[f(i+1)-f(i)]-sign[f(i)-f(i-1)]＝-2 (6)

Wherein:

Sign is a sign function,

The frame number at the maximum point place of finally trying to achieve is as shown in Figure 3, and it is the corresponding frame number of maximum point, the i.e. partitioning boundary of sub-camera lens that second order difference result equals-2 places.Can find out that this method can extract the border of sub-camera lens well, realize the sequential of camera lens is cut apart.

3. confirming of the number of key frames in the sub-camera lens

After sub-camera lens is cut apart completion, utilize the rate of change of sub-camera lens to extract key frame.Basic thought of the present invention is: because therefore frame pitch can confirm the key frame number through the interframe range rate from ability descriptor frame differences.The totalframes of supposing certain sub-camera lens is n, and this paper utilizes the sub-shot change rate of formula (7) expression, confirms the key frame number adaptively:

$k=\frac{\mathrm{dist}(1,n)}{\frac{1}{n-1}\underset{i=1}{\overset{n-1}{\mathrm{\&Sigma;}}}\mathrm{dist}(i,i+1)}---\left(7\right)$

Wherein:

K is a number of key frames;

N is the totalframes of sub-camera lens;

Dist (1, n) be the Euclidean distance of sub-camera lens head and the tail two frames;

(i i+1) is the Euclidean distance of adjacent two frames in the moving window to dist;

In the following formula, denominator is represented the average of interframe Euclidean distance in the whole sub-camera lens.

When k≤1, it is little to explain that video content in the sub-camera lens changes, and extracts a frame and gets final product;

When k＞1, the integer of choosing near k is the key frame number.Through calculating, the k of the 3rd sub-camera lens gets 3.

4. key frame is selected

Calculate the Euclidean distance d of interior present frame of sub-camera lens and former frame, get preceding k the maximum pairing frame number of value of distance, this k frame is the key frame of sub-camera lens.

The above; Be merely the preferable embodiment of the present invention, but protection scope of the present invention is not limited thereto, any technician who is familiar with the present technique field is in the technical scope that the present invention discloses; The variation that can expect easily or replacement all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion with the protection domain of claim.

Claims (9)

1. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens is characterized in that this method may further comprise the steps:

Step 1: read all frames of image in the camera lens, extract the color feature vector of each frame;

Step 2: on the basis of step 1, with moving window with combine apart from the separability criterion, camera lens is divided into each sub-camera lens;

Step 3: the number of key frames of obtaining each sub-camera lens according to the interframe range rate;

Step 4: on the basis of step 3, from sorting from big to small, choose the key frame that specifies number according to frame pitch.

2. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 1 is characterized in that said step 2 is specially:

Step 2.1: set up moving window, and try to achieve the two types of sample sets in front and back in the sliding window within class scatter matrix and type between the dispersion matrix;

Step 2.2: on the basis of step 2.1, set up and pass judgment on function;

Step 2.3: the judge function curve of trying to achieve camera lens;

Step 2.4: to passing judgment on function curve filtering, and the frame that will pass judgment on each maximum point correspondence of function curve is regarded as the split position of sub-camera lens.

3. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 2 is characterized in that the computing formula of said judge function is:

$F=\frac{\det \left(S_b\right)}{\det (S_1+S_2)}$

Wherein:

F is for passing judgment on function curve;

S _bBe dispersion matrix between class;

S ₁For the first two/moving window in the within class scatter matrix of sub-camera lens frame;

S ₂Within class scatter matrix for sub-camera lens frame in 1/2nd moving windows of back;

Det (S _b) dispersion determinant of a matrix value between type of being;

Det (S ₁+ S ₂) be the determinant of within class scatter matrix.

4. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 3 is characterized in that said S _bComputing formula be:

S _b＝(m ₁-m ₂)(m ₁-m ₂) ^T

Wherein:

m ₁For the first two/mean vector of sub-camera lens frame in the moving window;

m ₂Mean vector for the sub-camera lens frame in 1/2nd moving windows of back.

5. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 4 is characterized in that said S ₁Computing formula be:

$S_1=\underset{H\&Element;L_1}{\mathrm{\&Sigma;}}(H-m_1){(H-m_1)}^T$

Wherein:

L ₁For the first two/ set that the sample of sub-camera lens frame in the moving window is formed;

H is the proper vector of certain sample frame in the sample set.

6. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 5 is characterized in that said S ₂Computing formula be:

$S_2=\underset{H\&Element;L_2}{\mathrm{\&Sigma;}}(H-m_2){(H-m_2)}^T$

Wherein:

L ₂The set that the sample of the sub-camera lens frame in 1/2nd moving windows of back is formed.

7. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 2 is characterized in that the formula of said judge function curve filtering is:

$F^{\&prime;}=F-\stackrel{\&OverBar;}{F}$

Wherein:

F ' is filtered judge function curve;

is filter threshold.

8. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 7 is characterized in that the computing formula of said

is:

$\stackrel{\&OverBar;}{F}=F_{\mathrm{mean}}+F_{\mathrm{std}}$

Wherein:

F _MeanFor passing judgment on the average of function curve;

F _StdFor passing judgment on the standard deviation of function curve.

9. the self adaptation key frame method for distilling of cutting apart based on sub-camera lens according to claim 7 is characterized in that the computing formula of said number of key frames is:

$k=\frac{\mathrm{dist}(1,n)}{\frac{1}{n-1}\underset{i=1}{\overset{n-1}{\mathrm{\&Sigma;}}}\mathrm{dist}(i,i+1)}$

Wherein:

K is a number of key frames;

N is the totalframes of sub-camera lens;

Dist (1, n) be the Euclidean distance of sub-camera lens head and the tail two frames;

(i i+1) is the Euclidean distance of adjacent two frames in the moving window to dist.

Images