CN105488812A - Motion-feature-fused space-time significance detection method - Google Patents

Abstract

The invention belongs to the field of image and video processing, in particular to a space-time significance detection method which fuses space-time significance and motion features. The space-time significance detection method comprises the following steps: firstly, utilizing a superpixel partitioning algorithm to express each frame of image as one series of superpixels, and extracting a superpixel-level color histogram as features; then, obtaining a spatial salient map through the calculation of the global comparison and the spatial distribution of colors; thirdly, through optical flow estimation and block matching methods, obtaining a temporal salient map; and finally, using a dynamic fusion strategy to fuse the spatial salient map and the temporal salient map to obtain a final space-time salient map. The method fuses the space significance and the motion features to carry out significance detection, and the algorithm can be simultaneously applied to the significance detection in dynamic and static scenes.

Description

A kind of time and space significance detection method of fusional movement feature

1, technical field

The invention belongs to image and field of video processing, specifically a kind of time and space significance detection method of fusional movement feature.The present invention by based on based on the conspicuousness detection model in region, first, with super-pixel partitioning algorithm each two field picture is expressed as a series of super-pixel and the color histogram extracting super-pixel level as feature; Then, to be obtained moving significantly figure by the method for light stream estimation and Block-matching, obtain significant spatial figure according to the global contrast of color and space distribution; Finally, use a kind of dynamic fusion strategy to be merged by remarkable for motion figure and significant spatial figure and become final space-time remarkable figure.The method has merged motion feature and has carried out conspicuousness detection, can apply in Static and dynamic scene simultaneously.

2, background technology

Conspicuousness detection refers to locates exactly and is extracted in video and image the region comprising larger quantity of information, the attracting notice of energy.In image processing process, give salient region of image by higher processing priority, both can reduce the complexity of computation process, the efficiency of image procossing can be improved again.Therefore, conspicuousness detects and is with a wide range of applications in fields such as target identification, image retrieval, encoding video pictures.

According to the difference of the information source image of process, conspicuousness detection model can be divided into spatial domain conspicuousness detection model and time and space significance detection model.Spatial domain conspicuousness for static scene detects, and mainly contains the conspicuousness detection method based on biological heuristic models and feature integration theory, and the difference that this algorithm calculates feature on different scale by center-surrounding operator obtains significance.The saliency analyzed based on local contrast of the characteristic distance by calculating pixel and its neighboring pixel is had to detect.Also having Iamge Segmentation is some regions, and the spatial character of combining image and color contrast calculate the salient region detecting method of global contrast.

Time and space significance detection algorithm under dynamic scene not only will consider detection space salient region, also will consider the influence factor of conspicuousness on time shaft, as object of which movement, natural conditions change, camera move etc.According to the difference of account form, time and space significance detection algorithm is mainly divided into four classes: the model that (1) feature based merges: on the basis of saliency detection model, add motion feature, and the difference calculating two continuous frames image obtains the conspicuousness produced of moving; (2) based on the model of space plane: compose on the basis of remaining method, think on a timeline, the pixel that frame sequence is formed is at X-T, Y-T plane all meets the remaining conclusion of spectrum, by X-T, Y-T plane regards two-dimensional matrix as, carries out low-rank-Its Sparse Decomposition respectively, and finally merging becomes final remarkable figure; (3) based on the model of frequency-domain analysis: from frequency-domain analysis angle, by brightness, color, motion three category feature combination quaternary array, then obtain the phase spectrum of video on time-space domain by hypercomplex number Fourier transform, and utilize this phase spectrum to detect the conspicuousness of video; (4) based on the model of background detection: gauss hybrid models is also used to time and space significance and detects, and in single scene, the foreground area calculated by gauss hybrid models is as marking area.Above method is by static conspicuousness and the simple linear fusion of dynamic conspicuousness or only highlight motion feature, have ignored the dynamic perfromance in scene and spatial character, is difficult to obtain marking area accurately.

3, summary of the invention

The present invention is by based on based on the conspicuousness detection model in region, the method incorporating the contrast of light stream vectors block extracts the motion feature of image sequence, and propose a kind of by the strategy of spatial domain conspicuousness and time domain conspicuousness dynamic fusion, can apply in the natural scene of Static and dynamic simultaneously.

(1) super-pixel segmentation and feature extraction

Use simple linear iteration cluster by each two field picture F _tbe expressed as a series of super-pixel using super-pixel as basic processing unit, the quantity of processing unit can be reduced, and can ensure that final testing result can give prominence to remarkable object equably.The extraction of color characteristic adopts color histogram, quantizes each passage of Lab color space to obtain 12 different values, the quantity of color is reduced to qc=12 ³=1728, for each super-pixel to calculate in super-pixel all pixels in the average of Lab space, and carry out quantification and obtain color histogram CH _t, finally by color histogram normalization, make ${\mathrm{\Σ}}_{k=1}^{qc}{\mathrm{CH}}_t\left(k\right)=1.$

(2) based on the time conspicuousness of motion feature

The present invention adopts the method for light stream estimation and Block-matching to extract the motion feature of image sequence.The basic thought of light stream estimation method be by moving image function f (x, y, t) as basic function, setting up optical flow constraint equation according to image intensity conservation principle, by solving optical flow constraint equation, calculating kinematic parameter.For present frame F _t, use its former frame F _t-1as with reference to frame, calculate F by light stream estimation method _tmotion vector field (u _{(x, y)}, v _{(x, y)}), for present frame F _tin each super-pixel calculate its average motion vector size

In order to overcome the impact of background motion and camera shake, the present invention finds the super-pixel of mating most with present frame by the method for Block-matching in former frame, and the relative movement values calculating this super-pixel and its background super-pixel is as its saliency value.Concrete implementation method: at frame F _t-1the super-pixel that middle selection is mated with it most ? the super-pixel be connected is therewith as being associated super-pixel set ψ _i, represent the center of super-pixel i, j respectively.Therefore, super-pixel time conspicuousness be defined as:

$S_{motion}\left({\mathrm{sp}}_t^i\right)=\frac{\underset{{\mathrm{sp}}_{t-1}^j\∈{\mathrm{\ψ}}_i}{\Σ}\mathrm{\λ}({\mathrm{sp}}_t^i,{\mathrm{sp}}_{t-1}^j)\·S_{motion}\left({\mathrm{sp}}_{t-1}^j\right)}{\underset{{\mathrm{sp}}_{t-1}^j\∈{\mathrm{\ψ}}_i}{\Σ}\mathrm{\λ}({\mathrm{sp}}_t^i,{\mathrm{sp}}_{t-1}^j)}---\left(1\right)$

Wherein, frame-to-frame correlation $\mathrm{\λ}({\mathrm{sp}}_t^i,{\mathrm{sp}}_{t-1}^j)=\sqrt{c\left(i\right)\·c\left(j\right)}\·\[1-\frac{\left|\right|{\mathrm{\μ}}_t^i-{\mathrm{\μ}}_{t-1}^j+{\mathrm{mmv}}_t^i\left|\right|}{d}\],$ C (i) and c (j) represents the color value of super-pixel i and j after Lab color space quantization respectively.

(3) spatial saliency

The present invention calculates the spatial saliency of each two field picture according to the global contrast of color and space distribution.For the super-pixel in present frame its color global contrast saliency value is defined as:

$S_{GC}\left(i\right)=\underset{j=1}{\overset{n_t}{\Σ}}{f}_j\·\left|\right|c\left(i\right)-c\left(j\right)\left|\right|---\left(2\right)$

Wherein, f _jsuper-pixel the probability that occurs in entire image of histogram, c (i) and c (j) represents the color value of super-pixel i and j after Lab color space quantization respectively.

The space distribution of color also can the conspicuousness of effect diagram picture, and color distribution is tightr, then conspicuousness is higher, and therefore the space distribution conspicuousness of color is defined as:

$S_{sc}\left(i\right)=\frac{\underset{j=1}{\overset{n_t}{\Σ}}\mathrm{\λ}({\mathrm{sp}}_t^j,{\mathrm{sp}}_t^j)\·D\left({\mathrm{sp}}_t^j\right)}{\underset{j=1}{\overset{n_t}{\Σ}}\mathrm{\λ}({\mathrm{sp}}_t^i,{\mathrm{sp}}_t^j)}---\left(3\right)$

Wherein, for the center of super-pixel j and the distance of picture centre, $\mathrm{\λ}({\mathrm{sp}}_t^i,{\mathrm{sp}}_t^j)=\sqrt{c\left(i\right)\·c\left(j\right)}\·\[1-\frac{\left|\right|{\mathrm{\μ}}_t^i-{\mathrm{\μ}}_t^j\left|\right|}{d}\],$ When the color of super-pixel i and j is more close and distance is less, value larger.

Finally color overall situation distribution conspicuousness and space distribution conspicuousness are merged the spatial saliency obtaining image:

$S_{spatial}\left({\mathrm{sp}}_t^i\right)=S_{GC}\left({\mathrm{sp}}_t^i\right)\·S_{SC}\left({\mathrm{sp}}_t^i\right)---\left(4\right)$

(4) time and space significance

Time conspicuousness and spatial saliency adaptive line are merged and obtains space-time remarkable figure, that is:

$Saliency\left({\mathrm{sp}}_t^i\right)=\mathrm{\α}\·S_{spatial}\left({\mathrm{sp}}_t^i\right)+(1-\mathrm{\α})\·S_{nmporal}\left({\mathrm{sp}}_t^i\right)---\left(5\right)$

Weights α is defined as:

$\mathrm{\α}=max\[\frac{med({\mathrm{MS}}_{t-f},\mathrm{...},{\mathrm{MS}}_{t-1})}{med({\mathrm{MS}}_{t-f},\mathrm{...},{\mathrm{MS}}_{t-1})+{\mathrm{MS}}_t},0.5\]---\left(6\right)$

Wherein, represent the number of pixels in super-pixel i.Under dynamic scene when moving more obvious, the weights of time conspicuousness are larger, and under static scene, α is directly set to 1.

4, accompanying drawing explanation

Accompanying drawing is principle of the present invention and performing step.

5, embodiment

Accompanying drawing 1 is this working of an invention process flow diagram, and its concrete steps are:

(1) Image semantic classification: each two field picture of input is divided into by SLIC super-pixel partitioning algorithm the basic processing unit that even, the compact super-pixel of a series of size detects as conspicuousness.

(2) extraction of color characteristic: for each two field picture, in units of super-pixel, calculates all pixels in super-pixel and, in the average of Lab space, carries out quantification and obtain color histogram CH _tand normalization makes

(3) time conspicuousness calculates: calculate F by light stream estimation method _trelative to former frame F _t-1motion vector field (u _{(x, y)}, v _{(x, y)}), and calculate super-pixel interior mean vector field size then found in former frame and the super-pixel that present frame mates most and relevant super-pixel set thereof by the method for Block-matching, the time using formula (1) to obtain image significantly schemes.

(4) spatial saliency calculates: according to based on based on the static conspicuousness detection model in region, using formula (2) and formula (3) to obtain respectively take super-pixel as color global contrast conspicuousness and the Color-spatial distribution conspicuousness of base unit, and passes through formula (4) and merged and become final significant spatial figure.

(5) time and space significance calculates: according to formula (5), time conspicuousness and the fusion of spatial saliency adaptive line are obtained space-time remarkable figure.

Claims (5)

1. a time and space significance detection method for fusional movement feature, is characterized in that:

Often a two field picture is carried out super-pixel segmentation, and extract color histogram using super-pixel as elementary cell, the spatial saliency of each two field picture is calculated according to the global contrast of color and space distribution, the method of light stream estimation and Block-matching is adopted to extract the motion feature of image sequence, by spatial domain conspicuousness and time domain conspicuousness dynamic fusion, enable this algorithm apply to the detection of motion feature in the natural scene of Static and dynamic simultaneously.

2. the time and space significance detection method of a kind of fusional movement feature as claimed in claim 1, is characterized in that described super-pixel segmentation and super-pixel field color histogram feature extracting method are:

Use linear iteration clustering method by each two field picture F _tbe expressed as a series of super-pixel using super-pixel as basic processing unit, the quantity of processing unit can be reduced, and can ensure that final testing result can give prominence to remarkable object equably.

Color histogram is calculated to the color characteristic splitting the super-pixel region obtained.Each passage of Lab color space is quantized to obtain 12 different values, the quantity of color is reduced to qc=12 ³=1728.For each super-pixel to calculate in super-pixel all pixels in the average of Lab space, and carry out quantification and obtain color histogram CH _t, finally by color histogram normalization, make

3. the time and space significance detection method of a kind of fusional movement feature as claimed in claim 1, is characterized in that described spatial saliency computing method are:

The present invention calculates the spatial saliency of each two field picture according to the global contrast of color and space distribution.For the super-pixel in present frame its color global contrast saliency value is defined as:

Wherein, f _jsuper-pixel the probability that occurs in entire image of histogram, c (i) and c (j) represents the color value of super-pixel i and j after Lab color space quantization respectively.

The space distribution of color also can the conspicuousness of effect diagram picture, and color distribution is tightr, then conspicuousness is higher, and therefore the space distribution conspicuousness of color is defined as:

Wherein, for the center of super-pixel j and the distance of picture centre, when the color of super-pixel i and j is more close and distance is less, value larger.

Finally color overall situation distribution conspicuousness and space distribution conspicuousness are merged the spatial saliency obtaining image:

。

4. the time and space significance detection method of a kind of fusional movement feature as claimed in claim 1, is characterized in that the method for described employing light stream estimation and Block-matching to extract the computing method of the motion feature of image sequence is:

The basic thought of light stream estimation method be by moving image function f (x, y, t) as basic function, setting up optical flow constraint equation according to image intensity conservation principle, by solving optical flow constraint equation, calculating kinematic parameter.For present frame F _t, use its former frame F _t-1as with reference to frame, calculate F by light stream estimation method _tmotion vector field (u _{(x, y)}, v _{(x, y)}), for present frame F _tin each super-pixel calculate its average motion vector size

In order to overcome the impact of background motion and camera shake, the present invention finds the super-pixel of mating most with present frame by the method for Block-matching in former frame, and the relative movement values calculating this super-pixel and its background super-pixel is as its saliency value.Concrete implementation method: at frame F _t-1the super-pixel that middle selection is mated with it most ? the super-pixel be connected is therewith as being associated super-pixel set ψ _i, represent the center of super-pixel i, j respectively.Therefore, super-pixel time conspicuousness be defined as:

Wherein, frame-to-frame correlation c (i) and c (j) represents the color value of super-pixel i and j after Lab color space quantization respectively.

5. the time and space significance detection method of a kind of fusional movement feature as claimed in claim 1, is characterized in that described time conspicuousness and spatial saliency adaptive line merge the computing method obtaining space-time remarkable figure and be:

Weights α is defined as:

Wherein, represent the number of pixels in super-pixel i.Under dynamic scene when moving more obvious, the weights of time conspicuousness are larger, and under static scene, α is directly set to 1.