CN102088544B - Fast image stabilization method of dynamic scene video with foreground object - Google Patents

Abstract

The invention belongs to the field of computer image processing, and relates to a fast image stabilization method of a dynamic scene video with a foreground object. The fast image stabilization method comprises the following steps: selecting a template for a frame image, and equally dividing the template into a plurality of sub-blocks; respectively calculating the pixel absolute difference between each sub-block and a corresponding sub-block in a previous row or a previous line, and calculating the average value of each pixel absolute difference; respectively calculating the pixel absolute difference D<k>(i, j) (k=1, K, 4) between each sub-block and the sub-blocks at the upper side, the lower side, the left side and the right side, and setting a threshold value T and a threshold vale T', wherein T is equal to AD and T' is less than T; for each sub-block C(i, j), if the minimum value of D<k>(i, j) is more than T' or less than T, taking the sub-block C(I, j) as a foreground object module; carrying out foreground object separation on all pictures; and carrying out motion compensation on images so as to obtain stabilized images. By using the method provided by the invention, the number of modules which are used for participating in the calculation can be greatly decreased, and the precision and speed of the foreground object separation can be improved.

Description

The quick image stabilization method of dynamic scene video with foreground target

Technical field

The invention belongs to the Computer Image Processing field, can be used for association areas such as video sequence electronic steady image.

Background technology

In actual engineering, the content that video camera is taken often is not only a natural background, also comprises the foreground target of many motions, and like pedestrian, vehicle or the like, these moving targets make video image inside produce certain local motion.When carrying out surely as processing, the estimated accuracy of global motion vector can receive the influence of these local motions and reduce, thereby influences the final effect of steady picture; Therefore when asking for kinematic parameter, need earlier the foreground moving target to be extracted; Disturbing to remove, improve the extractive technique of accuracy of parameter estimation moving target, is comparison popular topic in the image processing techniques; The moving target extraction that mainly can be divided under the static background is extracted two big types with the target under the movement background; For the video sequence of shake, background changes mostly, so the extractive technique of scape target just seems particularly important under the movement background in actual electronic steady image process.It is one of the difficult point in moving target extraction field that moving target under the dynamic background extracts, and adopts the method based on supervised classification and motion vector field to solve at present usually.Based on the method for supervised classification mainly is through a large amount of sample datas of having done the classification mark being learnt, from problem, obtained the input varible value and corresponding target output variable value of training example, and then the relation between study input and target output therefrom; And apply in the actual graphical; Extract moving target,, then need earlier adjacent image to be carried out registration based on the method for motion vector field; Carry out carrying out calculus of differences again after the motion compensation, to realize the extraction of foreground moving target.

More than two kinds of methods though advantage is separately all arranged; And all obtained good effect, these two kinds of methods also all have the weak point of oneself, though can realize that based on the method for supervised classification the moving target under the movement background extracts; But need learn in advance and train; And in practical application, it is very difficult obtaining mark (known class) sample that is used for training and testing in a large number, so this algorithm has significant limitation; Though the method based on motion vector field can extract the foreground moving target more exactly; But the kinematic parameter that need carry out entire image is in advance estimated; And need cut apart video image based on sports ground, amount of calculation is big, and this makes its range of application receive restriction significantly.Therefore, need seek a kind of new target extraction method, this method should be saved time, to satisfy the needs of actual engineering when guaranteeing the moving target extraction effect as much as possible.

Summary of the invention

In the steady fast picture algorithm of the dynamic scene video with foreground target, existing moving target method for distilling more complicated, real-time is poor; Can not obtain gratifying effect; And in many actual engineerings, also than higher, so conventional method will have influence on the effect of operation such as follow-up estimation to the requirement of real-time; In order to address this problem; The present invention propose a kind of can be when guaranteeing the moving target extraction effect, effectively reduce and be used to the module number of participating in calculating, improve the quick image stabilization method of dynamic scene video of arithmetic speed.Technical scheme of the present invention is following:

A kind of quick image stabilization method of dynamic scene video with foreground target comprises the following steps:

1) for a two field picture, selected template is divided equally into m * n size with it and is the sub-piece of M * M, and the every sub-block after dividing equally is designated as C _{I, j}(i=1, K, m; J=1, K, n), sub-piece C _{I, j}The behavior i at place is capable, and the j that classifies as at place is listed as;

2) calculate all capable sub-pieces of i respectively according to formula and be listed as the pixel absolute difference between the corresponding sub-piece with j-1, be designated as with all sub-pieces of capable corresponding sub-piece of i-1 and j row

And

Wherein, x _{I, j}, y _{I, j}Be sub-piece C _{I, j}The coordinate of top left corner apex, (x y) is present frame x to f, the gray value at y place;

3) for sub-piece C _{I, j}, make that the sub-piece of four positions, upper and lower, left and right is R around it _u, R _d, R _l, R _r, then sub-piece C _{I, j}And R _u, R _d, R _l, R _rBetween the pixel absolute difference be designated as

4) Calculation

and the mean

5) capping threshold value T=AD, and a selected suitable lower threshold T ' (T '＜T), for not being positioned at the borderline every sub-block C of template _{I, j}If,

Then think sub-piece C _{I, j}Be the foreground target module;

6) utilize said method that all frame pictures are carried out foreground target and separate, obtain being used for the residue module of motion estimation;

7) after the completion foreground target separated, the affine transformation parameter of trying to achieve according to motion estimation carried out motion compensation to image, the image after surely being looked like.

The present invention is directed to foreground moving target in the dynamic video sequence and influence the problem of global motion vector estimated accuracy; Proposed a kind of method and extracted foreground target based on the absolute difference of piece; Come anticipation background module whether to have enough gradient informations through setting threshold and come estimated motion vector, utilize conventional methods such as finding the solution overdetermined equation to obtain the quick image stabilization method of dynamic scene video of global motion parameter at last.The present invention is with respect to the improvements based on traditional foreground target extraction algorithm such as supervised classification and motion vector field: need not carry out the registration of interframe in advance, confirm through setting dependent thresholds whether the background module has enough gradient informations simultaneously.Comparing biggest advantage with former method is: significantly reduced the module number of participating in calculating, can improve precision and speed that foreground target separates.

Description of drawings

Foreground target extraction effect figure when Fig. 1 judges for adopting threshold value

, black part is divided into the background module that determines.

Foreground target extraction effect figure when Fig. 2 judges for adopting

, black part is divided into the module that influences estimation of motion vectors precision and speed in the image that determines.

Foreground target extraction effect figure when Fig. 3 judges for adopting , chrominance section is to isolate the target context that finally obtains behind the foreground target.

The left figure of Fig. 4, middle figure and right figure are respectively first frame in continuous 3 frames, second frame and the 3rd frame video image in the actual traffic monitor video.

Fig. 5 carries out the result that foreground target extracts for adopting the present invention to continuous 3 two field pictures among Fig. 4.

Fig. 6 is for adopting the present invention to the continuous 3 two field picture separating video foreground targets among Fig. 4, and the module that is used for the kinematic parameter estimation is chosen the result.

Fig. 7 is to the effect behind the steady picture of continuous 3 two field pictures among Fig. 4.

Fig. 8 and Fig. 9 are respectively that continuous 3 frames among Fig. 4 are in the video sequence interframe difference result of steady picture front and back.

Figure 10 surely looks like the interframe PSNR of front and back for continuous 30 frame video sequences.

Embodiment

Below through accompanying drawing and embodiment the present invention is done further explanation.

The present invention is to provide a kind of foreground target extraction algorithm that has combined absolute difference of piece and anticipation local module, its concrete steps are following:

1) at first the entire image coarse positioning is gone out a template f, four limits of four the back gauge entire image in upper and lower, left and right of f are all greater than M pixel, and guarantee that this template can be divided equally into m * n size and be the sub-piece of M * M, and the every sub-block after dividing equally is designated as C _{I, j}(i=1, K, m; J=1, K, n), C _{I, j}The row at place is exactly that i is capable, and the row at place are exactly the j row.

2) the pixel absolute difference between the corresponding sub-piece of all sub-pieces of calculating i capable (or j row) capable with i-1 (or j-1 row) is designated as

(or

):

$D_{i,j}^{\mathrm{row}}=\underset{m=1}{\overset{M}{\mathrm{\&Sigma;}}}\underset{n=1}{\overset{M}{\mathrm{\&Sigma;}}}|f(x_{i,j}+m,y_{i,j}+n)-f(x_{i,j}+m,y_{i,j}-M+n)|$ (1)

$D_{i,j}^{\mathrm{col}}=\underset{m=1}{\overset{M}{\mathrm{\&Sigma;}}}\underset{n=1}{\overset{M}{\mathrm{\&Sigma;}}}|f(x_{i,j}+m,y_{i,j}+n)-f(x_{i,j}-M+m,y_{i,j}+n)|$

X wherein _{I, j}, y _{I, j}Be current block C _{I, j}The coordinate of top left corner apex, (x y) is present frame x to f, and the gray value at y place, M * M are the size of piece.For current block C _{I, j}, making its piece on every side is R _u, R _d, R _l, R _r(representing four positions, upper and lower, left and right respectively), then current block C _{I, j}And R _u, R _d, R _l, R _rBetween the pixel absolute difference be designated as

$D_{i,j}^1=D_{i,j}^{\mathrm{row}},D_{i,j}^2=D_{i+1,j}^{\mathrm{row}}$ (2)

$D_{i,j}^3=D_{i,j}^{\mathrm{col}},D_{i,j}^4=D_{i,j+1}^{\mathrm{col}}$

3) calculate

and average AD:

$\mathrm{AD}=\frac{1}{(m+1)\&times;(n+1)}\underset{i=1}{\overset{m+1}{\mathrm{\&Sigma;}}}\underset{j=1}{\overset{n+1}{\mathrm{\&Sigma;}}}(D_{i,j}^{\mathrm{row}}+D_{i,j}^{\mathrm{col}})---\left(3\right)$

Setting threshold T=AD; if

then think that current block is a foreground target; Otherwise replace current block with black block, as shown in Figure 1.Wherein min representes to get minimum value.

4) if directly with black region shown in Figure 1 as the motion estimation module; The module of then participating in calculating can be a lot; Thereby increase the operand of estimation of motion vectors, and may have the module that causes inaccurate estimation of motion vectors, therefore; The present invention adopts the method for anticipation local module, promptly through set a suitable lower threshold T ' (T '＜T) judge when whether front module has enough gradient informations and come estimated motion vector.When

can remove the module that influences estimation of motion vectors precision and speed in the image, as shown in Figure 2.The foreground target module is separated from Fig. 2, remaining module promptly can be used as the motion estimation module, and is as shown in Figure 3 again.The present invention has set lower threshold T ' and upper limit threshold T, according to

Choose qualified foreground module, the number of modules N that selects for use can adjust through T ', if satisfy N _Min＜N＜N _Max(N wherein _Min, N _MaxThreshold value for N), then finish the operation that module is chosen, otherwise adjustment T ' repeats to choose estimation block.

So far, the selection operation that foreground target separates with specific estimation block finishes, and Fig. 3 is the motion estimation zone of when T '=AD/2, setting, and the foreground target among the figure is almost all cast out, and can guarantee the precision and the speed of estimation of motion vectors.

5) accomplish foreground target and separate after, can adopt motion module among Fig. 3 through Newton iteration or find the solution method such as overdetermined equation and obtain affine transformation parameter, and image is carried out motion compensation, thus the image after surely being looked like.

Select for use the actual traffic monitor video that the method that the present invention proposes is carried out emulation experiment, wherein continuous 3 frame video images are as shown in Figure 4, and the image resolution ratio size is 320 * 240.As can be seen from the figure, significantly rocking has appearred in video image, belongs to dynamic background, and the vehicle that dealing is advanced among the figure is more, has bigger local motion between frame of video.

At first image being divided into size is 16 * 16 fritter, utilizes the present invention that foreground target is extracted, and it is as shown in Figure 6 that the foreground target module is extracted the result.

As can be seen from Figure 5, the foreground target in the video sequence (vehicle) extracts basically.At this moment, the AD that calculates (being threshold value T) is between 900～920.Black region (being the background module) is used for the global motion parameter Estimation among the figure if directly select, and the module of then participating in calculating can be a lot, and this can influence the amount of calculation of piece coupling and estimation of motion vectors to a great extent.Therefore; The present invention adopts the method for anticipation local module; Before carrying out estimation of motion vectors; Through setting anticipation threshold value T '=AD/2, the reliability that modular movement vector is estimated is carried out anticipation, rejecting may cause the module of inaccurate estimation of motion vectors; The only selected module that satisfies condition

is used for kinematic parameter and estimates that the module of extraction is as shown in Figure 6.

From Fig. 5, can find out that the present invention has successfully separated the foreground target in the video sequence in 6, participate in the module number that kinematic parameter estimates among Fig. 6 and be merely about 50, be about 1/6 of entire image module number, significantly reduced amount of calculation.

In order to verify the validity of selected module to estimation of motion vectors, the present invention has chosen continuous 30 two field pictures and has carried out surely as experiment in the actual monitored video.Utilize template matching algorithm, obtain initial threshold T ₀=100, carry out the piece coupling more respectively to the black region module (background module) among extraction front module, Fig. 5, and through the module after the anticipation screening, experimental result is as shown in table 1.

Table 1 is chosen the impact effect of different motion estimation block to matching result

Can find out from table 1; Compare with the background module after original module and foreground target extract; Module number through after the anticipation screening obviously reduces, thereby greatly reduces module calculation of Matching amount, only be about average match time with original module calculate 1/5; With the background module calculate 1/4, and the compensation after picture frame between the PSNR value also a little more than the result after mating with original module and background module.The above results shows; Adopt the method for absolute difference of piece and anticipation local module, not only reduced greatly to be used for the module number that kinematic parameter is estimated, reduced operation time; And rejected the module that may cause inaccurate estimation of motion vectors, improved the accuracy of template matches.

Utilize the present invention that the actual traffic monitor video is carried out surely as processing, 3 two field pictures compensation back effect is as shown in Figure 7 among Fig. 4, and as can be seen from the figure, the present invention has successfully removed the shake of video pictures.

Fig. 8 and Fig. 9 are respectively the video sequence interframe difference result that surely looks like front and back, and as can be seen from the figure, it is obvious surely to look like preceding frame difference result; Be to have tangible side-play amount between consecutive frame; And the consecutive frame after the compensation overlaps basically, and part foreground moving target is only arranged, and has obtained surely to look like effect preferably.

Interframe PSNR before and after continuous 30 frame video sequences surely look like is shown in figure 10; As can be seen from the figure; The PSNR value of stable back image sequence interframe is apparently higher than the PSNR of former sequence; On average exceed about 7dB, explain that the present invention has obtained the dynamic video sequence with foreground target and surely look like effect preferably.

Claims (1)

1. the quick image stabilization method of dynamic scene video with foreground target comprises the following steps:

1) for a two field picture, selected template is divided equally into m * n size with it and is the sub-piece of M * M, and the every sub-block after dividing equally is designated as C _{I, j}, wherein, i=1 ..., m; J=1 ..., n, sub-piece C _{I, j}The behavior i at place is capable, and the j that classifies as at place is listed as;

2) calculate all capable sub-pieces of i respectively according to formula and be listed as the pixel absolute difference between the corresponding sub-piece with j-1, be designated as with all sub-pieces of capable corresponding sub-piece of i-1 and j row

And

Wherein, x _{I, j}, y _{I, j}Be sub-piece C _{I, j}The coordinate of top left corner apex, (x y) is present frame x to f, the gray value at y place;

3) for sub-piece C _{I, j}, make that the sub-piece of four positions, upper and lower, left and right is R around it _u, R _d, R _l, R _r, with sub-piece C _{I, j}And R _u, R _d, R _l, R _rBetween the pixel absolute difference be designated as

4) Calculation?

and the mean

5) capping threshold value T=AD, and selected suitable lower threshold T ' less than T are for every sub-block C _{I, j}If, Then think sub-piece C _{I, j}Be the foreground target module;

6) utilize said method that all frame pictures are carried out foreground target and separate, obtain being used for the residue module of motion estimation;

7) after the completion foreground target separated, the affine transformation parameter of trying to achieve according to motion estimation carried out motion compensation to image, the image after surely being looked like.

Images