CN106550174A - A kind of real time video image stabilization based on homography matrix - Google Patents

Abstract

The invention discloses a kind of real time video image stabilization based on homography matrix, is carried out to the video sequence that the equipment such as hand-held DV, unmanned plane are obtained steady as processing.The method of the present invention comprises the steps：The angle point being evenly distributed in A, extraction video image；B, the light stream vector for calculating interframe, track the angle point of interframe movement using light stream vector；C, the angle point tracked using the algorithm correction of layering motion correction；D, the homography matrix that interframe is asked for using random sampling unification algorism；E, separates active motion compensation amount, the distortion correction amount of interframe using Kalman filter；F, homography matrix is used, motion compensation quantity and distortion correction amount carry out steady stable present frame output being obtained as conversion to video image.The present invention can effectively remove the shake of video sequence presence, and algorithm complex is relatively low, and the speed of service is fast, provides good using value for real-time processing system for video.

Description

A kind of real time video image stabilization based on homography matrix

Technical field

The invention belongs to the technical field of video information process, the present invention is a kind of video sequence based on homography matrix The fields such as digital image stabilization method, it is adaptable to handheld device, unmanned plane.

Background technology

UAV (Unmanned Aerial Vehicle) plays more and more important effect in the field of taking photo by plane at present.Due to Oneself motor vibration and the jiggly impact of flight path are received, the video that unmanned plane shoots inevitably has shake Problem.These shakes have not only had a strong impact on viewing effect, and can be to the target detection of video sequence, target following and mesh Mark not Deng post processing bring larger error.Therefore, the high dither tool in video is removed using fast and effectively Video stabilization There is important Practical significance.

Existing Video Stabilization algorithm is broadly divided into two big class：One is the Video stabilization converted based on 2D images, and two is base In the Video stabilization of 3D scene rebuildings.

The motion of the actual 3D of camera is reduced at the plane motion of 2D based on the Video stabilization that 2D images are converted Reason.This kind of Video stabilization is mainly using the block feature and point feature of image carrying out the estimation of inter-frame information.It is 30FPS in frame per second Video flowing in, the interframe time differenceRelative motion is less, using four parameters 2D similarity transformation models or six parameters it is imitative Penetrate motion of the transformation model to interframe to be modeled, corresponding active movement amount pair is separated using the low pass filter of suitable parameter Registering image carries out motion compensation, has certain removal effect to video jitter.But, affine mould used in 2D Video stabilizations Type, the camera path of the 2D that scale model is estimated and the 3D camera paths non-equivalence of reality, this causes such Video stabilization pair Obvious Video Stabilization effect is converted in visual angle and the depth of field unsatisfactory.

SFM (Struct From Motion) is used based on the Video Stabilization algorithm of 3D scene rebuildings, using 2D images Sequence estimation goes out the 3D structures and the movable information of 3D of scene.Then the camera motion path of 3D is filtered using these information Popin is slided, and using final smooth rear parameter compensation shake frame.This Video stabilization based on scene rebuilding is regarded for some Angle and depth of field change significantly shake video has steady as effect well.But the algorithm is computationally intensive, parameter was calculated Degree relies on the tracking accuracy with interframe feature, thus robustness is general.

Paper name：Content preserving warps for 3D video stabilization, periodical：ACM Transactions on Graphics, time：2009.Liu et al. proposes a kind of based on the steady as calculating of 3D scene rebuildings Method, and depth camera is introduced into during steady picture is processed to improve the robustness of image stabilization system.But the algorithm needs to carry out 3D to scene Rebuild, amount of calculation is larger, it is impossible to realizes real-time processing, is difficult to apply in real-time scene.

Paper name：Calibration-free rolling shutter removal, meeting：IEEE International Conference on Computational Photography (ICCP), time：2012. Angle point in Grundmann et al. extraction images is used as feature, and tracks interframe angle point using optical flow method.But track algorithm pair There is big motion in interframe, frame ambiguity effect is degenerated obvious.

For above-mentioned background content, a kind of Fast Digital Image Stabilization algorithm of effective process 3D camera path is studied, to real-time system System is using significant.

The content of the invention

The invention aims to overcome the shortcomings of existing Video stabilization, there is provided a kind of based on the real-time of homography matrix Video Stabilization algorithm, can effectively remove, recover the video of stable and continuous, together When accomplish relatively low complexity and the speed of service faster, provide good using value for real-time application system.

Technical scheme：

A kind of real time video image stabilization based on homography matrix, step are as follows：

(1) obtain the frame image features point set calculating being evenly distributed to tremble using the Harris Corner Detection Algorithms of distance restraint Current color image is first transformed into gray level image by the Harris characteristic points of dynamic video sequence image, is examined using Harris angle points Method of determining and calculating obtains angle point, and enters row constraint with apart from r, obtains equally distributed frame image features point set；

(2) using the coordinate of the optical flow tracking algorithm keeps track next frame angle point for being layered affine constraint

1) length and width are respectively with w, the image of h sets up 5 layers of yardstick pyramid, and the corresponding length of L tomographic images is a width of

0th layer is original image yardstick, meanwhile, the corresponding angular coordinate of L layers is：

2) the interframe angle point under same yardstick is tracked in the bottom, tracking problem is converted into into inter frame image window SSD letters Number, minimizes：

Wherein, I (x, y), J (x+ υ_x,y+υ_y) t is represented respectively₁,The image at moment；(2·ω_x+1)×(2·ω_y+ 1) represent with (p_x,p_y) centered on neighborhood window size (see Fig. 6)；Pixel (p_x,p_y) elapsed timeMotion vector (υ_x,υ_y)；

Tracking problem is further converted toFinal solution obtains the corresponding angular coordinate (x of L layers^L,y^L)；

3) affine constraint is carried out to the angle point set that the tracking of L layers is obtained, using error formula：

Wherein, F^LIt is next frame L layer global homography models, its form (5)；It is F^L CorrespondenceDimensional variation coefficient；Represent present frame L layer angular coordinates；Represent by with The corresponding coordinate of next frame L layers that track is obtained；

The corresponding error threshold of L layers：

Work as ε^LMore than correspondence threshold value th^L, using updating formula：

Initial error th is set⁰=3, L layers angular coordinate after correction

4) search starting point of L-1 layers isThe angle point of L-1 layers is repeated it is above-mentioned 2)- 3) tracking process, obtains the 0th layer i.e. corresponding angular coordinate of original image yardstick next frame by iterating；

(3) frame global homography matrix H before and after being calculated under original scale using RANSAC iterative algorithms⁰, its form (5)：

(4) each amount of exercise before and after separating in the global homography matrix of frame, obtains corresponding using Kalman filter Smooth active movement amount

1) make H '=(H⁰)^-1, to H ' disengaging movement amounts；The translation T of amount of exercise to be separated comprising relative former frame (x, Y), the distortion D of 2D anglecs of rotation θ, 2D scaling Z and homography matrix；

2) separation of actual motion amount is carried out using the method for four point transformation, define the length and width correspondence of original rectangle ABCD Be frame of video length and width, area is S, obtains A ' B ' C ' D ' to rectangular transform using H ', and area is S ', now amount of zoom：

Calculate A ' B ' and C ' D ' angle thetas in the horizontal direction₁And θ₃, angle thetas of the A ' D ' and B ' C ' in vertical direction₄And θ₂, Reduce the 2D angular errors brought by distortion with averaging method：

Because scaling and rotation can affect the calculating of translational movement, this Video stabilization is using the Z and θ for having obtained to quadrangle A ' B ' C ' D ' compensate conversion, remove rotation and scale；Center before and after then calculating is converted respectively C (x, y), C ' (x, Y), obtaining translational movement (see Fig. 5) is：

T(t_x,t_y)=C (x, y)-C ' (x, y) (10)

3) translation T, the rotation θ and scaling Z to having obtained is filtered using Kalman filter and can obtain correspondence Active movement amount I_T、I_θAnd I_Z, wherein for the state equation of two-dimensional vector T correspondence Kalman filter is：

Corresponding observing matrix is：

The state equation of the corresponding Kalman filter of scalar θ, Z：

Observing matrix：

(1 0 0) (14)

Wherein dt is the sampling interval；

4) above-mentioned amount of exercise is 2D, and homography conversion can bring the pattern distortion of real visual angle change and generation, it is necessary to This distortion is corrected.Using the translation T of existing 2D, θ is rotated, scaling Z enters line translation and remove 2D to put down to A ' B ' C ' D ' Face is moved, and obtains A " B " C " D ", and A " B " C " D " only exists the distortion brought by 3D visual angle changes.The karr set up using translation T Graceful wave filter to conversion after four apex coordinate A " B " C " D " be filtered respectively and obtain A_iB_iC_iD_i。A_iB_iC_iD_iTo ABCD tables Show the corresponding active variable quantity of camera 3D visual angle changes, i.e., required distortion correction amount I_D。

(5) the steady as conversion of nth frame is calculated using the global homography for having obtained and each smooth active movement amount MODEL C_N, image is carried out steady as conversion.

1) nth frame does homography conversion H of benchmark with N-1 frames_N, 2D plane motion compensation matrix motionH_N, 3D visual angles Change corresponding distortion compensation conversion distortH_N, Compensation Transformation of the nth frame with reference to N-1 frames is obtained using above-mentioned conversion：

T_N=motionH_N·distortH_N·H_N (15)

In order to the frame of video for obtaining continuous-stable is exported, using the first frame as global reference frame, therefore final nth frame is steady As transformation model is：

C_N=T₁·...·T_N-1·T_N (16)

Using C_NEnter line translation to present frame and obtain stable two field picture.

The beneficial outcomes of the present invention：

(1) interframe corner location is tracked using the optical flow tracking method for being layered affine constraint, takes full advantage of and regard The information of frequency interframe.Meanwhile, by Layer constraint reduction transmission error between layers.With traditional unconfined light of layering Stream tracking contrast, the optical flow tracking of the affine constraint of the multilayer in this algorithm can significantly improve interior ratio after tracking.

(2) present invention uses the homography model of 8 frees degree is modeled calculating to Inter-frame Transformation, change is not only wrapped The translation T rotation θ scaling Z of 2D planes are contained, but also have included the image change brought by camera perspective change so that interframe Image can be more accurately registering.

(3) present invention carries out smothing filtering to each component motion using Kalman filter, while becoming for homography Changing the distortion produced in successive frame carries out the separation of amount of distortion, and using the active movement amount of Kalman filter point distortion, Steady in topography is obtained better than tradition as in based on similar matrix, the Video stabilization of affine matrix.

(4) inventive algorithm complexity is only the linear function of image size, for the Video processing of 720 × 512 sizes Speed reaches 30FPS, realizes real-time processing.

Description of the drawings

Fig. 1 is a kind of real time video image stabilization schematic flow sheet based on homography matrix.

Fig. 2 is the optical flow tracking schematic diagram for being layered affine constraint.

Fig. 3 is to whether there is distortion correction Comparative result, wherein, (a) pending frame；B () is undistorted timing result； (c) this method distortion correction result.

Fig. 4 is the result of three kinds of digital image stabilization methods of video of taking photo by plane, and under reference frame, 21 frames and 381 frames, (a) is original respectively Two field picture；B () is steady as result based on similarity transformation；The steady picture result of (c) L1 norm optimizations；D () is steady as result herein.

Fig. 5 is four point transformation disengaging movement amount schematic diagram.

Fig. 6 seeks poor window schematic diagram for SSD neighborhoods.

Specific embodiment

Below in conjunction with accompanying drawing and technical scheme, the specific embodiment of the present invention is further illustrated.

Embodiment

A, nth frame image is obtained from video, original angle point set is obtained using Harris Corner Detection Algorithms, for The image of 720 × 512 yardsticks, selects constraint distance to be r=5pixel, in order to take into account the demand of real-time processing, arranges maximum angular Points M is set to 200.

B, reading N+1 two field pictures, using the optical flow tracking algorithm keeps track nth frame angle point of Layer constraint in N+1 frames Position, specific flow process is as shown in Figure 2.

B1, the gray level image to original N+1 two field pictures set up image pyramid, and in actual algorithm, the pyramid number of plies sets It is set to the error threshold th of the 5, the 0th layer of affine constraint⁰=3pixel.Threshold value th of L layers is calculated by formula (5)^L(L ＞ 0), and using formula (2) obtain the corresponding tracking initial point of L layers.

B2, using formula (3) utilizeObtain light stream vector υ_opt, nth frame characteristic point is then obtained in N+1 frames The corresponding coordinate of L layers.

B3, the matching point set using two frames in front and back in L layers calculate the global affine constraint matrix F of L layers^L, utilize Formula (4) (6) is corrected to the point set of N+1 frame L layers.

C, using the corresponding set of characteristic points of frame in front and back, the overall situation iterated to calculate by RANSAC under original image yardstick is single Answering property matrix H⁰, the estimate of the exterior point ratio of wherein RANSAC is set to 0.1, and the pixel error upper limit is 3pixel, interior ratio Lower limit be 0.25.

D, separate each component motion from global homography inverse of a matrix matrix, obtain smooth by Kalman filtering Active movement component.

D1, calculating homography matrix H⁰Inverse matrix H ', H ' represent nth frame to N+1 frames kinematic matrix.

D2, one rectangle ABCD of initialization, its centre coordinate C (x, y), long width values are respectively the length and width of image.Use H ' enters line translation to rectangle ABCD, the quadrangle A ' B ' C ' D ' after being converted, and then calculates its centre coordinate C ' (x, y). Translation vector T (x, y) is calculated using formula (7) (8) (9), anglec of rotation θ scales Z.

D3, above three component motion is filtered using Kalman filter, is provided with the R of Kalman filter =100, Q=0.01, set interval dt=1/30s.The active movement for obtaining three components after filtering afterwards is I_T、I_θ、 I_Z。

D4, using the translation T of existing 2D, rotate θ, scaling Z enters line translation to A ' B ' C ' D ' and removes 2D plane motions, obtains To A " B " C " D ", operation is filtered using the Kalman filter model of translation filtering respectively to four summits of A ' B ' C ' D ', is obtained To A_iB_iC_iD_i, calculate A_iB_iC_iD_iTo homography conversion I of ABCD_D, I_DIt is exactly required abnormal for successive frame homography conversion Become correction matrix.

E, calculated using formula (14) (15) it is final steady as compensation matrix C_N, and line translation is entered to present frame obtain surely Settled prior image frame.

In F, example, the key frame for updating angle point is set to frame number for iFrame=10i's (i=1,2,3,4 ...) Frame.

Claims (1)

1. a kind of real time video image stabilization based on homography matrix, it is characterised in that step is as follows：

(1) the frame image features point set being evenly distributed is obtained using the Harris Corner Detection Algorithms of distance restraint

The Harris characteristic points of shake video sequence image are calculated, current color image is transformed into into gray level image first, used Harris Corner Detection Algorithms obtain angle point, and enter row constraint with apart from r, obtain equally distributed frame image features point set；

(2) using the coordinate of the optical flow tracking algorithm keeps track next frame angle point for being layered affine constraint

1) length and width are respectively with w, the image of h sets up 5 layers of yardstick pyramid, and the corresponding length of L tomographic images is a width of

$\begin{array}{c}{w}^L=\frac{w^{l-1}+1}{2}\\ h^L=\frac{h^{L-1}+1}{2}\end{array}---\left(1\right)$

0th layer is original image yardstick, meanwhile, the corresponding angular coordinate of L layers is：

$\begin{array}{c}{x}^L=\frac{x^{L-1}}{2}\\ y^L=\frac{y^{L-1}}{2}\end{array}---\left(2\right)$

2) the interframe angle point under same yardstick is tracked in the bottom, tracking problem is converted into into inter frame image window SSD functions, is asked Minimum of a value：

$\mathrm{\ϵ}({\mathrm{\υ}}_x,{\mathrm{\υ}}_y)=\min \left(\underset{x=p_x-{\mathrm{\ω}}_x}{\overset{p_x+{\mathrm{\ω}}_x}{\Σ}}\underset{y=p_y-{\mathrm{\ω}}_y}{\overset{p_y+{\mathrm{\ω}}_y}{\Σ}}{\left(I\left(x,y\right)-J\left(x+{\mathrm{\υ}}_x,y+{\mathrm{\υ}}_y\right)\right)}^2\right)---\left(3\right)$

Wherein, I (x, y), J (x+ υ_x,y+υ_y) t is represented respectively₁, t₁The image of+Δ t；(2·ω_x+1)×(2·ω_y+1) Represent with (p_x,p_y) centered on neighborhood window size；Pixel (p_x,p_y) elapsed time Δ t motion vector (υ_x,υ_y)；

Tracking problem is further converted toFinal solution obtains the corresponding angular coordinate (x of L layers^L,y^L)；

3) affine constraint is carried out to the angle point set that the tracking of L layers is obtained, using error formula：

${\mathrm{\ϵ}}^L=\left|\right|\mathrm{\ω}.\left(\begin{array}{ccc}{x}_A^L& y_A^L& 1\end{array}\right)-F^L.\left(\begin{array}{ccc}{x}_B^L& y_B^L& 1\end{array}\right)\left|\right|---\left(4\right)$

Wherein, F^LIt is next frame L layer global homography models, its form (5)；It is F^LCorrespondenceDimensional variation coefficient；Represent present frame L layer angular coordinates；Represent by tracking The corresponding coordinate of next frame L layers for arriving；

$F^L=\left[\begin{array}{ccc}{m}_0& m_1& m_2\\ m_3& m_4& m_5\\ m_6& m_7& 1\end{array}\right]---\left(5\right)$

The corresponding error threshold of L layers：

${\mathrm{th}}^L=\frac{{\mathrm{th}}^0}{2^L}---\left(6\right)$

Work as ε^LMore than correspondence threshold value th^L, using updating formula：

$\left[\begin{array}{ccc}{x}_B^L& y_B^L& 1\end{array}\right]=\frac{1}{\mathrm{\ω}}\·F^L\·\left[\begin{array}{ccc}{x}_A^L& y_A^L& 1\end{array}\right]---\left(7\right)$

Initial error th is set⁰=3, L layers angular coordinate after correction

4) search starting point of L-1 layers isRepeat above-mentioned 2) -3 to the angle point of L-1 layers) Tracking process, obtains the 0th layer i.e. corresponding angular coordinate of original image yardstick next frame by iterating；

(3) frame global homography matrix H before and after being calculated under original scale using RANSAC iterative algorithms⁰, its form (5)：

(4) each amount of exercise before and after separating in the global homography matrix of frame, obtains respective smoothed using Kalman filter Active movement amount

1) make H '=(H⁰)^-1, to H ' disengaging movement amounts；Translation T (x, y) of amount of exercise to be separated comprising relative former frame, 2D Anglec of rotation θ, 2D scales the distortion D of Z and homography matrix；

2) separation of actual motion amount is carried out using the method for four point transformation, the length and width of the original rectangle ABCD of definition are corresponding to be The length and width of frame of video, area are S, and A ' B ' C ' D ' are obtained to rectangular transform using H ', and area is S ', now amount of zoom：

$Z=\sqrt{\frac{S^{\′}}{S}}---\left(8\right)$

Calculate A ' B ' and C ' D ' angle thetas in the horizontal direction₁And θ₃, angle thetas of the A ' D ' and B ' C ' in vertical direction₄And θ₂, with Value method reduces the 2D angular errors brought by distortion：

$\mathrm{\θ}=\frac{1}{4}\·\underset{n=1}{\overset{4}{\Σ}}{\mathrm{\θ}}_n---\left(9\right)$

Conversion is compensated to quadrangle A ' B ' C ' D ' using the Z and θ for having obtained, is removed rotation and is scaled；Then calculate and become Center before and after changing is respectively C (x, y), C ' (x, y), and obtaining translational movement is：

T(t_x,t_y)=C (x, y)-C ' (x, y) (10)

3) translation T, the rotation θ and scaling Z to having obtained is filtered using Kalman filter and obtains corresponding active fortune Momentum I_T、I_θAnd I_Z, wherein for the state equation of two-dimensional vector T correspondence Kalman filter is：

$\left(\begin{array}{cccccc}1& 0& dt& 0& {\mathrm{dt}}^2/2& 0\\ 0& 1& 0& dt& 0& {\mathrm{dt}}^2/2\\ 0& 0& 1& 0& dt& 0\\ 0& 0& 0& 1& 0& dt\\ 0& 0& 0& 0& 1& 0\\ 0& 0& 0& 0& 0& 1\end{array}\right)---\left(11\right)$

Corresponding observing matrix is：

$\left(\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0& 0\end{array}\right)---\left(12\right)$

The state equation of the corresponding Kalman filter of scalar θ, Z：

$\left(\begin{array}{ccc}1& dt& {\mathrm{dt}}^2/2\\ 0& 1& dt\\ 0& 0& 1\end{array}\right)---\left(13\right)$

Observing matrix：

(1 0 0) (14)

Wherein dt is the sampling interval；

4) above-mentioned amount of exercise is 2D, the pattern distortion that homography conversion can be brought real visual angle change and produce, to distort into Row correction；Line translation is entered to A ' B ' C ' D ' and removes 2D plane motions using the translation T of existing 2D, rotation θ and scaling Z, obtain A " B " C " D ", A " B " C " D " only exist the distortion brought by 3D visual angle changes；Using the Kalman filter of translation T foundation to becoming Four apex coordinate A " B " C " D " after changing are filtered respectively and obtain A_iB_iC_iD_i；A_iB_iC_iD_iCamera 3D visual angles are represented to ABCD Change corresponding active variable quantity, i.e., required distortion correction amount I_D；

(5) the steady as transformation model of nth frame is calculated using the global homography for having obtained and each smooth active movement amount C_N, image is carried out steady as conversion

1) nth frame does homography conversion H of benchmark with N-1 frames_N, 2D plane motion compensation matrix motionH_N, 3D visual angle changes Corresponding distortion compensation converts distortH_N, Compensation Transformation of the nth frame with reference to N-1 frames is obtained using above-mentioned conversion：

T_N=motionH_N·distortH_N·H_N (15)

In order to the frame of video for obtaining continuous-stable is exported, using the first frame as global reference frame, therefore the steady picture of final nth frame becomes Mold changing type is：

C_N=T₁·...·T_N-1·T_N (16)

Using C_NEnter line translation to present frame and obtain stable two field picture.