CN103426182B - The electronic image stabilization method of view-based access control model attention mechanism - Google Patents

The electronic image stabilization method of view-based access control model attention mechanism Download PDF

Info

Publication number
CN103426182B
CN103426182B CN201310287353.XA CN201310287353A CN103426182B CN 103426182 B CN103426182 B CN 103426182B CN 201310287353 A CN201310287353 A CN 201310287353A CN 103426182 B CN103426182 B CN 103426182B
Authority
CN
China
Prior art keywords
sub
frame
block
image
pixel
Prior art date
Application number
CN201310287353.XA
Other languages
Chinese (zh)
Other versions
CN103426182A (en
Inventor
朱娟娟
郭宝龙
Original Assignee
西安电子科技大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 西安电子科技大学 filed Critical 西安电子科技大学
Priority to CN201310287353.XA priority Critical patent/CN103426182B/en
Publication of CN103426182A publication Critical patent/CN103426182A/en
Application granted granted Critical
Publication of CN103426182B publication Critical patent/CN103426182B/en

Links

Abstract

An electronic image stabilization method for view-based access control model attention mechanism, comprises the following steps: carry out foreground moving region detection to reference frame, mark foreground moving sub-block; Extract overall remarkable characteristic in reference frame; Feature point pair matching; Error hiding feature point pairs is rejected; Kinematic parameter obtains; Motion filtering; Rapid movement compensates; Rebuild undefined boundary information, obtain panoramic picture.The inventive method is by the right extraction of overall remarkable characteristic, coupling, checking and beginning parameter transform model, and auto adapted filtering smooth motion obtains compensating parameter, improve the vision degree of stability between frame of video and sharpness, eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker.

Description

The electronic image stabilization method of view-based access control model attention mechanism
Technical field
The invention belongs to digital image processing techniques field, particularly relate to a kind of electronic image stabilization method of view-based access control model attention mechanism.
Background technology
Because the vision system of people has eye storage characteristic, when picture pick-up device frequency of occurrences higher jitter, vision system easily catches the shake of picture in image, thus can feel the dimness of vision and be difficult to object observing.The high dither of video camera can cause the instability of video sequence, and the motion in video sequence can be divided into target travel and camera motion, and wherein, the former is local motion, and the latter is global motion.For the ease of observing, need, to the smoothing stable process of the motion in video sequence, to have occurred electronic image stabilizing thus.Electronic image stabilizing adopts the method for image procossing to estimate side-play amount between frame of video and the technology compensated.Current electronic image stabilizing has been widely used in the fields such as object detecting and tracking, walking robot, video compress and image mosaic.
Electronic steady image systematic research is mainly concentrated in overall motion estimation and the large gordian technique of image motion compensation two.The task of overall motion estimation determines interframe movement side-play amount, and the study hotspot of current overall motion estimation is the characteristic matching method that can process translation, rotation and zoom condition.In prior art, the normal feature matching method adopted mainly contains following a few class:
(1) based on the characteristic matching method of Hough straight line.As number of patent application be 201010528024.6, electronic image stabilization method disclosed in the Chinese invention patent application of the denomination of invention ship-borne camera system electronic image stabilization method that is feature based straight line, the method utilizes Hough transform to extract sea horizon linear feature in image, is then mated by the parameter of straight-line segment and position.The algorithm thinking of these class methods simply and easily realize, but being suitable for scene comparatively limits to, and can be difficult to even cannot extract straight line, and can extract too much short and small in a jumble straight line for complex scene, bring difficulty to matching line segments for simple scenario.
(2) based on the matching method of Sift point of interest.The method, based on Scale-space theory, is extracted the Sift unique point set of reference frame and present frame, is carried out registration, can process the translation of two width images, rotation, affine and view transformation, accurately obtain the kinematic parameter of image, have higher robustness.But it is excessive to there is extraction sift unique point quantity, and matching process is complicated, cannot carry out the deficiency of application in real time.
(3) based on the matching method of Harris angle point.As the patent No. be 201110178881.2, electronic image stabilization method disclosed in the Chinese invention patent of electronic image stabilization method that mates for feature based of denomination of invention, the method chooses the elementary cell of Corner Feature as estimation of the lesser amt in image, carries out signature tracking.Because its feature point extraction is carried out in entire image, therefore easily drop on local motion object, usually need to carry out signature verification and iteration, to reject local feature region, thus affect speed and the precision of overall motion estimation.
Image motion compensation method, as another gordian technique of electronic steady image system, is carry out filtering to original kinematic parameter sequence, thus obtains jittering component, compensates present frame using jittering component as compensating parameter.It retains scanning motion while focusing on removing shake.Filtering method conventional at present has movement dampens method, mean filter method, Kalman filter method etc.Wherein, set by rule of thumb in the attenuation coefficient experiment in movement dampens method, all video sequences cannot be applicable to; Mean filter adopts and is simply averaging computing, can introduce unnecessary low-frequency noise; Kalman filter then require process noise and observation noise priori known, and obey the Gaussian distribution of zero-mean, this is implacable in systems in practice.
The method for estimating adopted in aforementioned electronic image stabilization system and motion compensation process, the speed dependent of algorithm is in the extraction of characteristic information, coupling and interative computation, and there is local motion object in compound movement scene, because traditional feature point detection is carried out entire image, thus unique point cannot be avoided to be selected on foreground target, to cause the precise decreasing of overall motion estimation; Meanwhile, algorithm is difficult to the complicated randomized jitter and the scanning motion that process video camera simultaneously, easily occurs filtering divergence so that export false scene, and really scans scene and differs greatly, thus affects observing effect; In addition, to present frame node-by-node algorithm conversion parameter during compensation, consume operation time, the processing capability in real time of influential system, the loss of boundary information also can affect Visual Observations Observations.
Summary of the invention
For the deficiency of above-mentioned technology, the object of the present invention is to provide a kind of electronic image stabilization method of view-based access control model attention mechanism, image blurring and the unstable phenomenon that carrier movement produces can be eliminated, effectively stablize output video, improve the observation effect of video monitoring or tracker.
To achieve these goals, the present invention takes following technical solution:
An electronic image stabilization method for view-based access control model attention mechanism, comprises the following steps:
Step 1, foreground moving region detection is carried out to reference frame, mark foreground moving sub-block step;
Sub-step 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), x, y represent x-axis and the y-axis coordinate of pixel;
Sub-step 1b, the image in definition k-1 moment are reference frame f k-1(x, y), the image in k moment is present frame f k(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D k-1(x, y)=abs [f k-1(x, y)-B (x, y)],
Present frame difference image D k(x, y)=abs [f k(x, y)-B (x, y)];
Sub-step 1c, with reference to frame difference image D k-1(x, y) and present frame difference image D k(x, y) is divided into M × N number of sub-block of non-overlapping copies respectively, and described sub-block size is I × J pixel, calculates the mean absolute error in each sub-block:
Reference frame image sub-block mean absolute error
Current frame image sub-block mean absolute error
Wherein, i=1 ..., I, j=l ..., J, m=1 ..., M, n=l ..., N;
Sub-step 1d, computing reference frame sub-block difference mean value and present frame sub-block difference mean value, respectively as threshold value Th1 and Th2:
Th1=∑B k-1(m,n)/(M×N),
Th2=∑B k(m,n)/(M×N);
Sub-step 1e, tentatively judge whether each sub-block is moving sub-block by binaryzation, definition MO k-1(m, n) is reference frame moving sub-block, MO k(m, n) is current frame motion sub-block, and Rule of judgment is as follows:
MO k - 1 ( m , n ) = 1 , if B k - 1 ( m , n ) > Th 1 0 , else ,
MO k ( m , n ) = 1 , if B k ( m , n ) > Th 2 0 , else ;
Sub-step 1f, to reference frame moving sub-block MO k-1(m, n) carries out spatial domain similarity detection, the sub-block not belonging to sport foreground is deleted;
Sub-step 1g, to reference frame moving sub-block MO k-1(m, n) carries out the detection of time domain similarity, the sub-block not belonging to sport foreground is deleted;
After spatial domain, time domain similarity detect, the final moving sub-block retained is sport foreground region;
Overall remarkable characteristic step in step 2, extraction reference frame;
Sub-step 2a, with reference to frame f k-1(x, y) utilizes following formula compute gradient image:
X = f k - 1 ⊗ ( - 1,0,1 ) Y = f k - 1 ⊗ ( - 1,0,1 ) T ;
Wherein, represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, [] trepresent matrix transpose operation;
Sub-step 2b, structure autocorrelation matrix R:
R = X 2 ⊗ w XY ⊗ w XY ⊗ w Y 2 ⊗ w ;
Wherein, for Gaussian smoothing window function, for the standard deviation of window function;
Sub-step 2c, calculating Harris angle point response R h:
R H=λ 1×λ 2-0.05·(λ 12);
Wherein, λ 1and λ 2for two eigenwerts of autocorrelation matrix R;
Sub-step 2d, with reference to frame f k-1(x, y) is divided into M × N number of sub-block of non-overlapping copies, and sub-block size is I × J pixel, with reference to frame f k-1maximum Harris angle point response in each sub-block of (x, y) is as the characteristic response value R of this sub-block hMAX(m, n);
Sub-step 2e, by characteristic response value R hMAX(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x i, y i);
Sub-step 2f, utilize the result of sub-step 1g to reference frame unique point (x i, y i) judge, judge the reference frame moving sub-block MO of this Feature point correspondence k-1whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted;
Step 3, feature point pair matching step;
Sub-step 3a, at reference frame f k-1with reference frame unique point (x in (x, y) i, y i) centered by, build the Window being of a size of P × Q pixel;
Sub-step 3b, utilize full search strategy and least error and SAD criterion, at present frame f kcorresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is: SAD ( x , y ) = Σ p = 1 P Σ q = 1 Q | f k - 1 ( p , q ) - f k ( p + x , q + y ) | , p=1,…,P,q=1,…,Q,x,y=-T,…,T;
Step 4: error hiding feature point pairs rejects step;
According to Euclidean distance formula i-th pair of feature point pairs of computing reference frame and present frame in the horizontal direction with the distance of vertical direction translational movement, distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, obtain the C of correct coupling to feature point pairs;
Step 5: the acquisition step of kinematic parameter;
Sub-step 5a, foundation describe reference frame unique point (x i, y i) and present frame matching characteristic point between the kinematic parameter model of relation: x ^ y ^ = 1 - θ θ 1 x y + u v , Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter;
Sub-step 5b, the C that correctly mates is substituted into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation: B = x ^ 1 y ^ 1 x ^ 2 y ^ 2 · · · x ^ c y ^ c , A = x 1 y 1 1 x 2 y 2 1 · · · x c y c 1 , m = θ u v ;
Sub-step 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A ta) -1aB, thus obtain kinematic parameter;
Step 6: motion filtering step;
Sub-step 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)] t, measurement vector Z (k)=[u (k), v (k)] twherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Sub-step 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k-1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein, F = 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 State-transition matrix, H = 1 0 0 0 0 1 0 0 Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Sub-step 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (k|k-1)=FS (k-1|k-1);
The covariance matrix P (k|k-1) of S (k|k-1) is predicted: P (k|k-1)=FP (k-1) F t+ Φ (k-1), Φ are the variance matrix of process noise;
System state renewal equation is: S (k|k)=S (k|k-1)+K g(k) ε (K);
The filter error variance matrix of S (k|k) under renewal k moment state: P (k|k)=(Ψ-K g(k) H) P (k|k-1);
Wherein, K g(k)=P (k|k-1) H t(HP (k|k-1) H t+ Γ (k)) -1for Kalman gain, ε (k)=Z (k)-HS (k|k-1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order;
Step 7, rapid movement compensation process;
Sub-step 7a, by the difference u of forward and backward for filtering translation motion component jitter=u-u filter, v jitter=v-v filtercombining image anglec of rotation θ, as compensating parameter wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u jitterfor filtered pixel vertical translation amount, v jitterfor filtered pixel level translational movement;
Sub-step 7b, kinematic parameter model is utilized to calculate present frame f kthe rotation results of first pixel of (x, y) first trip [x, y]: x ′ y ′ = 1 - θ θ 1 x y + u jitter v jitter ;
Sub-step 7c, carry out plus and minus calculation according to image coordinate linear structure, calculate present frame f kthe pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame;
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
With reference frame f k-1(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the gray-scale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
f ( x ′ , y ′ ) = f k - 1 ( x ′ , y ′ ) ( x ′ , y ′ ) ∈ f k - 1 τ f k - 1 ( x ′ , y ′ ) + ξ f k ( x ′ , y ′ ) ( x ′ , y ′ ) ∈ ( f k - 1 ∩ f k ) f k ( x ′ , y ′ ) ( x ′ , y ′ ) ∈ f k
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1.
Further concrete scheme is: described step 6 also comprises the correction step of covariance matrix, continues to perform following steps after completing sub-step 6c:
Sub-step 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses: ε (k) tε (k)≤γ Trace [HP (k|k-1) H t+ Γ (k)];
Wherein, γ is adjustability coefficients and γ > 1;
Sub-step 6e, when in sub-step 6d formula set up time, illustrate that wave filter is in normal operating conditions, directly obtain the optimal estimation value of current state; When formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (k|k-1) in weighting coefficient C (k) sub-paragraphs 6c, the auto adapted filtering of kinematic parameter is completed after correction
Correction formula is as follows:
P(k|k-1)=C(k)·F·P(k-1)·F T+Φ(k),
C ( k ) = &epsiv; ( k ) T &CenterDot; &epsiv; ( k ) - Trace [ H &CenterDot; &Phi; ( k ) &CenterDot; H T + &Gamma; ( k ) ] Trace [ H &CenterDot; F &CenterDot; P ( k ) &CenterDot; F T &CenterDot; H T ] .
Further concrete scheme is: to reference frame moving sub-block MO in described sub-step 1f k-1the concrete steps that (m, n) carries out spatial domain similarity detection are: statistical-reference frame moving sub-block MO k-1the quantity of the moving sub-block that (m, n) surrounding 8 is adjacent, as moving sub-block quantity is less than 3, illustrate that this moving sub-block is the isolated sub-block differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this sub-block is similar with field block, all belong to foreground moving region:
MO k - 1 ( m , n ) = 1 , if &Sigma; l = 0 1 MO k - 1 ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 3 0 , else , L to be value be 0,1 variable;
Further concrete scheme is: to reference frame moving sub-block MO in described sub-step 1g k-1the concrete steps that (m, n) carries out the detection of time domain similarity are: at current frame motion sub-block MO kjudge whether moving sub-block in the moving sub-block that (m, n) surrounding 8 is adjacent, if having, illustrated that target is continuous in time, real sport foreground, otherwise, the flase drop occurred once in a while should be considered as, need to delete, the final moving sub-block retained is sport foreground region:
MO k - 1 ( m , n ) = 1 , if &Sigma; l = 0 1 MO k ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 1 0 , else , L to be value be 0,1 variable.
Further concrete scheme is: carrying out distance verification step to the feature point pairs of coupling in described step 4 is: the i-th pair of feature point pairs judging reference frame and present frame in the horizontal direction with the distance d of vertical direction translational movement iwhether meet the following conditions:
| d i-μ | > 3 σ, μ, σ are respectively d iaverage and standard deviation,
When meeting above condition, thinking that this feature point pairs is error hiding feature point pairs, being rejected.
From above technical scheme, the inventive method utilizes the body dynamics information of image, based on movement differential, the sport foreground in video sequence and background are split, the background area eliminating foreground target is carried out to extraction and the registration of overall remarkable characteristic, improve the precision of overall motion estimation; Meanwhile, with the vision attention of human eye for instructing, analog vision smoothness properties, builds the low frequency uniform motion pattern of video camera within the continuous imaging time, namely carries out filtering to the high fdrequency component in global motion vector sequence, obtains compensating parameter; During compensation, utilize the linear memory structure of image, only need first conversion parameter of computed image first trip, improve the real-time performance of system, and combining image splicing is rebuild to border, obtains steady and audible panoramic picture.
Compared with prior art, the present invention has following technique effect:
(1) Harris angle point operator is improved, ensure that unique point is the vision remarkable characteristic with unique information: the present invention is to image block, extract the unique point of the larger some of angle point response as remarkable characteristic, the fixed qty of unique point prevents in the too much situation of the unique point number of complex region extraction, and the remarkable information of unique point avoids the error hiding of simple textures repeat region;
(2) owing to adopting foreground moving zone marker and eliminating, ensure that the unique point of extraction is all in background: feature extraction of the present invention is not at entire image extracting directly, but after first moving region being carried out judging, mark and get rid of, only carry out feature detection in background area, thus ensure that unique point well represents the motion of video camera, i.e. global motion information;
(3) because the distance in conjunction with characteristic matching is verified, the precision of further raising overall motion estimation: when there is error hiding, adopts distance criterion, judges and reject this feature point pairs fast, make the unique point participating in beginning parameter transform model be correct coupling, improve the precision of estimation;
(4) self-adaptation Sage-Husa filtering algorithm is adopted, analog vision smoothness properties, on the one hand preferably smoothing motion vector to reduce video jitter, effectively follow on the other hand the scanning motion of having a mind to of camera system: lowlyer relative to traditional Kalman filter precision even to disperse, self-adaptation Sage-Husa filtering constantly revises predicted value by observation data, the statistical property of process noise and observation noise is substituted in standard Kalman filter simultaneously, estimate in real time and revise, strengthen the tracking power to mutation status, thus reach reduction model error, improve filtering accuracy,
(5) reconstruction in quick compensation and undefined district, ensure the smoothness of output video when Visual Observations Observations and integrality: the present invention utilizes the relative position between pixel to have the feature of rotational invariance, adopt the fast-compensation method of image rotation, improve the efficiency that coordinate calculates, ensure the real-time performance of system; Meanwhile, for avoiding image compensation to occur undefined dark border, bring harmful effect to visual effect, the present invention utilizes splicing and warm technology, rebuilds undefined district, exports panoramic image sequence.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, below by need in embodiment or description of the prior art use accompanying drawing do simple introduction, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the process flow diagram of the inventive method;
Fig. 2 is feature point pairs distance normal distribution;
Fig. 3 a is the reference frame image after marked moving sub-block;
Fig. 3 b is the current frame image after marked moving sub-block;
Fig. 3 c is the reference frame image of carrying out spatial domain similarity detection;
Fig. 3 d is the current frame image carrying out the detection of time domain similarity;
Fig. 4 a is the reference frame image being extracted all unique points;
Fig. 4 b is the reference frame image being extracted remarkable characteristic;
Fig. 4 c is the result figure after removing local feature region;
Fig. 4 d is the unique point result figure of registration in present frame;
Fig. 5 a is the image before rotating;
Fig. 5 b is postrotational image;
Fig. 6 a is the comparing result figure after horizontal-shift component adopts the inventive method preferred version auto adapted filtering and Kalman filter process;
Fig. 6 b is the comparing result figure after vertical shift component adopts the inventive method preferred version auto adapted filtering and Kalman filter process.
Embodiment
In order to allow above and other objects of the present invention, feature and advantage can be more obvious, the embodiment of the present invention cited below particularly, and coordinate appended diagram, be described below in detail.
Known based on the sensitivity analysis noted motion in human visual system, rocking of background can weaken the notice of vision system to foreground target, and the inconsistent motion of foreground target can the detection of jamming pattern global motion.In order to eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker, the basic ideas of the inventive method are:
First, in motion estimation module, consecutive frame is utilized on average to obtain background image, then adjacent reference frame and present frame are carried out difference with background image respectively, then detect based on the time domain of image block and spatial domain similarity and obtain foreground moving region, prospect and background are separated, in the background area of reference frame, then extract the Harris angle point larger unique point of response and carry out registration, solve the least square solution of determined linear equation, obtain globe motion parameter;
Secondly, at motion compensating module, preferably by improvement Sage-Husa filtering, the statistical property of real-time estimation and makeover process noise and observation noise, On-line Estimation is carried out to process noise and observation noise, obtains final compensating parameter, adopt the quick backoff algorithm of linearity and image mosaic to realize the steady picture of real time panoramic, export the real scene of the complete smoothness of vision, ensure that the real-time of system.
By the right extraction of overall remarkable characteristic, coupling, checking and beginning parameter transform model, and auto adapted filtering smooth motion obtains compensating parameter, improves the vision degree of stability between frame of video and sharpness.
It is more than core concept of the present invention, below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme of the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Set forth a lot of detail in the following description so that fully understand the present invention, but the present invention can also adopt other to be different from alternate manner described here to implement, those skilled in the art can when without prejudice to doing similar popularization when intension of the present invention, therefore the present invention is by the restriction of following public specific embodiment.
Reference Fig. 1, Fig. 1 are the process flow diagram of the inventive method, and the concrete steps of the inventive method are as follows:
Step 1, carry out foreground moving region detection with reference to frame, mark foreground moving sub-block step;
In this step after extraction background image, two continuous frames image (reference frame and present frame) and background image are carried out difference, multiple sub-block is divided into reference to frame difference image and present frame difference image, compare using the mean difference score value in sub-block as threshold value, to determine moving sub-block, utilize time-space domain similarity to carry out piecemeal to detect foreground moving region and to go forward side by side row labels, realize the Fast Segmentation of prospect in video sequence and background, remove moving target to the interference of vision attention;
The concrete steps of step 1 are as follows:
Sub-step 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), as 25 frames before video sequence (1 second) image being averaged and obtaining background image, x, y represent x-axis and the y-axis coordinate of pixel;
Sub-step 1b, the image in definition k-1 moment are reference frame f k-1(x, y), the image in k moment is present frame f k(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D k-1(x, y)=abs [f k-1(x, y)-B (x, y)],
Present frame difference image D k(x, y)=abs [f k(x, y)-B (x, y)];
Sub-step 1c, with reference to frame difference image D k-1(x, y) and present frame difference image D k(x, y) is divided into M × N number of sub-block of non-overlapping copies respectively, and described sub-block size is I × J pixel, as 16 × 16 pixels, calculates the mean absolute error in each sub-block:
Reference frame image sub-block mean absolute error
Current frame image sub-block mean absolute error
Wherein, i=1 ..., I, j=1 ..., J, m=1 ..., M, n=1 ..., N;
Sub-step 1d, computing reference frame sub-block difference mean value and present frame sub-block difference mean value, respectively as threshold value Th1 and Th2:
Th1=∑B k-1(m,n)/(M×N),
Th2=∑B k(m,n)/(M×N);
Sub-step 1e, tentatively to be judged by binaryzation whether each sub-block is moving sub-block (Movingobject is called for short MO), definition MO k-1(m, n) is reference frame moving sub-block, MO k(m, n) is current frame motion sub-block, and Rule of judgment is as follows:
MO k - 1 ( m , n ) = 1 , if B k - 1 ( m , n ) > Th 1 0 , else ,
MO k ( m , n ) = 1 , if B k ( m , n ) > Th 2 0 , else ;
Sub-step 1f, to reference frame moving sub-block MO k-1(m, n) carries out spatial domain similarity detection, i.e. statistical-reference frame moving sub-block MO k-1the quantity of the moving sub-block that (m, n) surrounding 8 is adjacent, as moving sub-block quantity is less than 3, illustrate that this moving sub-block is the isolated sub-block differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this sub-block is similar with field block, all belong to foreground moving region:
MO k - 1 ( m , n ) = 1 , if &Sigma; l = 0 1 MO k - 1 ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 3 0 , else , L to be value be 0,1 variable;
Sub-step 1g, to reference frame moving sub-block MO k-1(m, n) carries out the detection of time domain similarity, namely at the current frame motion sub-block MO of correspondence kjudged whether moving sub-block in the moving sub-block that (m, n) surrounding 8 is adjacent, if having, illustrating that target is continuous in time, is real sport foreground, otherwise, the flase drop occurred once in a while should be considered as, need to delete:
MO k - 1 ( m , n ) = 1 , if &Sigma; l = 0 1 MO k ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 1 0 , else , L to be value be 0,1 variable;
After spatial domain and time domain similarity are detected, the final moving sub-block retained is sport foreground region.
The overall remarkable characteristic step of step 2, extraction reference frame;
First Harris angle point response is calculated in this step, and at reference frame f k-1(x, y) find maximum Harris angle point response as characteristic response value in each sub-block, then sort, take out position corresponding to larger front 20% characteristic response value as unique point, be the remarkable characteristic with unique information observed visually, according to the mark result in foreground moving region in step 1, judge whether this unique point is positioned at foreground moving region, if be positioned at foreground moving region, delete, what remain is the overall remarkable characteristic observed visually;
The concrete steps of step 2 are as follows:
Sub-step 2a, with reference to frame f k-1(x, y) utilizes following formula compute gradient image:
X = f k - 1 &CircleTimes; ( - 1,0,1 ) Y = f k - 1 &CircleTimes; ( - 1,0,1 ) T ;
Wherein, represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, [] trepresent matrix transpose operation;
Sub-step 2b, structure autocorrelation matrix R:
R = X 2 &CircleTimes; w XY &CircleTimes; w XY &CircleTimes; w Y 2 &CircleTimes; w ;
Wherein, for Gaussian smoothing window function, for the standard deviation of window function;
Sub-step 2c, calculating Harris angle point response R h:
R H=λ 1×λ 2-0.05·(λ 12);
Wherein, λ 1and λ 2for two eigenwerts of autocorrelation matrix R;
Sub-step 2d, with reference to frame f k-1(x, y) is divided into M × N number of sub-block of non-overlapping copies, and sub-block size is I × J pixel, with reference to frame f k-1maximum Harris angle point response in each sub-block of (x, y) is as the characteristic response value R of this sub-block hMAX(m, n);
Sub-step 2e, by characteristic response value R hMAX(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x i, y i);
Sub-step 2f, utilize the result of sub-step 1g to reference frame unique point (x i, y i) judge, judge the reference frame moving sub-block MO of this Feature point correspondence k-1whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted.
Step 3, feature point pair matching step;
According to the surrounding pixel block message consistance of view-based access control model correct judgment matching double points, by setting up Window around each unique point of reference frame, and obtain matching window corresponding to present frame, the central point of matching window is matching characteristic point, reference frame unique point and present frame matching characteristic point constitutive characteristic point pair.
The concrete steps of step 3 are as follows:
Sub-step 3a, at reference frame f k-1with reference frame unique point (x in (x, y) i, y i) centered by, build the Window being of a size of P × Q pixel;
Sub-step 3b, utilize full search strategy and least error and SAD criterion, at present frame f kcorresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is: SAD ( x , y ) = &Sigma; p = 1 P &Sigma; q = 1 Q | f k - 1 ( p , q ) - f k ( p + x , q + y ) | , p=1,…,P,q=1,…,Q,x,y=-T,…,T;
Step 4: error hiding feature point pairs rejects step;
The distance of the feature point pairs translational movement in the horizontal direction and the vertical direction of statistics consecutive frame (reference frame and present frame), distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, finally obtain the C of correct coupling to feature point pairs;
According to Euclidean distance formula, wherein, d ifor reference frame and present frame i-th pair of feature point pairs in the horizontal direction with the distance of vertical direction translational movement, as | d i-μ | during > 3 σ, think that this feature point pairs is error hiding feature point pairs, rejected, μ, σ are respectively d iaverage and standard deviation.
As shown in Figure 2, experimentally add up, d iapproximate Normal Distribution, known according to " the 3 σ criterion " of normal distribution, the data on [μ-3 σ, μ+3 σ] interval account for 99.7%(Fig. 2 of total data), therefore think and work as | d i-μ | during > 3 σ, this feature point pairs is error hiding feature point pairs.
Step 5: the acquisition step of kinematic parameter;
Set up kinematic parameter model in this step, the feature point pairs correctly mated is substituted into kinematic parameter model, arranges and obtain kinematic parameter matrix equation, obtain kinematic parameter by the least square solution solving overdetermined linear system;
The concrete steps of step 5 are as follows:
Sub-step 5a, foundation describe reference frame unique point (x i, y i) and present frame matching characteristic point between the kinematic parameter model of relation: x ^ y ^ = 1 - &theta; &theta; 1 x y + u v , Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter (i.e. rotation, translation parameters);
Sub-step 5b, step 4 verified the C of the correct coupling obtained substitutes into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation: B = x ^ 1 y ^ 1 x ^ 2 y ^ 2 &CenterDot; &CenterDot; &CenterDot; x ^ c y ^ c , A = x 1 y 1 1 x 2 y 2 1 &CenterDot; &CenterDot; &CenterDot; x c y c 1 , m = &theta; u v ;
Sub-step 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A ta) -1aB, thus obtain kinematic parameter.
Step 6: motion filtering step;
Cumulative motion gain of parameter translation motion parametric line, answers smothing filtering to translation motion parametric line, simulates visual motion smoothing, strengthen the tracking power to mutation status;
The concrete steps of step 6 are as follows:
Sub-step 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)] t, measurement vector Z (k)=[u (k), v (k)] twherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Sub-step 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k-1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein, F = 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 State-transition matrix, H = 1 0 0 0 0 1 0 0 Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Sub-step 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (k|k-1)=FS (k-1|k-1);
The covariance matrix P (k|k-1) of S (k|k-1) is predicted: P (k|k-1)=FP (k-1) F t+ Φ (k-1), Φ are the variance matrix of process noise;
System state renewal equation is: S (k|k)=S (k|k-1)+K g(k) ε (K);
The filter error variance matrix of S (k|k) under renewal k moment state: P (k|k)=(Ψ-K g(k) H) P (k|k-1), wherein, Kg (k)=P (k|k-1) H t(HP (k|k-1) H t+ Г (k)) -1for Kalman gain, ε (k)=Z (k)-HS (k|k-1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order.
As further preferred version, the present invention improves aforementioned Sage-Husa filtering, increase the correction step of covariance matrix P (k|k-1), by estimating in real time and the statistical property of makeover process noise and observation noise, carry out self-adaptive smooth filtering to translation motion parametric line, concrete steps are as follows:
Sub-step 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses:
ε(k) T·ε(k)≤γ·Trace[H·P(k|k-1)·H T+Γ(k)];
Wherein, γ is adjustability coefficients and γ > 1;
Sub-step 6e, when in sub-step 6d formula set up time, wave filter is in normal operating conditions, directly obtains the optimal estimation value of current state; When current formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (k|k-1) in weighting coefficient C (k) sub-paragraphs 6c, complete the auto adapted filtering of kinematic parameter after correction;
Correction formula is as follows:
P(k|k-1)=C(k)·F·P(k-1)·F T+Φ(k),
C ( k ) = &epsiv; ( k ) T &CenterDot; &epsiv; ( k ) - Trace [ H &CenterDot; &Phi; ( k ) &CenterDot; H T + &Gamma; ( k ) ] Trace [ H &CenterDot; F &CenterDot; P ( k ) &CenterDot; F T &CenterDot; H T ] .
Step 7, rapid movement compensation process;
According to compensating parameter to present frame f k(x, y) converts, combining image linear memory structure, adopts the plus-minus of linear operation, realizes current frame image f kthe quick compensation of (x, y);
The concrete steps of step 7 are as follows:
Sub-step 7a, by the difference u of forward and backward for filtering translation motion component jitter=u-u filter, v jitter=v-v filtercombining image anglec of rotation θ, as compensating parameter wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u jitterfor filtered pixel vertical translation amount, v jitterfor filtered pixel level translational movement;
Sub-step 7b, kinematic parameter model is utilized to calculate present frame f kthe rotation results of first pixel of (x, y) first trip [x, y]: x &prime; y &prime; = 1 - &theta; &theta; 1 x y + u jitter v jitter ;
Sub-step 7c, carry out plus and minus calculation according to coordinate linear structure, calculate present frame f kthe pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame.
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
Compensating parameter is utilized according to step 7 to present frame f kafter (x, y) pixel carries out coordinate transform, with reference frame f k-1(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the gray-scale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
f ( x &prime; , y &prime; ) = f k - 1 ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; f k - 1 &tau; f k - 1 ( x &prime; , y &prime; ) + &xi; f k ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; ( f k - 1 &cap; f k ) f k ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; f k
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and present frame and reference frame overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1.Thus achieve in overlapping region by reference frame f k-1(x, y) is slowly smoothly transitted into present frame f k(x, y), makes the warm image effect obtained more natural, can not affect the observing effect of whole video, improve the integrality of vision.
Effect of the present invention is described by following test experience.
According to the perception of human visual to motion, human eye cannot observe the difference of single pixel, but judges motion with the lasting change in whole region.As shown in Figure 3 a and Figure 3 b shows, Fig. 3 a is the reference frame image after marked moving sub-block, and Fig. 3 b is the current frame image after marked moving sub-block, and in Fig. 3 a and Fig. 3 b, "+" symbol represents that this sub-block is moving sub-block.
Spatial domain similarity analysis is carried out to 8 neighborhood blocks of moving sub-block in reference frame image, if block existence is around no less than 3 moving sub-block, illustrates that this block is similar with field block, all belong to foreground moving region; Otherwise judge that this block is as the more isolated block of residual error in background, deleted, the candidate's prospect sub-block retained after deleting as shown in Figure 3 c.
Carry out time domain similarity analysis to moving sub-block in reference frame image again: if in continuous print present frame, there is moving sub-block in 8 similar domain location, is then judged as real motion; Otherwise should be considered as the flase drop occurred once in a while, need to delete, the final sub-block retained is the sport foreground region detected, as shown in Figure 3 d.
With reference to Fig. 4 a to Fig. 4 d, for overall remarkable characteristic extracts and the procedure chart of feature point pair matching.
Fig. 4 a is all unique points of reference frame image multi-subarea extracting, can find out, unique point is evenly distributed in entire image, Partial Feature point has been selected on moving target, a large amount of unique point is separately had to be positioned at the background area (as sky and ground) of texture repetition, this two category features point can cause error hiding, thus causes the reduction of overall motion estimation precision.
Fig. 4 b is that remarkable characteristic extracts result figure, remains front 20% unique point that characteristic response is larger, improves the uniqueness of characteristic point information.
Fig. 4 c is the result figure after removing error hiding unique point, the experimental result of composition graphs 3d, whether judging characteristic point position is positioned at foreground moving or the unreliable region of surrounding, if then directly deleted, and retain the overall remarkable characteristic of background area, thus be beneficial to the correct coupling of unique point.
Fig. 4 d is the unique point result figure of registration corresponding in current frame image, unique point success registration, and can process in real time.
With reference to Fig. 5 a and 5b, the principle that the linearity occurred in description above compensates fast is described as follows:
The linear memory structure of image ensure that the relative position between pixel has rotational invariance.With reference to Fig. 5 a and 5b, Fig. 5 a is image rotation preceding pixel point position, image definition in rectangular domain ABCD, any pixel E (x, y) and the first row pixel E1 (x with its same column 1, y 1), the first row pixel E2 (x that goes together with it 2, y 2) and the first row pixel A (x of the first row a, y a) in the four summit relations being geometrically rectangle.In arbitrary system, an apex coordinate E (x, y) of rectangle can be determined by other three apex coordinates:
x = x 1 + ( x 2 - x A ) y = y 1 + ( y 2 - y A ) - - - ( 1 )
Rotational transform is linear transformation, and the shape of rectangle does not change because of rotation.Fig. 5 b is pixel position after image rotation, after image rotation, four summits of rectangle be A ' (x ' a, y ' a), E1 ' (x 1', y 1'), E2 ' (x 2', y 2'), E ' (x ', y '), its coordinate relation still meets relational expression:
x &prime; = x 1 &prime; + ( x 2 &prime; - x A &prime; ) y &prime; = y 1 &prime; + ( x 2 &prime; - x A &prime; ) - - - ( 2 )
Therefore, the rotation results of image first trip and first pixel is utilized just can to calculate the rotation results of other all pixel.
Concrete steps are: do coordinate transform with the pixel of first row by similar variation model to the first row, variation model is x &prime; y &prime; = 1 - &theta; &theta; 1 x y + u v ; Plus and minus calculation can be carried out with above formula to the pixel of all the other ranks.This avoid and matrix multiplication operation is done to each pixel of entire image, thus effectively save operation time, improve the efficiency that coordinate calculates.
With reference to Fig. 6 a and Fig. 6 b, to be filtered into example to horizontal and vertical offset component, give the comparing result figure of the Kalman filter of its inventive method preferred version auto adapted filtering and prior art.Owing to horizontal direction existing nearly camera-scanning motion at the uniform velocity, therefore the empirical curve of Fig. 6 a is in stablizing propradation; Only there is randomized jitter in video camera, therefore the empirical curve of Fig. 6 b is in 0 positional fluctuation in the vertical direction.From this figure, select different process noise Q larger on compensation result impact in Kalman filter: when process noise Q value is larger, filter curve and primary curve approach, therefore without obvious filter effect; When process noise Q value is less, though can obtain level and smooth filter curve, after observing 52 frames by Fig. 6 b, original dither terminates, and its filter result departs from 0 vector, thus causes filtering divergence.And self-adaptive routing of the present invention effectively can avoid the phenomenon of this filtering divergence, can good level and smooth jittering component, the real scan motion of tracking camera effectively simultaneously.
According to above technical scheme, first the inventive method under the moving scene environment that there is moving target, proposes detection and the method for registering of overall remarkable characteristic, improves speed and the precision of overall motion estimation; Next solves in Camera location scanning process and occurs shake, improves Sage-Husa filtering to distinguish scanning and shake adaptively, while smooth motion, follows the tracks of real scan scene; The time-consuming computing that the matrix multiplication finally avoiding pointwise during image conversion brings, based on linearity storage organization, propose linear operation thus realize compensating fast, and the border that combining image merges losing when compensating rebuilds, eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker.
The above, it is only preferred embodiment of the present invention, not any pro forma restriction is done to the present invention, although the present invention discloses as above with preferred embodiment, but and be not used to limit the present invention, any those skilled in the art, do not departing within the scope of technical solution of the present invention, make a little change when the technology contents of above-mentioned announcement can be utilized or be modified to the Equivalent embodiments of equivalent variations, in every case be the content not departing from technical solution of the present invention, according to any simple modification that technical spirit of the present invention is done above embodiment, equivalent variations and modification, all still belong in the scope of technical solution of the present invention.

Claims (2)

1. an electronic image stabilization method for view-based access control model attention mechanism, is characterized in that, comprises the following steps:
Step 1, foreground moving region detection is carried out to reference frame, mark foreground moving sub-block step;
Sub-step 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), x, y represent x-axis and the y-axis coordinate of pixel;
Sub-step 1b, the image in definition k-1 moment are reference frame f k-1(x, y), the image in k moment is present frame f k(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D k-1(x, y)=abs [f k-1(x, y)-B (x, y)],
Present frame difference image D k(x, y)=abs [f k(x, y)-B (x, y)];
Sub-step 1c, with reference to frame difference image D k-1(x, y) and present frame difference image D k(x, y) is divided into M × N number of sub-block of non-overlapping copies respectively, and described sub-block size is I × J pixel, calculates the mean absolute error in each sub-block:
Reference frame image sub-block mean absolute error
Current frame image sub-block mean absolute error
Wherein, i=1 ..., I, j=1 ..., J, m=1 ..., M, n=1 ..., N;
Sub-step 1d, computing reference frame sub-block difference mean value and present frame sub-block difference mean value, respectively as threshold value Th1 and Th2:
Th1=∑B k-1(m,n)/(M×N),
Th2=∑B k(m,n)/(M×N);
Sub-step 1e, tentatively judge whether each sub-block is moving sub-block by binaryzation, definition MO k-1(m, n) is reference frame moving sub-block, MO k(m, n) is current frame motion sub-block, and Rule of judgment is as follows:
MO k - 1 ( m , n ) = { 1 , i f B k - 1 ( m , n ) > Th 1 0 , e l s e ,
MO k ( m , n ) = { 1 , i f B k ( m , n ) > Th 2 0 , e l s e ;
Sub-step 1f, to reference frame moving sub-block MO k-1(m, n) carries out spatial domain similarity detection, the sub-block not belonging to sport foreground is deleted;
Sub-step 1g, to reference frame moving sub-block MO k-1(m, n) carries out the detection of time domain similarity, the sub-block not belonging to sport foreground is deleted;
After spatial domain, time domain similarity detect, the final moving sub-block retained is sport foreground region;
Overall remarkable characteristic step in step 2, extraction reference frame;
Sub-step 2a, with reference to frame f k-1(x, y) utilizes following formula compute gradient image:
{ X = f k - 1 &CircleTimes; ( - 1 , 0 , 1 ) Y = f k - 1 &CircleTimes; ( - 1 , 0 , 1 ) T ;
Wherein, represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, [] trepresent matrix transpose operation;
Sub-step 2b, structure autocorrelation matrix R:
R = X 2 &CircleTimes; w X Y &CircleTimes; w X Y &CircleTimes; w Y 2 &CircleTimes; w ;
Wherein, for Gaussian smoothing window function, for the standard deviation of window function;
Sub-step 2c, calculating Harris angle point response R h:
R H=λ 1×λ 2-0.05·(λ 12);
Wherein, λ 1and λ 2for two eigenwerts of autocorrelation matrix R;
Sub-step 2d, with reference to frame f k-1(x, y) is divided into M × N number of sub-block of non-overlapping copies, and sub-block size is I × J pixel, with reference to frame f k-1maximum Harris angle point response in each sub-block of (x, y) is as the characteristic response value R of this sub-block hMAX(m, n);
Sub-step 2e, by characteristic response value R hMAX(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x i, y i);
Sub-step 2f, utilize the result of sub-step 1g to reference frame unique point (x i, y i) judge, judge the reference frame moving sub-block MO of this Feature point correspondence k-1whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted;
Step 3, feature point pair matching step;
Sub-step 3a, at reference frame f k-1with reference frame unique point (x in (x, y) i, y i) centered by, build the Window being of a size of P × Q pixel;
Sub-step 3b, utilize full search strategy and least error and SAD criterion, at present frame f kcorresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is: S A D ( x , y ) = &Sigma; p = 1 P &Sigma; q = 1 Q | f k - 1 ( p , q ) - f k ( p + x , q + y ) | , p=1,…,P,q=1,…,Q,x,y=-T,…,T;
Step 4: error hiding feature point pairs rejects step;
According to Euclidean distance formula i-th pair of feature point pairs of computing reference frame and present frame in the horizontal direction with the distance of vertical direction translational movement, distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, obtain the C of correct coupling to feature point pairs;
Step 5: the acquisition step of kinematic parameter;
Sub-step 5a, foundation describe reference frame unique point (x i, y i) and present frame matching characteristic point between the kinematic parameter model of relation: x ^ y ^ = 1 - &theta; &theta; 1 x y + u v , Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter;
Sub-step 5b, the C that correctly mates is substituted into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation: B = x ^ 1 y ^ 1 x ^ 2 y ^ 2 . . . x ^ c y ^ c , A = x 1 y 1 1 x 2 y 2 1 . . . x c y c 1 , m = &theta; u v ;
Sub-step 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A ta) -1aB, thus obtain kinematic parameter;
Step 6: motion filtering step;
Sub-step 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)] t, measurement vector Z (k)=[u (k), v (k)] twherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Sub-step 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k-1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein, F = 1 0 1 0 0 1 0 1 0 0 1 0 0 0 0 1 State-transition matrix, H = 1 0 0 0 0 1 0 0 Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Sub-step 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (k|k-1)=FS (k-1|k-1);
The covariance matrix P (k|k-1) of S (k|k-1) is predicted: P (k|k-1)=FP (k-1) F t+ Φ (k-1), Φ are the variance matrix of process noise;
System state renewal equation is: S (k|k)=S (k|k-1)+K g(k) ε (K);
The filter error variance matrix of S (k|k) under renewal k moment state: P (k|k)=(Ψ-K g(k) H) P (k|k-1);
Wherein, K g(k)=P (k|k-1) H t(HP (k|k-1) H t+ Γ (k)) -1for Kalman gain, ε (k)=Z (k)-HS (k|k-1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order;
Step 7, rapid movement compensation process;
Sub-step 7a, by the difference u of forward and backward for filtering translation motion component jitter=u-u filter, v jitter=v-v filtercombining image anglec of rotation θ, as compensating parameter wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u jitterfor filtered pixel vertical translation amount, v jitterfor filtered pixel level translational movement;
Sub-step 7b, kinematic parameter model is utilized to calculate present frame f kthe rotation results of first pixel of (x, y) first trip [x, y]: x &prime; y &prime; = 1 - &theta; &theta; 1 x y + u j i t t e r v j i t t e r ;
Sub-step 7c, carry out plus and minus calculation according to image coordinate linear structure, calculate present frame f kthe pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame;
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
With reference frame f k-1(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the gray-scale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
f ( x &prime; , y &prime; ) = f k - 1 ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; f k - 1 &tau;f k - 1 ( x &prime; , y &prime; ) + &xi;f k ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; ( f k - 1 &cap; f k ) f k ( x &prime; , y &prime; ) ( x &prime; , y &prime; ) &Element; f k
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1;
To reference frame moving sub-block MO in described sub-step 1f k-1the concrete steps that (m, n) carries out spatial domain similarity detection are: statistical-reference frame moving sub-block MO k-1the quantity of the moving sub-block that (m, n) surrounding 8 is adjacent, as moving sub-block quantity is less than 3, illustrate that this moving sub-block is the isolated sub-block differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this sub-block is similar with field block, all belong to foreground moving region:
MO k - 1 ( m , n ) = { 1 , i f &Sigma; l = 0 1 MO k - 1 ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 3 0 , e l s e , L to be value be 0,1 variable;
To reference frame moving sub-block MO in described sub-step 1g k-1the concrete steps that (m, n) carries out the detection of time domain similarity are: at current frame motion sub-block MO kjudge whether moving sub-block in the moving sub-block that (m, n) surrounding 8 is adjacent, if having, illustrated that target is continuous in time, real sport foreground, otherwise, be considered as the flase drop occurred once in a while, need to delete, the final moving sub-block retained is sport foreground region:
MO k - 1 ( m , n ) = { 1 , i f &Sigma; l = 0 1 MO k ( m &PlusMinus; l , n &PlusMinus; l ) &GreaterEqual; 1 0 , e l s e , L to be value be 0,1 variable;
Described step 6 also comprises the correction step of covariance matrix, continues to perform following steps after completing sub-step 6c:
Sub-step 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses: ε (k) tε (k)≤γ Trace [HP (k|k-1) H t+ Γ (k)];
Wherein, γ is adjustability coefficients and γ > 1;
Sub-step 6e, when in sub-step 6d formula set up time, illustrate that wave filter is in normal operating conditions, directly obtain the optimal estimation value of current state; When formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (k|k-1) in weighting coefficient C (k) sub-paragraphs 6c, the auto adapted filtering of kinematic parameter is completed after correction
Correction formula is as follows:
P(k|k-1)=C(k)·F·P(k-1)·F T+Φ(k),
C ( k ) = &epsiv; ( k ) T &CenterDot; &epsiv; ( k ) - T r a c e &lsqb; H &CenterDot; &Phi; ( k ) &CenterDot; H T + &Gamma; ( k ) &rsqb; T r a c e &lsqb; H &CenterDot; F &CenterDot; P ( k ) &CenterDot; F T &CenterDot; H T &rsqb; .
2. the electronic image stabilization method of view-based access control model attention mechanism according to claim 1, is characterized in that: carrying out distance verification step to the feature point pairs of coupling in described step 4 is: the i-th pair of feature point pairs judging reference frame and present frame in the horizontal direction with the distance d of vertical direction translational movement iwhether meet the following conditions:
| d i-μ | > 3 σ, μ, σ are respectively d iaverage and standard deviation,
When meeting above condition, think that this feature point pairs is error hiding feature point pairs, rejected.
CN201310287353.XA 2013-07-09 2013-07-09 The electronic image stabilization method of view-based access control model attention mechanism CN103426182B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201310287353.XA CN103426182B (en) 2013-07-09 2013-07-09 The electronic image stabilization method of view-based access control model attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201310287353.XA CN103426182B (en) 2013-07-09 2013-07-09 The electronic image stabilization method of view-based access control model attention mechanism

Publications (2)

Publication Number Publication Date
CN103426182A CN103426182A (en) 2013-12-04
CN103426182B true CN103426182B (en) 2016-01-06

Family

ID=49650872

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201310287353.XA CN103426182B (en) 2013-07-09 2013-07-09 The electronic image stabilization method of view-based access control model attention mechanism

Country Status (1)

Country Link
CN (1) CN103426182B (en)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103905826A (en) * 2014-04-10 2014-07-02 北京工业大学 Self-adaptation global motion estimation method
CN104853064B (en) * 2015-04-10 2018-04-17 海视英科光电(苏州)有限公司 Electronic image stabilization method based on thermal infrared imager
CN105263026B (en) * 2015-10-12 2018-04-17 西安电子科技大学 Global vector acquisition methods based on probability statistics and image gradient information
CN105758867B (en) * 2016-03-11 2019-11-05 苏州睿仟医疗科技有限公司 A kind of microscopic defect check method of high speed
CN106375659B (en) * 2016-06-06 2019-06-11 中国矿业大学 Electronic image stabilization method based on multiresolution Gray Projection
CN106128007B (en) * 2016-08-26 2018-04-20 宁波圣达精工智能科技有限公司 Intelligent compact shelf controls guard system
CN106210447B (en) * 2016-09-09 2019-05-14 长春大学 Based on the matched video image stabilization method of background characteristics point
CN106357958B (en) * 2016-10-10 2019-04-16 山东大学 A kind of swift electron digital image stabilization method based on Region Matching
CN108881668A (en) * 2017-06-02 2018-11-23 北京旷视科技有限公司 Video increases steady method, apparatus, system and computer-readable medium
CN107197121B (en) * 2017-06-14 2019-07-26 长春欧意光电技术有限公司 A kind of electronic image stabilization method based on on-board equipment
CN107423409B (en) * 2017-07-28 2020-03-31 维沃移动通信有限公司 Image processing method, image processing device and electronic equipment
CN107578428A (en) * 2017-08-31 2018-01-12 成都观界创宇科技有限公司 Method for tracking target and panorama camera applied to panoramic picture
CN108174087B (en) * 2017-12-26 2019-07-02 北京理工大学 A kind of steady reference frame update method and system as in of Gray Projection
CN108492328B (en) * 2018-03-23 2021-02-26 云南大学 Video inter-frame target matching method and device and implementation device
CN108765532A (en) * 2018-05-04 2018-11-06 北京物灵智能科技有限公司 Children paint this method for establishing model, reading machine people and storage device
CN109561253B (en) * 2018-12-18 2020-07-10 影石创新科技股份有限公司 Method and device for preventing panoramic video from shaking, portable terminal and storage medium
CN109816006B (en) * 2019-01-18 2020-11-13 深圳大学 Sea-sky-line detection method and device and computer-readable storage medium
CN109922258B (en) * 2019-02-27 2020-11-03 杭州飞步科技有限公司 Electronic image stabilizing method and device for vehicle-mounted camera and readable storage medium
CN110046555A (en) * 2019-03-26 2019-07-23 合肥工业大学 Endoscopic system video image stabilization method and device
CN110120023A (en) * 2019-05-14 2019-08-13 浙江工大盈码科技发展有限公司 A kind of image feedback antidote

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101316368A (en) * 2008-07-18 2008-12-03 西安电子科技大学 Full view stabilizing method based on global characteristic point iteration
CN101729763A (en) * 2009-12-15 2010-06-09 中国科学院长春光学精密机械与物理研究所 Electronic image stabilizing method for digital videos
CN103024247A (en) * 2011-09-28 2013-04-03 中国航天科工集团第二研究院二〇七所 Electronic image stabilization method based on improved block matching

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8817190B2 (en) * 2007-11-28 2014-08-26 Canon Kabushiki Kaisha Image processing apparatus, image processing method, and computer program

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101316368A (en) * 2008-07-18 2008-12-03 西安电子科技大学 Full view stabilizing method based on global characteristic point iteration
CN101729763A (en) * 2009-12-15 2010-06-09 中国科学院长春光学精密机械与物理研究所 Electronic image stabilizing method for digital videos
CN103024247A (en) * 2011-09-28 2013-04-03 中国航天科工集团第二研究院二〇七所 Electronic image stabilization method based on improved block matching

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
电子稳像理论及其应用;朱娟娟;《中国博士学位论文全文数据库(Information Science and Technology)》;20091007;89-130 *

Also Published As

Publication number Publication date
CN103426182A (en) 2013-12-04

Similar Documents

Publication Publication Date Title
Jaritz et al. Sparse and dense data with cnns: Depth completion and semantic segmentation
Sun et al. Correlation tracking via joint discrimination and reliability learning
Wang et al. Detect globally, refine locally: A novel approach to saliency detection
Seki et al. Sgm-nets: Semi-global matching with neural networks
Li et al. Flow guided recurrent neural encoder for video salient object detection
Kueng et al. Low-latency visual odometry using event-based feature tracks
Zhou et al. Efficient road detection and tracking for unmanned aerial vehicle
Pandey et al. Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information
Qiu et al. Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image
Xiao et al. Fast image dehazing using guided joint bilateral filter
Choi et al. RGB-D edge detection and edge-based registration
Liu et al. Guided inpainting and filtering for kinect depth maps
CN105245841B (en) A kind of panoramic video monitoring system based on CUDA
US9105093B2 (en) Method and apparatus for bi-layer segmentation
Gallup et al. Piecewise planar and non-planar stereo for urban scene reconstruction
Park et al. Look wider to match image patches with convolutional neural networks
CN103325112B (en) Moving target method for quick in dynamic scene
Zhang et al. Semantic segmentation of urban scenes using dense depth maps
Wang et al. A region based stereo matching algorithm using cooperative optimization
Lai et al. Single-image dehazing via optimal transmission map under scene priors
Schneider et al. RegNet: Multimodal sensor registration using deep neural networks
Yang et al. Color-guided depth recovery from RGB-D data using an adaptive autoregressive model
Saxena et al. Learning 3-d scene structure from a single still image
Kong et al. General road detection from a single image
CN108573276B (en) Change detection method based on high-resolution remote sensing image

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
C14 Grant of patent or utility model
GR01 Patent grant
TR01 Transfer of patent right
TR01 Transfer of patent right

Effective date of registration: 20210330

Address after: Floor 5, block D, Boyuan science and Technology Plaza, No.99, Yanxiang Road, Yanta District, Xi'an City, Shaanxi Province, 710000

Patentee after: Xijiao Sichuang Intelligent Technology Research Institute (Xi'an) Co.,Ltd.

Address before: 710071 No. 2 Taibai South Road, Shaanxi, Xi'an

Patentee before: XIDIAN University