CN103426182B  The electronic image stabilization method of viewbased access control model attention mechanism  Google Patents
The electronic image stabilization method of viewbased access control model attention mechanism Download PDFInfo
 Publication number
 CN103426182B CN103426182B CN201310287353.XA CN201310287353A CN103426182B CN 103426182 B CN103426182 B CN 103426182B CN 201310287353 A CN201310287353 A CN 201310287353A CN 103426182 B CN103426182 B CN 103426182B
 Authority
 CN
 China
 Prior art keywords
 sub
 frame
 block
 image
 pixel
 Prior art date
Links
 238000011105 stabilization Methods 0.000 title claims abstract description 13
 238000001914 filtration Methods 0.000 claims abstract description 38
 238000001514 detection method Methods 0.000 claims abstract description 20
 230000001808 coupling Effects 0.000 claims abstract description 14
 238000010168 coupling process Methods 0.000 claims abstract description 14
 238000005859 coupling reaction Methods 0.000 claims abstract description 14
 238000000605 extraction Methods 0.000 claims abstract description 13
 239000011159 matrix material Substances 0.000 claims description 54
 238000000034 method Methods 0.000 claims description 34
 230000004044 response Effects 0.000 claims description 22
 230000000875 corresponding Effects 0.000 claims description 15
 238000005516 engineering process Methods 0.000 claims description 8
 230000000717 retained Effects 0.000 claims description 7
 238000009499 grossing Methods 0.000 claims description 6
 238000004364 calculation method Methods 0.000 claims description 4
 238000005259 measurement Methods 0.000 claims description 3
 102000012192 Cystatin C Human genes 0.000 claims description 2
 108010061642 Cystatin C Proteins 0.000 claims description 2
 230000000694 effects Effects 0.000 abstract description 14
 239000000284 extract Substances 0.000 abstract description 7
 238000004422 calculation algorithm Methods 0.000 description 6
 230000000007 visual effect Effects 0.000 description 6
 238000006243 chemical reaction Methods 0.000 description 3
 238000010586 diagram Methods 0.000 description 3
 230000000087 stabilizing Effects 0.000 description 3
 238000004458 analytical method Methods 0.000 description 2
 230000004048 modification Effects 0.000 description 2
 238000006011 modification reaction Methods 0.000 description 2
 230000035772 mutation Effects 0.000 description 2
 230000001131 transforming Effects 0.000 description 2
 108020001143 ABCD Proteins 0.000 description 1
 239000000969 carrier Substances 0.000 description 1
 150000001875 compounds Chemical class 0.000 description 1
 230000001186 cumulative Effects 0.000 description 1
 230000003247 decreasing Effects 0.000 description 1
 230000001419 dependent Effects 0.000 description 1
 238000003384 imaging method Methods 0.000 description 1
 230000002045 lasting Effects 0.000 description 1
 239000003550 marker Substances 0.000 description 1
 239000000203 mixture Substances 0.000 description 1
 230000011218 segmentation Effects 0.000 description 1
 238000010206 sensitivity analysis Methods 0.000 description 1
Abstract
An electronic image stabilization method for viewbased access control model attention mechanism, comprises the following steps: carry out foreground moving region detection to reference frame, mark foreground moving subblock; Extract overall remarkable characteristic in reference frame; Feature point pair matching; Error hiding feature point pairs is rejected; Kinematic parameter obtains; Motion filtering; Rapid movement compensates; Rebuild undefined boundary information, obtain panoramic picture.The inventive method is by the right extraction of overall remarkable characteristic, coupling, checking and beginning parameter transform model, and auto adapted filtering smooth motion obtains compensating parameter, improve the vision degree of stability between frame of video and sharpness, eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker.
Description
Technical field
The invention belongs to digital image processing techniques field, particularly relate to a kind of electronic image stabilization method of viewbased access control model attention mechanism.
Background technology
Because the vision system of people has eye storage characteristic, when picture pickup device frequency of occurrences higher jitter, vision system easily catches the shake of picture in image, thus can feel the dimness of vision and be difficult to object observing.The high dither of video camera can cause the instability of video sequence, and the motion in video sequence can be divided into target travel and camera motion, and wherein, the former is local motion, and the latter is global motion.For the ease of observing, need, to the smoothing stable process of the motion in video sequence, to have occurred electronic image stabilizing thus.Electronic image stabilizing adopts the method for image procossing to estimate sideplay amount between frame of video and the technology compensated.Current electronic image stabilizing has been widely used in the fields such as object detecting and tracking, walking robot, video compress and image mosaic.
Electronic steady image systematic research is mainly concentrated in overall motion estimation and the large gordian technique of image motion compensation two.The task of overall motion estimation determines interframe movement sideplay amount, and the study hotspot of current overall motion estimation is the characteristic matching method that can process translation, rotation and zoom condition.In prior art, the normal feature matching method adopted mainly contains following a few class:
(1) based on the characteristic matching method of Hough straight line.As number of patent application be 201010528024.6, electronic image stabilization method disclosed in the Chinese invention patent application of the denomination of invention shipborne camera system electronic image stabilization method that is feature based straight line, the method utilizes Hough transform to extract sea horizon linear feature in image, is then mated by the parameter of straightline segment and position.The algorithm thinking of these class methods simply and easily realize, but being suitable for scene comparatively limits to, and can be difficult to even cannot extract straight line, and can extract too much short and small in a jumble straight line for complex scene, bring difficulty to matching line segments for simple scenario.
(2) based on the matching method of Sift point of interest.The method, based on Scalespace theory, is extracted the Sift unique point set of reference frame and present frame, is carried out registration, can process the translation of two width images, rotation, affine and view transformation, accurately obtain the kinematic parameter of image, have higher robustness.But it is excessive to there is extraction sift unique point quantity, and matching process is complicated, cannot carry out the deficiency of application in real time.
(3) based on the matching method of Harris angle point.As the patent No. be 201110178881.2, electronic image stabilization method disclosed in the Chinese invention patent of electronic image stabilization method that mates for feature based of denomination of invention, the method chooses the elementary cell of Corner Feature as estimation of the lesser amt in image, carries out signature tracking.Because its feature point extraction is carried out in entire image, therefore easily drop on local motion object, usually need to carry out signature verification and iteration, to reject local feature region, thus affect speed and the precision of overall motion estimation.
Image motion compensation method, as another gordian technique of electronic steady image system, is carry out filtering to original kinematic parameter sequence, thus obtains jittering component, compensates present frame using jittering component as compensating parameter.It retains scanning motion while focusing on removing shake.Filtering method conventional at present has movement dampens method, mean filter method, Kalman filter method etc.Wherein, set by rule of thumb in the attenuation coefficient experiment in movement dampens method, all video sequences cannot be applicable to; Mean filter adopts and is simply averaging computing, can introduce unnecessary lowfrequency noise; Kalman filter then require process noise and observation noise priori known, and obey the Gaussian distribution of zeromean, this is implacable in systems in practice.
The method for estimating adopted in aforementioned electronic image stabilization system and motion compensation process, the speed dependent of algorithm is in the extraction of characteristic information, coupling and interative computation, and there is local motion object in compound movement scene, because traditional feature point detection is carried out entire image, thus unique point cannot be avoided to be selected on foreground target, to cause the precise decreasing of overall motion estimation; Meanwhile, algorithm is difficult to the complicated randomized jitter and the scanning motion that process video camera simultaneously, easily occurs filtering divergence so that export false scene, and really scans scene and differs greatly, thus affects observing effect; In addition, to present frame nodebynode algorithm conversion parameter during compensation, consume operation time, the processing capability in real time of influential system, the loss of boundary information also can affect Visual Observations Observations.
Summary of the invention
For the deficiency of abovementioned technology, the object of the present invention is to provide a kind of electronic image stabilization method of viewbased access control model attention mechanism, image blurring and the unstable phenomenon that carrier movement produces can be eliminated, effectively stablize output video, improve the observation effect of video monitoring or tracker.
To achieve these goals, the present invention takes following technical solution:
An electronic image stabilization method for viewbased access control model attention mechanism, comprises the following steps:
Step 1, foreground moving region detection is carried out to reference frame, mark foreground moving subblock step;
Substep 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), x, y represent xaxis and the yaxis coordinate of pixel;
Substep 1b, the image in definition k1 moment are reference frame f
_{k1}(x, y), the image in k moment is present frame f
_{k}(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D
_{k1}(x, y)=abs [f
_{k1}(x, y)B (x, y)],
Present frame difference image D
_{k}(x, y)=abs [f
_{k}(x, y)B (x, y)];
Substep 1c, with reference to frame difference image D
_{k1}(x, y) and present frame difference image D
_{k}(x, y) is divided into M × N number of subblock of nonoverlapping copies respectively, and described subblock size is I × J pixel, calculates the mean absolute error in each subblock:
Reference frame image subblock mean absolute error
Current frame image subblock mean absolute error
Wherein, i=1 ..., I, j=l ..., J, m=1 ..., M, n=l ..., N;
Substep 1d, computing reference frame subblock difference mean value and present frame subblock difference mean value, respectively as threshold value Th1 and Th2:
Th1＝∑B
_{k1}(m,n)/(M×N)，
Th2＝∑B
_{k}(m,n)/(M×N)；
Substep 1e, tentatively judge whether each subblock is moving subblock by binaryzation, definition MO
_{k1}(m, n) is reference frame moving subblock, MO
_{k}(m, n) is current frame motion subblock, and Rule of judgment is as follows:
Substep 1f, to reference frame moving subblock MO
_{k1}(m, n) carries out spatial domain similarity detection, the subblock not belonging to sport foreground is deleted;
Substep 1g, to reference frame moving subblock MO
_{k1}(m, n) carries out the detection of time domain similarity, the subblock not belonging to sport foreground is deleted;
After spatial domain, time domain similarity detect, the final moving subblock retained is sport foreground region;
Overall remarkable characteristic step in step 2, extraction reference frame;
Substep 2a, with reference to frame f
_{k1}(x, y) utilizes following formula compute gradient image:
Wherein,
represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, []
^{t}represent matrix transpose operation;
Substep 2b, structure autocorrelation matrix R:
Wherein,
for Gaussian smoothing window function,
for the standard deviation of window function;
Substep 2c, calculating Harris angle point response R
_{h}:
R
_{H}＝λ
_{1}×λ
_{2}0.05·(λ
_{1}+λ
_{2})；
Wherein, λ
_{1}and λ
_{2}for two eigenwerts of autocorrelation matrix R;
Substep 2d, with reference to frame f
_{k1}(x, y) is divided into M × N number of subblock of nonoverlapping copies, and subblock size is I × J pixel, with reference to frame f
_{k1}maximum Harris angle point response in each subblock of (x, y) is as the characteristic response value R of this subblock
_{hMAX}(m, n);
Substep 2e, by characteristic response value R
_{hMAX}(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x
_{i}, y
_{i});
Substep 2f, utilize the result of substep 1g to reference frame unique point (x
_{i}, y
_{i}) judge, judge the reference frame moving subblock MO of this Feature point correspondence
_{k1}whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted;
Step 3, feature point pair matching step;
Substep 3a, at reference frame f
_{k1}with reference frame unique point (x in (x, y)
_{i}, y
_{i}) centered by, build the Window being of a size of P × Q pixel;
Substep 3b, utilize full search strategy and least error and SAD criterion, at present frame f
_{k}corresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point
wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is:
$\mathrm{SAD}(x,y)=\underset{p=1}{\overset{P}{\mathrm{\Σ}}}\underset{q=1}{\overset{Q}{\mathrm{\Σ}}}{f}_{k1}(p,q){f}_{k}(p+x,q+y),$ p＝1,…，P，q＝1,…，Q，x,y＝T,…，T；
Step 4: error hiding feature point pairs rejects step;
According to Euclidean distance formula
ith pair of feature point pairs of computing reference frame and present frame in the horizontal direction with the distance of vertical direction translational movement, distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, obtain the C of correct coupling to feature point pairs;
Step 5: the acquisition step of kinematic parameter;
Substep 5a, foundation describe reference frame unique point (x
_{i}, y
_{i}) and present frame matching characteristic point
between the kinematic parameter model of relation:
$\left[\begin{array}{c}\hat{x}\\ \hat{y}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}u\\ v\end{array}\right],$ Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter;
Substep 5b, the C that correctly mates is substituted into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation:
$B=\left[\begin{array}{ccc}{\hat{x}}_{1}& & {\hat{y}}_{1}\\ {\hat{x}}_{2}& & {\hat{y}}_{2}\\ & \·& \\ & \·& \\ & \·& \\ {\hat{x}}_{c}& & {\hat{y}}_{c}\end{array}\right],A=\left[\begin{array}{ccc}{x}_{1}& {y}_{1}& 1\\ {x}_{2}& {y}_{2}& 1\\ & \·& \\ & \·& \\ & \·& \\ {x}_{c}& {y}_{c}& 1\end{array}\right],m=\left[\begin{array}{c}\mathrm{\θ}\\ u\\ v\end{array}\right];$
Substep 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A
^{t}a)
^{1}aB, thus obtain kinematic parameter;
Step 6: motion filtering step;
Substep 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)]
^{t}, measurement vector Z (k)=[u (k), v (k)]
^{t}wherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Substep 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein,
$F=\left[\begin{array}{cccc}1& 0& 1& 0\\ 0& 1& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$ Statetransition matrix,
$H=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right]$ Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Substep 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (kk1)=FS (k1k1);
The covariance matrix P (kk1) of S (kk1) is predicted: P (kk1)=FP (k1) F
^{t}+ Φ (k1), Φ are the variance matrix of process noise;
System state renewal equation is: S (kk)=S (kk1)+K
_{g}(k) ε (K);
The filter error variance matrix of S (kk) under renewal k moment state: P (kk)=(ΨK
_{g}(k) H) P (kk1);
Wherein, K
_{g}(k)=P (kk1) H
^{t}(HP (kk1) H
^{t}+ Γ (k))
^{1}for Kalman gain, ε (k)=Z (k)HS (kk1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order;
Step 7, rapid movement compensation process;
Substep 7a, by the difference u of forward and backward for filtering translation motion component
_{jitter}=uu
_{filter}, v
_{jitter}=vv
_{filter}combining image anglec of rotation θ, as compensating parameter
wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u
_{jitter}for filtered pixel vertical translation amount, v
_{jitter}for filtered pixel level translational movement;
Substep 7b, kinematic parameter model is utilized to calculate present frame f
_{k}the rotation results of first pixel of (x, y) first trip [x, y]:
$\left[\begin{array}{c}{x}^{\′}\\ {y}^{\′}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}{u}_{\mathrm{jitter}}\\ {v}_{\mathrm{jitter}}\end{array}\right];$
Substep 7c, carry out plus and minus calculation according to image coordinate linear structure, calculate present frame f
_{k}the pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame;
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
With reference frame f
_{k1}(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the grayscale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1.
Further concrete scheme is: described step 6 also comprises the correction step of covariance matrix, continues to perform following steps after completing substep 6c:
Substep 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses: ε (k)
^{t}ε (k)≤γ Trace [HP (kk1) H
^{t}+ Γ (k)];
Wherein, γ is adjustability coefficients and γ > 1;
Substep 6e, when in substep 6d formula set up time, illustrate that wave filter is in normal operating conditions, directly obtain the optimal estimation value of current state; When formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (kk1) in weighting coefficient C (k) subparagraphs 6c, the auto adapted filtering of kinematic parameter is completed after correction
Correction formula is as follows:
P(kk1)＝C(k)·F·P(k1)·F
^{T}+Φ(k)，
Further concrete scheme is: to reference frame moving subblock MO in described substep 1f
_{k1}the concrete steps that (m, n) carries out spatial domain similarity detection are: statisticalreference frame moving subblock MO
_{k1}the quantity of the moving subblock that (m, n) surrounding 8 is adjacent, as moving subblock quantity is less than 3, illustrate that this moving subblock is the isolated subblock differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this subblock is similar with field block, all belong to foreground moving region:
Further concrete scheme is: to reference frame moving subblock MO in described substep 1g
_{k1}the concrete steps that (m, n) carries out the detection of time domain similarity are: at current frame motion subblock MO
_{k}judge whether moving subblock in the moving subblock that (m, n) surrounding 8 is adjacent, if having, illustrated that target is continuous in time, real sport foreground, otherwise, the flase drop occurred once in a while should be considered as, need to delete, the final moving subblock retained is sport foreground region:
Further concrete scheme is: carrying out distance verification step to the feature point pairs of coupling in described step 4 is: the ith pair of feature point pairs judging reference frame and present frame in the horizontal direction with the distance d of vertical direction translational movement
_{i}whether meet the following conditions:
 d
_{i}μ  > 3 σ, μ, σ are respectively d
_{i}average and standard deviation,
When meeting above condition, thinking that this feature point pairs is error hiding feature point pairs, being rejected.
From above technical scheme, the inventive method utilizes the body dynamics information of image, based on movement differential, the sport foreground in video sequence and background are split, the background area eliminating foreground target is carried out to extraction and the registration of overall remarkable characteristic, improve the precision of overall motion estimation; Meanwhile, with the vision attention of human eye for instructing, analog vision smoothness properties, builds the low frequency uniform motion pattern of video camera within the continuous imaging time, namely carries out filtering to the high fdrequency component in global motion vector sequence, obtains compensating parameter; During compensation, utilize the linear memory structure of image, only need first conversion parameter of computed image first trip, improve the realtime performance of system, and combining image splicing is rebuild to border, obtains steady and audible panoramic picture.
Compared with prior art, the present invention has following technique effect:
(1) Harris angle point operator is improved, ensure that unique point is the vision remarkable characteristic with unique information: the present invention is to image block, extract the unique point of the larger some of angle point response as remarkable characteristic, the fixed qty of unique point prevents in the too much situation of the unique point number of complex region extraction, and the remarkable information of unique point avoids the error hiding of simple textures repeat region;
(2) owing to adopting foreground moving zone marker and eliminating, ensure that the unique point of extraction is all in background: feature extraction of the present invention is not at entire image extracting directly, but after first moving region being carried out judging, mark and get rid of, only carry out feature detection in background area, thus ensure that unique point well represents the motion of video camera, i.e. global motion information;
(3) because the distance in conjunction with characteristic matching is verified, the precision of further raising overall motion estimation: when there is error hiding, adopts distance criterion, judges and reject this feature point pairs fast, make the unique point participating in beginning parameter transform model be correct coupling, improve the precision of estimation;
(4) selfadaptation SageHusa filtering algorithm is adopted, analog vision smoothness properties, on the one hand preferably smoothing motion vector to reduce video jitter, effectively follow on the other hand the scanning motion of having a mind to of camera system: lowlyer relative to traditional Kalman filter precision even to disperse, selfadaptation SageHusa filtering constantly revises predicted value by observation data, the statistical property of process noise and observation noise is substituted in standard Kalman filter simultaneously, estimate in real time and revise, strengthen the tracking power to mutation status, thus reach reduction model error, improve filtering accuracy,
(5) reconstruction in quick compensation and undefined district, ensure the smoothness of output video when Visual Observations Observations and integrality: the present invention utilizes the relative position between pixel to have the feature of rotational invariance, adopt the fastcompensation method of image rotation, improve the efficiency that coordinate calculates, ensure the realtime performance of system; Meanwhile, for avoiding image compensation to occur undefined dark border, bring harmful effect to visual effect, the present invention utilizes splicing and warm technology, rebuilds undefined district, exports panoramic image sequence.
Accompanying drawing explanation
In order to be illustrated more clearly in the embodiment of the present invention or technical scheme of the prior art, below by need in embodiment or description of the prior art use accompanying drawing do simple introduction, apparently, accompanying drawing in the following describes is only some embodiments of the present invention, for those of ordinary skill in the art, under the prerequisite not paying creative work, other accompanying drawing can also be obtained according to these accompanying drawings.
Fig. 1 is the process flow diagram of the inventive method;
Fig. 2 is feature point pairs distance normal distribution;
Fig. 3 a is the reference frame image after marked moving subblock;
Fig. 3 b is the current frame image after marked moving subblock;
Fig. 3 c is the reference frame image of carrying out spatial domain similarity detection;
Fig. 3 d is the current frame image carrying out the detection of time domain similarity;
Fig. 4 a is the reference frame image being extracted all unique points;
Fig. 4 b is the reference frame image being extracted remarkable characteristic;
Fig. 4 c is the result figure after removing local feature region;
Fig. 4 d is the unique point result figure of registration in present frame;
Fig. 5 a is the image before rotating;
Fig. 5 b is postrotational image;
Fig. 6 a is the comparing result figure after horizontalshift component adopts the inventive method preferred version auto adapted filtering and Kalman filter process;
Fig. 6 b is the comparing result figure after vertical shift component adopts the inventive method preferred version auto adapted filtering and Kalman filter process.
Embodiment
In order to allow above and other objects of the present invention, feature and advantage can be more obvious, the embodiment of the present invention cited below particularly, and coordinate appended diagram, be described below in detail.
Known based on the sensitivity analysis noted motion in human visual system, rocking of background can weaken the notice of vision system to foreground target, and the inconsistent motion of foreground target can the detection of jamming pattern global motion.In order to eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker, the basic ideas of the inventive method are:
First, in motion estimation module, consecutive frame is utilized on average to obtain background image, then adjacent reference frame and present frame are carried out difference with background image respectively, then detect based on the time domain of image block and spatial domain similarity and obtain foreground moving region, prospect and background are separated, in the background area of reference frame, then extract the Harris angle point larger unique point of response and carry out registration, solve the least square solution of determined linear equation, obtain globe motion parameter;
Secondly, at motion compensating module, preferably by improvement SageHusa filtering, the statistical property of realtime estimation and makeover process noise and observation noise, Online Estimation is carried out to process noise and observation noise, obtains final compensating parameter, adopt the quick backoff algorithm of linearity and image mosaic to realize the steady picture of real time panoramic, export the real scene of the complete smoothness of vision, ensure that the realtime of system.
By the right extraction of overall remarkable characteristic, coupling, checking and beginning parameter transform model, and auto adapted filtering smooth motion obtains compensating parameter, improves the vision degree of stability between frame of video and sharpness.
It is more than core concept of the present invention, below in conjunction with the accompanying drawing in the embodiment of the present invention, the technical scheme of the embodiment of the present invention is clearly and completely described, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Set forth a lot of detail in the following description so that fully understand the present invention, but the present invention can also adopt other to be different from alternate manner described here to implement, those skilled in the art can when without prejudice to doing similar popularization when intension of the present invention, therefore the present invention is by the restriction of following public specific embodiment.
Reference Fig. 1, Fig. 1 are the process flow diagram of the inventive method, and the concrete steps of the inventive method are as follows:
Step 1, carry out foreground moving region detection with reference to frame, mark foreground moving subblock step;
In this step after extraction background image, two continuous frames image (reference frame and present frame) and background image are carried out difference, multiple subblock is divided into reference to frame difference image and present frame difference image, compare using the mean difference score value in subblock as threshold value, to determine moving subblock, utilize timespace domain similarity to carry out piecemeal to detect foreground moving region and to go forward side by side row labels, realize the Fast Segmentation of prospect in video sequence and background, remove moving target to the interference of vision attention;
The concrete steps of step 1 are as follows:
Substep 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), as 25 frames before video sequence (1 second) image being averaged and obtaining background image, x, y represent xaxis and the yaxis coordinate of pixel;
Substep 1b, the image in definition k1 moment are reference frame f
_{k1}(x, y), the image in k moment is present frame f
_{k}(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D
_{k1}(x, y)=abs [f
_{k1}(x, y)B (x, y)],
Present frame difference image D
_{k}(x, y)=abs [f
_{k}(x, y)B (x, y)];
Substep 1c, with reference to frame difference image D
_{k1}(x, y) and present frame difference image D
_{k}(x, y) is divided into M × N number of subblock of nonoverlapping copies respectively, and described subblock size is I × J pixel, as 16 × 16 pixels, calculates the mean absolute error in each subblock:
Reference frame image subblock mean absolute error
Current frame image subblock mean absolute error
Wherein, i=1 ..., I, j=1 ..., J, m=1 ..., M, n=1 ..., N;
Substep 1d, computing reference frame subblock difference mean value and present frame subblock difference mean value, respectively as threshold value Th1 and Th2:
Th1＝∑B
_{k1}(m,n)/(M×N)，
Th2＝∑B
_{k}(m,n)/(M×N)；
Substep 1e, tentatively to be judged by binaryzation whether each subblock is moving subblock (Movingobject is called for short MO), definition MO
_{k1}(m, n) is reference frame moving subblock, MO
_{k}(m, n) is current frame motion subblock, and Rule of judgment is as follows:
Substep 1f, to reference frame moving subblock MO
_{k1}(m, n) carries out spatial domain similarity detection, i.e. statisticalreference frame moving subblock MO
_{k1}the quantity of the moving subblock that (m, n) surrounding 8 is adjacent, as moving subblock quantity is less than 3, illustrate that this moving subblock is the isolated subblock differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this subblock is similar with field block, all belong to foreground moving region:
Substep 1g, to reference frame moving subblock MO
_{k1}(m, n) carries out the detection of time domain similarity, namely at the current frame motion subblock MO of correspondence
_{k}judged whether moving subblock in the moving subblock that (m, n) surrounding 8 is adjacent, if having, illustrating that target is continuous in time, is real sport foreground, otherwise, the flase drop occurred once in a while should be considered as, need to delete:
After spatial domain and time domain similarity are detected, the final moving subblock retained is sport foreground region.
The overall remarkable characteristic step of step 2, extraction reference frame;
First Harris angle point response is calculated in this step, and at reference frame f
_{k1}(x, y) find maximum Harris angle point response as characteristic response value in each subblock, then sort, take out position corresponding to larger front 20% characteristic response value as unique point, be the remarkable characteristic with unique information observed visually, according to the mark result in foreground moving region in step 1, judge whether this unique point is positioned at foreground moving region, if be positioned at foreground moving region, delete, what remain is the overall remarkable characteristic observed visually;
The concrete steps of step 2 are as follows:
Substep 2a, with reference to frame f
_{k1}(x, y) utilizes following formula compute gradient image:
Wherein,
represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, []
^{t}represent matrix transpose operation;
Substep 2b, structure autocorrelation matrix R:
Wherein,
for Gaussian smoothing window function,
for the standard deviation of window function;
Substep 2c, calculating Harris angle point response R
_{h}:
R
_{H}＝λ
_{1}×λ
_{2}0.05·(λ
_{1}+λ
_{2})；
Wherein, λ
_{1}and λ
_{2}for two eigenwerts of autocorrelation matrix R;
Substep 2d, with reference to frame f
_{k1}(x, y) is divided into M × N number of subblock of nonoverlapping copies, and subblock size is I × J pixel, with reference to frame f
_{k1}maximum Harris angle point response in each subblock of (x, y) is as the characteristic response value R of this subblock
_{hMAX}(m, n);
Substep 2e, by characteristic response value R
_{hMAX}(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x
_{i}, y
_{i});
Substep 2f, utilize the result of substep 1g to reference frame unique point (x
_{i}, y
_{i}) judge, judge the reference frame moving subblock MO of this Feature point correspondence
_{k1}whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted.
Step 3, feature point pair matching step;
According to the surrounding pixel block message consistance of viewbased access control model correct judgment matching double points, by setting up Window around each unique point of reference frame, and obtain matching window corresponding to present frame, the central point of matching window is matching characteristic point, reference frame unique point and present frame matching characteristic point constitutive characteristic point pair.
The concrete steps of step 3 are as follows:
Substep 3a, at reference frame f
_{k1}with reference frame unique point (x in (x, y)
_{i}, y
_{i}) centered by, build the Window being of a size of P × Q pixel;
Substep 3b, utilize full search strategy and least error and SAD criterion, at present frame f
_{k}corresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point
wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is:
$\mathrm{SAD}(x,y)=\underset{p=1}{\overset{P}{\mathrm{\Σ}}}\underset{q=1}{\overset{Q}{\mathrm{\Σ}}}{f}_{k1}(p,q){f}_{k}(p+x,q+y),$ p＝1,…，P，q＝1,…，Q，x,y＝T,…，T；
Step 4: error hiding feature point pairs rejects step;
The distance of the feature point pairs translational movement in the horizontal direction and the vertical direction of statistics consecutive frame (reference frame and present frame), distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, finally obtain the C of correct coupling to feature point pairs;
According to Euclidean distance formula,
wherein, d
_{i}for reference frame and present frame ith pair of feature point pairs in the horizontal direction with the distance of vertical direction translational movement, as  d
_{i}μ  during > 3 σ, think that this feature point pairs is error hiding feature point pairs, rejected, μ, σ are respectively d
_{i}average and standard deviation.
As shown in Figure 2, experimentally add up, d
_{i}approximate Normal Distribution,
known according to " the 3 σ criterion " of normal distribution, the data on [μ3 σ, μ+3 σ] interval account for 99.7%(Fig. 2 of total data), therefore think and work as  d
_{i}μ  during > 3 σ, this feature point pairs is error hiding feature point pairs.
Step 5: the acquisition step of kinematic parameter;
Set up kinematic parameter model in this step, the feature point pairs correctly mated is substituted into kinematic parameter model, arranges and obtain kinematic parameter matrix equation, obtain kinematic parameter by the least square solution solving overdetermined linear system;
The concrete steps of step 5 are as follows:
Substep 5a, foundation describe reference frame unique point (x
_{i}, y
_{i}) and present frame matching characteristic point
between the kinematic parameter model of relation:
$\left[\begin{array}{c}\hat{x}\\ \hat{y}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}u\\ v\end{array}\right],$ Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter (i.e. rotation, translation parameters);
Substep 5b, step 4 verified the C of the correct coupling obtained substitutes into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation:
$B=\left[\begin{array}{ccc}{\hat{x}}_{1}& & {\hat{y}}_{1}\\ {\hat{x}}_{2}& & {\hat{y}}_{2}\\ & \·& \\ & \·& \\ & \·& \\ {\hat{x}}_{c}& & {\hat{y}}_{c}\end{array}\right],A=\left[\begin{array}{ccc}{x}_{1}& {y}_{1}& 1\\ {x}_{2}& {y}_{2}& 1\\ & \·& \\ & \·& \\ & \·& \\ {x}_{c}& {y}_{c}& 1\end{array}\right],m=\left[\begin{array}{c}\mathrm{\θ}\\ u\\ v\end{array}\right];$
Substep 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A
^{t}a)
^{1}aB, thus obtain kinematic parameter.
Step 6: motion filtering step;
Cumulative motion gain of parameter translation motion parametric line, answers smothing filtering to translation motion parametric line, simulates visual motion smoothing, strengthen the tracking power to mutation status;
The concrete steps of step 6 are as follows:
Substep 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)]
^{t}, measurement vector Z (k)=[u (k), v (k)]
^{t}wherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Substep 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein,
$F=\left[\begin{array}{cccc}1& 0& 1& 0\\ 0& 1& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$ Statetransition matrix,
$H=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right]$ Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Substep 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (kk1)=FS (k1k1);
The covariance matrix P (kk1) of S (kk1) is predicted: P (kk1)=FP (k1) F
^{t}+ Φ (k1), Φ are the variance matrix of process noise;
System state renewal equation is: S (kk)=S (kk1)+K
_{g}(k) ε (K);
The filter error variance matrix of S (kk) under renewal k moment state: P (kk)=(ΨK
_{g}(k) H) P (kk1), wherein, Kg (k)=P (kk1) H
^{t}(HP (kk1) H
^{t}+ Г (k))
^{1}for Kalman gain, ε (k)=Z (k)HS (kk1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order.
As further preferred version, the present invention improves aforementioned SageHusa filtering, increase the correction step of covariance matrix P (kk1), by estimating in real time and the statistical property of makeover process noise and observation noise, carry out selfadaptive smooth filtering to translation motion parametric line, concrete steps are as follows:
Substep 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses:
ε(k)
^{T}·ε(k)≤γ·Trace[H·P(kk1)·H
^{T}+Γ(k)]；
Wherein, γ is adjustability coefficients and γ > 1;
Substep 6e, when in substep 6d formula set up time, wave filter is in normal operating conditions, directly obtains the optimal estimation value of current state; When current formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (kk1) in weighting coefficient C (k) subparagraphs 6c, complete the auto adapted filtering of kinematic parameter after correction;
Correction formula is as follows:
P(kk1)＝C(k)·F·P(k1)·F
^{T}+Φ(k)，
Step 7, rapid movement compensation process;
According to compensating parameter to present frame f
_{k}(x, y) converts, combining image linear memory structure, adopts the plusminus of linear operation, realizes current frame image f
_{k}the quick compensation of (x, y);
The concrete steps of step 7 are as follows:
Substep 7a, by the difference u of forward and backward for filtering translation motion component
_{jitter}=uu
_{filter}, v
_{jitter}=vv
_{filter}combining image anglec of rotation θ, as compensating parameter
wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u
_{jitter}for filtered pixel vertical translation amount, v
_{jitter}for filtered pixel level translational movement;
Substep 7b, kinematic parameter model is utilized to calculate present frame f
_{k}the rotation results of first pixel of (x, y) first trip [x, y]:
$\left[\begin{array}{c}{x}^{\′}\\ {y}^{\′}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}{u}_{\mathrm{jitter}}\\ {v}_{\mathrm{jitter}}\end{array}\right];$
Substep 7c, carry out plus and minus calculation according to coordinate linear structure, calculate present frame f
_{k}the pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame.
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
Compensating parameter is utilized according to step 7
to present frame f
_{k}after (x, y) pixel carries out coordinate transform, with reference frame f
_{k1}(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the grayscale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and present frame and reference frame overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1.Thus achieve in overlapping region by reference frame f
_{k1}(x, y) is slowly smoothly transitted into present frame f
_{k}(x, y), makes the warm image effect obtained more natural, can not affect the observing effect of whole video, improve the integrality of vision.
Effect of the present invention is described by following test experience.
According to the perception of human visual to motion, human eye cannot observe the difference of single pixel, but judges motion with the lasting change in whole region.As shown in Figure 3 a and Figure 3 b shows, Fig. 3 a is the reference frame image after marked moving subblock, and Fig. 3 b is the current frame image after marked moving subblock, and in Fig. 3 a and Fig. 3 b, "+" symbol represents that this subblock is moving subblock.
Spatial domain similarity analysis is carried out to 8 neighborhood blocks of moving subblock in reference frame image, if block existence is around no less than 3 moving subblock, illustrates that this block is similar with field block, all belong to foreground moving region; Otherwise judge that this block is as the more isolated block of residual error in background, deleted, the candidate's prospect subblock retained after deleting as shown in Figure 3 c.
Carry out time domain similarity analysis to moving subblock in reference frame image again: if in continuous print present frame, there is moving subblock in 8 similar domain location, is then judged as real motion; Otherwise should be considered as the flase drop occurred once in a while, need to delete, the final subblock retained is the sport foreground region detected, as shown in Figure 3 d.
With reference to Fig. 4 a to Fig. 4 d, for overall remarkable characteristic extracts and the procedure chart of feature point pair matching.
Fig. 4 a is all unique points of reference frame image multisubarea extracting, can find out, unique point is evenly distributed in entire image, Partial Feature point has been selected on moving target, a large amount of unique point is separately had to be positioned at the background area (as sky and ground) of texture repetition, this two category features point can cause error hiding, thus causes the reduction of overall motion estimation precision.
Fig. 4 b is that remarkable characteristic extracts result figure, remains front 20% unique point that characteristic response is larger, improves the uniqueness of characteristic point information.
Fig. 4 c is the result figure after removing error hiding unique point, the experimental result of composition graphs 3d, whether judging characteristic point position is positioned at foreground moving or the unreliable region of surrounding, if then directly deleted, and retain the overall remarkable characteristic of background area, thus be beneficial to the correct coupling of unique point.
Fig. 4 d is the unique point result figure of registration corresponding in current frame image, unique point success registration, and can process in real time.
With reference to Fig. 5 a and 5b, the principle that the linearity occurred in description above compensates fast is described as follows:
The linear memory structure of image ensure that the relative position between pixel has rotational invariance.With reference to Fig. 5 a and 5b, Fig. 5 a is image rotation preceding pixel point position, image definition in rectangular domain ABCD, any pixel E (x, y) and the first row pixel E1 (x with its same column
_{1}, y
_{1}), the first row pixel E2 (x that goes together with it
_{2}, y
_{2}) and the first row pixel A (x of the first row
_{a}, y
_{a}) in the four summit relations being geometrically rectangle.In arbitrary system, an apex coordinate E (x, y) of rectangle can be determined by other three apex coordinates:
Rotational transform is linear transformation, and the shape of rectangle does not change because of rotation.Fig. 5 b is pixel position after image rotation, after image rotation, four summits of rectangle be A ' (x '
_{a}, y '
_{a}), E1 ' (x
_{1}', y
_{1}'), E2 ' (x
_{2}', y
_{2}'), E ' (x ', y '), its coordinate relation still meets relational expression:
Therefore, the rotation results of image first trip and first pixel is utilized just can to calculate the rotation results of other all pixel.
Concrete steps are: do coordinate transform with the pixel of first row by similar variation model to the first row, variation model is
$\left[\begin{array}{c}{x}^{\′}\\ {y}^{\′}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}u\\ v\end{array}\right];$ Plus and minus calculation can be carried out with above formula to the pixel of all the other ranks.This avoid and matrix multiplication operation is done to each pixel of entire image, thus effectively save operation time, improve the efficiency that coordinate calculates.
With reference to Fig. 6 a and Fig. 6 b, to be filtered into example to horizontal and vertical offset component, give the comparing result figure of the Kalman filter of its inventive method preferred version auto adapted filtering and prior art.Owing to horizontal direction existing nearly camerascanning motion at the uniform velocity, therefore the empirical curve of Fig. 6 a is in stablizing propradation; Only there is randomized jitter in video camera, therefore the empirical curve of Fig. 6 b is in 0 positional fluctuation in the vertical direction.From this figure, select different process noise Q larger on compensation result impact in Kalman filter: when process noise Q value is larger, filter curve and primary curve approach, therefore without obvious filter effect; When process noise Q value is less, though can obtain level and smooth filter curve, after observing 52 frames by Fig. 6 b, original dither terminates, and its filter result departs from 0 vector, thus causes filtering divergence.And selfadaptive routing of the present invention effectively can avoid the phenomenon of this filtering divergence, can good level and smooth jittering component, the real scan motion of tracking camera effectively simultaneously.
According to above technical scheme, first the inventive method under the moving scene environment that there is moving target, proposes detection and the method for registering of overall remarkable characteristic, improves speed and the precision of overall motion estimation; Next solves in Camera location scanning process and occurs shake, improves SageHusa filtering to distinguish scanning and shake adaptively, while smooth motion, follows the tracks of real scan scene; The timeconsuming computing that the matrix multiplication finally avoiding pointwise during image conversion brings, based on linearity storage organization, propose linear operation thus realize compensating fast, and the border that combining image merges losing when compensating rebuilds, eliminate or alleviate the wild effect of video sequence, improve the observation effect of video monitoring or tracker.
The above, it is only preferred embodiment of the present invention, not any pro forma restriction is done to the present invention, although the present invention discloses as above with preferred embodiment, but and be not used to limit the present invention, any those skilled in the art, do not departing within the scope of technical solution of the present invention, make a little change when the technology contents of abovementioned announcement can be utilized or be modified to the Equivalent embodiments of equivalent variations, in every case be the content not departing from technical solution of the present invention, according to any simple modification that technical spirit of the present invention is done above embodiment, equivalent variations and modification, all still belong in the scope of technical solution of the present invention.
Claims (2)
1. an electronic image stabilization method for viewbased access control model attention mechanism, is characterized in that, comprises the following steps:
Step 1, foreground moving region detection is carried out to reference frame, mark foreground moving subblock step;
Substep 1a, be averaged to one section of sequential frame image of video sequence, obtain background image B (x, y), x, y represent xaxis and the yaxis coordinate of pixel;
Substep 1b, the image in definition k1 moment are reference frame f
_{k1}(x, y), the image in k moment is present frame f
_{k}(x, y), calculates the difference image of they and background image B (x, y) respectively:
Reference frame difference image D
_{k1}(x, y)=abs [f
_{k1}(x, y)B (x, y)],
Present frame difference image D
_{k}(x, y)=abs [f
_{k}(x, y)B (x, y)];
Substep 1c, with reference to frame difference image D
_{k1}(x, y) and present frame difference image D
_{k}(x, y) is divided into M × N number of subblock of nonoverlapping copies respectively, and described subblock size is I × J pixel, calculates the mean absolute error in each subblock:
Reference frame image subblock mean absolute error
Current frame image subblock mean absolute error
Wherein, i=1 ..., I, j=1 ..., J, m=1 ..., M, n=1 ..., N;
Substep 1d, computing reference frame subblock difference mean value and present frame subblock difference mean value, respectively as threshold value Th1 and Th2:
Th1＝∑B
_{k1}(m,n)/(M×N)，
Th2＝∑B
_{k}(m,n)/(M×N)；
Substep 1e, tentatively judge whether each subblock is moving subblock by binaryzation, definition MO
_{k1}(m, n) is reference frame moving subblock, MO
_{k}(m, n) is current frame motion subblock, and Rule of judgment is as follows:
Substep 1f, to reference frame moving subblock MO
_{k1}(m, n) carries out spatial domain similarity detection, the subblock not belonging to sport foreground is deleted;
Substep 1g, to reference frame moving subblock MO
_{k1}(m, n) carries out the detection of time domain similarity, the subblock not belonging to sport foreground is deleted;
After spatial domain, time domain similarity detect, the final moving subblock retained is sport foreground region;
Overall remarkable characteristic step in step 2, extraction reference frame;
Substep 2a, with reference to frame f
_{k1}(x, y) utilizes following formula compute gradient image:
Wherein,
represent convolution, X represents the gradient image of horizontal direction, and Y represents the gradient image of vertical direction, []
^{t}represent matrix transpose operation;
Substep 2b, structure autocorrelation matrix R:
Wherein,
for Gaussian smoothing window function,
for the standard deviation of window function;
Substep 2c, calculating Harris angle point response R
_{h}:
R
_{H}＝λ
_{1}×λ
_{2}0.05·(λ
_{1}+λ
_{2})；
Wherein, λ
_{1}and λ
_{2}for two eigenwerts of autocorrelation matrix R;
Substep 2d, with reference to frame f
_{k1}(x, y) is divided into M × N number of subblock of nonoverlapping copies, and subblock size is I × J pixel, with reference to frame f
_{k1}maximum Harris angle point response in each subblock of (x, y) is as the characteristic response value R of this subblock
_{hMAX}(m, n);
Substep 2e, by characteristic response value R
_{hMAX}(m, n) carries out sequence from high to low, takes out front 20% higher value, and position corresponding for described characteristic response value is designated as reference frame unique point (x
_{i}, y
_{i});
Substep 2f, utilize the result of substep 1g to reference frame unique point (x
_{i}, y
_{i}) judge, judge the reference frame moving subblock MO of this Feature point correspondence
_{k1}whether be 1 in (m, n) and around 8 adjacent areas, if 1, then show that this unique point belongs to moving target or the unreliable region at moving boundaries, this unique point is deleted;
Step 3, feature point pair matching step;
Substep 3a, at reference frame f
_{k1}with reference frame unique point (x in (x, y)
_{i}, y
_{i}) centered by, build the Window being of a size of P × Q pixel;
Substep 3b, utilize full search strategy and least error and SAD criterion, at present frame f
_{k}corresponding matching window is found in (x, y), matching window is of a size of (P+2T) × (Q+2T) pixel, the central point of matching window is present frame matching characteristic point
wherein, T represents the pixel maximum offset of horizontal direction and vertical direction, and the computing formula of SAD criterion is:
$SAD(x,y)=\underset{p=1}{\overset{P}{\Σ}}\underset{q=1}{\overset{Q}{\Σ}}{f}_{k1}(p,q){f}_{k}(p+x,q+y),$ p＝1,…,P，q＝1,…,Q，x,y＝T,…,T；
Step 4: error hiding feature point pairs rejects step;
According to Euclidean distance formula
ith pair of feature point pairs of computing reference frame and present frame in the horizontal direction with the distance of vertical direction translational movement, distance normality distribution characteristics is utilized by the feature point pairs of coupling to carry out distance checking, reject error hiding feature point pairs, obtain the C of correct coupling to feature point pairs;
Step 5: the acquisition step of kinematic parameter;
Substep 5a, foundation describe reference frame unique point (x
_{i}, y
_{i}) and present frame matching characteristic point
between the kinematic parameter model of relation:
$\left[\begin{array}{c}\hat{x}\\ \hat{y}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}u\\ v\end{array}\right],$ Wherein, θ is image rotation angle, and u is pixel vertical translation amount, and v is pixel level translational movement, θ, u and v component movement parameter;
Substep 5b, the C that correctly mates is substituted into kinematic parameter model to feature point pairs, arranges and obtain kinematic parameter matrix equation:
$B=\left[\begin{array}{ccc}{\hat{x}}_{1}& & {\hat{y}}_{1}\\ {\hat{x}}_{2}& & {\hat{y}}_{2}\\ & .& \\ & .& \\ & .& \\ {\hat{x}}_{c}& & {\hat{y}}_{c}\end{array}\right],A=\left[\begin{array}{ccc}{x}_{1}& {y}_{1}& 1\\ {x}_{2}& {y}_{2}& 1\\ & .& \\ & .& \\ & .& \\ {x}_{c}& {y}_{c}& 1\end{array}\right],m=\left[\begin{array}{c}\mathrm{\θ}\\ u\\ v\end{array}\right];$
Substep 5c, solve determined linear equation B=Am, the least square solution of kinematic parameter matrix m is m=(A
^{t}a)
^{1}aB, thus obtain kinematic parameter;
Step 6: motion filtering step;
Substep 6a, writ state vector S (k)=[u (k), v (k), du (k), dv (k)]
^{t}, measurement vector Z (k)=[u (k), v (k)]
^{t}wherein, the pixel vertical translation amount that u (k) is the k moment, the pixel level translational movement that v (k) is the k moment, du (k) is instantaneous velocity corresponding to k moment pixel vertical translation amount, and dv (k) is instantaneous velocity corresponding to k moment pixel level translational movement;
Substep 6b, set up linear discrete system model, obtain state equation and observation equation:
State equation is S (k)=FS (k1)+δ,
Observation equation is Z (k)=HS (k)+η;
Wherein,
$F=\left[\begin{array}{cccc}1& 0& 1& 0\\ 0& 1& 0& 1\\ 0& 0& 1& 0\\ 0& 0& 0& 1\end{array}\right]$ Statetransition matrix,
$H=\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\end{array}\right]$ Be observing matrix, δ, η are separate white noise, δ ~ N (0, Φ), η ~ N (0, Γ), and Φ is the variance matrix of process noise, and Γ is the variance matrix of observation noise;
Substep 6c, set up system state predictive equation, and its covariance matrix predicted and upgrades, complete motion filtering:
System state predictive equation is: S (kk1)=FS (k1k1);
The covariance matrix P (kk1) of S (kk1) is predicted: P (kk1)=FP (k1) F
^{t}+ Φ (k1), Φ are the variance matrix of process noise;
System state renewal equation is: S (kk)=S (kk1)+K
_{g}(k) ε (K);
The filter error variance matrix of S (kk) under renewal k moment state: P (kk)=(ΨK
_{g}(k) H) P (kk1);
Wherein, K
_{g}(k)=P (kk1) H
^{t}(HP (kk1) H
^{t}+ Γ (k))
^{1}for Kalman gain, ε (k)=Z (k)HS (kk1) is innovation sequence, and Γ is the variance matrix of observation noise, and Ψ is the unit matrix of same order;
Step 7, rapid movement compensation process;
Substep 7a, by the difference u of forward and backward for filtering translation motion component
_{jitter}=uu
_{filter}, v
_{jitter}=vv
_{filter}combining image anglec of rotation θ, as compensating parameter
wherein, u is pixel vertical translation amount, and v is pixel level translational movement, u
_{jitter}for filtered pixel vertical translation amount, v
_{jitter}for filtered pixel level translational movement;
Substep 7b, kinematic parameter model is utilized to calculate present frame f
_{k}the rotation results of first pixel of (x, y) first trip [x, y]:
$\left[\begin{array}{c}{x}^{\′}\\ {y}^{\′}\end{array}\right]=\left[\begin{array}{cc}1& \mathrm{\θ}\\ \mathrm{\θ}& 1\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right]+\left[\begin{array}{c}{u}_{jitter}\\ {v}_{jitter}\end{array}\right];$
Substep 7c, carry out plus and minus calculation according to image coordinate linear structure, calculate present frame f
_{k}the pixel of (x, y) all the other ranks, obtains the new coordinate [x ', y '] of current frame pixel, realizes the compensation of present frame;
Step 8, rebuild undefined boundary information, obtain panoramic picture step;
With reference frame f
_{k1}(x, y) as initial panorama sketch, utilize image mosaic technology, reference frame and present frame are merged, determine the grayscale value of each pixel (x ', y ') of fused image according to the warm strategy of image, be compensated image f (x ', y '), realize panoramic picture and export:
τ, ξ in above formula represent weighted value, represent the ratio of this pixel relative position and overlapping region width, the i.e. difference of this pixel and frontier point position, and τ+ξ=1,0 < τ, ξ < 1, in overlapping region, τ is by 1 gradual change to 0, and ξ is by 0 gradual change to 1;
To reference frame moving subblock MO in described substep 1f
_{k1}the concrete steps that (m, n) carries out spatial domain similarity detection are: statisticalreference frame moving subblock MO
_{k1}the quantity of the moving subblock that (m, n) surrounding 8 is adjacent, as moving subblock quantity is less than 3, illustrate that this moving subblock is the isolated subblock differed greatly in background, do not belong to sport foreground, deleted, otherwise illustrate that this subblock is similar with field block, all belong to foreground moving region:
To reference frame moving subblock MO in described substep 1g
_{k1}the concrete steps that (m, n) carries out the detection of time domain similarity are: at current frame motion subblock MO
_{k}judge whether moving subblock in the moving subblock that (m, n) surrounding 8 is adjacent, if having, illustrated that target is continuous in time, real sport foreground, otherwise, be considered as the flase drop occurred once in a while, need to delete, the final moving subblock retained is sport foreground region:
Described step 6 also comprises the correction step of covariance matrix, continues to perform following steps after completing substep 6c:
Substep 6d, utilize the character of innovation sequence ε (k) to judge whether filtering disperses: ε (k)
^{t}ε (k)≤γ Trace [HP (kk1) H
^{t}+ Γ (k)];
Wherein, γ is adjustability coefficients and γ > 1;
Substep 6e, when in substep 6d formula set up time, illustrate that wave filter is in normal operating conditions, directly obtain the optimal estimation value of current state; When formula is false, show that actual error will exceed the γ of theoretical estimated value doubly, filtering will be dispersed, and now be revised by the covariance matrix P (kk1) in weighting coefficient C (k) subparagraphs 6c, the auto adapted filtering of kinematic parameter is completed after correction
Correction formula is as follows:
P(kk1)＝C(k)·F·P(k1)·F
^{T}+Φ(k)，
2. the electronic image stabilization method of viewbased access control model attention mechanism according to claim 1, is characterized in that: carrying out distance verification step to the feature point pairs of coupling in described step 4 is: the ith pair of feature point pairs judging reference frame and present frame in the horizontal direction with the distance d of vertical direction translational movement
_{i}whether meet the following conditions:
 d
_{i}μ  > 3 σ, μ, σ are respectively d
_{i}average and standard deviation,
When meeting above condition, think that this feature point pairs is error hiding feature point pairs, rejected.
Priority Applications (1)
Application Number  Priority Date  Filing Date  Title 

CN201310287353.XA CN103426182B (en)  20130709  20130709  The electronic image stabilization method of viewbased access control model attention mechanism 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

CN201310287353.XA CN103426182B (en)  20130709  20130709  The electronic image stabilization method of viewbased access control model attention mechanism 
Publications (2)
Publication Number  Publication Date 

CN103426182A CN103426182A (en)  20131204 
CN103426182B true CN103426182B (en)  20160106 
Family
ID=49650872
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

CN201310287353.XA CN103426182B (en)  20130709  20130709  The electronic image stabilization method of viewbased access control model attention mechanism 
Country Status (1)
Country  Link 

CN (1)  CN103426182B (en) 
Families Citing this family (20)
Publication number  Priority date  Publication date  Assignee  Title 

CN103905826A (en) *  20140410  20140702  北京工业大学  Selfadaptation global motion estimation method 
CN104853064B (en) *  20150410  20180417  海视英科光电（苏州）有限公司  Electronic image stabilization method based on thermal infrared imager 
CN105263026B (en) *  20151012  20180417  西安电子科技大学  Global vector acquisition methods based on probability statistics and image gradient information 
CN105758867B (en) *  20160311  20191105  苏州睿仟医疗科技有限公司  A kind of microscopic defect check method of high speed 
CN106375659B (en) *  20160606  20190611  中国矿业大学  Electronic image stabilization method based on multiresolution Gray Projection 
CN106128007B (en) *  20160826  20180420  宁波圣达精工智能科技有限公司  Intelligent compact shelf controls guard system 
CN106210447B (en) *  20160909  20190514  长春大学  Based on the matched video image stabilization method of background characteristics point 
CN106357958B (en) *  20161010  20190416  山东大学  A kind of swift electron digital image stabilization method based on Region Matching 
CN108881668A (en) *  20170602  20181123  北京旷视科技有限公司  Video increases steady method, apparatus, system and computerreadable medium 
CN107197121B (en) *  20170614  20190726  长春欧意光电技术有限公司  A kind of electronic image stabilization method based on onboard equipment 
CN107423409B (en) *  20170728  20200331  维沃移动通信有限公司  Image processing method, image processing device and electronic equipment 
CN107578428A (en) *  20170831  20180112  成都观界创宇科技有限公司  Method for tracking target and panorama camera applied to panoramic picture 
CN108174087B (en) *  20171226  20190702  北京理工大学  A kind of steady reference frame update method and system as in of Gray Projection 
CN108492328B (en) *  20180323  20210226  云南大学  Video interframe target matching method and device and implementation device 
CN108765532A (en) *  20180504  20181106  北京物灵智能科技有限公司  Children paint this method for establishing model, reading machine people and storage device 
CN109561253B (en) *  20181218  20200710  影石创新科技股份有限公司  Method and device for preventing panoramic video from shaking, portable terminal and storage medium 
CN109816006B (en) *  20190118  20201113  深圳大学  Seaskyline detection method and device and computerreadable storage medium 
CN109922258B (en) *  20190227  20201103  杭州飞步科技有限公司  Electronic image stabilizing method and device for vehiclemounted camera and readable storage medium 
CN110046555A (en) *  20190326  20190723  合肥工业大学  Endoscopic system video image stabilization method and device 
CN110120023A (en) *  20190514  20190813  浙江工大盈码科技发展有限公司  A kind of image feedback antidote 
Citations (3)
Publication number  Priority date  Publication date  Assignee  Title 

CN101316368A (en) *  20080718  20081203  西安电子科技大学  Full view stabilizing method based on global characteristic point iteration 
CN101729763A (en) *  20091215  20100609  中国科学院长春光学精密机械与物理研究所  Electronic image stabilizing method for digital videos 
CN103024247A (en) *  20110928  20130403  中国航天科工集团第二研究院二〇七所  Electronic image stabilization method based on improved block matching 
Family Cites Families (1)
Publication number  Priority date  Publication date  Assignee  Title 

US8817190B2 (en) *  20071128  20140826  Canon Kabushiki Kaisha  Image processing apparatus, image processing method, and computer program 

2013
 20130709 CN CN201310287353.XA patent/CN103426182B/en active IP Right Grant
Patent Citations (3)
Publication number  Priority date  Publication date  Assignee  Title 

CN101316368A (en) *  20080718  20081203  西安电子科技大学  Full view stabilizing method based on global characteristic point iteration 
CN101729763A (en) *  20091215  20100609  中国科学院长春光学精密机械与物理研究所  Electronic image stabilizing method for digital videos 
CN103024247A (en) *  20110928  20130403  中国航天科工集团第二研究院二〇七所  Electronic image stabilization method based on improved block matching 
NonPatent Citations (1)
Title 

电子稳像理论及其应用;朱娟娟;《中国博士学位论文全文数据库（Information Science and Technology）》;20091007;89130 * 
Also Published As
Publication number  Publication date 

CN103426182A (en)  20131204 
Similar Documents
Publication  Publication Date  Title 

Jaritz et al.  Sparse and dense data with cnns: Depth completion and semantic segmentation  
Sun et al.  Correlation tracking via joint discrimination and reliability learning  
Wang et al.  Detect globally, refine locally: A novel approach to saliency detection  
Seki et al.  Sgmnets: Semiglobal matching with neural networks  
Li et al.  Flow guided recurrent neural encoder for video salient object detection  
Kueng et al.  Lowlatency visual odometry using eventbased feature tracks  
Zhou et al.  Efficient road detection and tracking for unmanned aerial vehicle  
Pandey et al.  Automatic targetless extrinsic calibration of a 3d lidar and camera by maximizing mutual information  
Qiu et al.  Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image  
Xiao et al.  Fast image dehazing using guided joint bilateral filter  
Choi et al.  RGBD edge detection and edgebased registration  
Liu et al.  Guided inpainting and filtering for kinect depth maps  
CN105245841B (en)  A kind of panoramic video monitoring system based on CUDA  
US9105093B2 (en)  Method and apparatus for bilayer segmentation  
Gallup et al.  Piecewise planar and nonplanar stereo for urban scene reconstruction  
Park et al.  Look wider to match image patches with convolutional neural networks  
CN103325112B (en)  Moving target method for quick in dynamic scene  
Zhang et al.  Semantic segmentation of urban scenes using dense depth maps  
Wang et al.  A region based stereo matching algorithm using cooperative optimization  
Lai et al.  Singleimage dehazing via optimal transmission map under scene priors  
Schneider et al.  RegNet: Multimodal sensor registration using deep neural networks  
Yang et al.  Colorguided depth recovery from RGBD data using an adaptive autoregressive model  
Saxena et al.  Learning 3d scene structure from a single still image  
Kong et al.  General road detection from a single image  
CN108573276B (en)  Change detection method based on highresolution remote sensing image 
Legal Events
Date  Code  Title  Description 

C06  Publication  
PB01  Publication  
C10  Entry into substantive examination  
SE01  Entry into force of request for substantive examination  
C14  Grant of patent or utility model  
GR01  Patent grant  
TR01  Transfer of patent right  
TR01  Transfer of patent right 
Effective date of registration: 20210330 Address after: Floor 5, block D, Boyuan science and Technology Plaza, No.99, Yanxiang Road, Yanta District, Xi'an City, Shaanxi Province, 710000 Patentee after: Xijiao Sichuang Intelligent Technology Research Institute (Xi'an) Co.,Ltd. Address before: 710071 No. 2 Taibai South Road, Shaanxi, Xi'an Patentee before: XIDIAN University 