CN100525452C - Frequency domain fast sub picture element global motion estimating method for image stability - Google Patents

Abstract

A frequency-domain fast sub-pixel global motion estimating method used to stabilize image includes calculating mutual power spectral function between two adjacent frame images in video sequence, generating motion distribution function by making anti-Fourier transform on mutual power spectral function, searching its maximum peak point to obtain pixel global motion vector, judging and recoding a numbers of sub-peak values adjacent to maximum peak value point coordinate in said motion distribution function to calculate out sub-pixel global motion vector (SGMV) and carrying out approximate treatment on calculated SGMV for obtaining SGMV with assigned precision.

Description

The frequency domain fast sub picture element global motion estimating method that is used for image stabilization

Technical field

The present invention relates to a kind of frequency domain fast sub picture element global motion estimating method that is used for image stabilization, calculating and storage overhead, raising sub picture element global estimation accuracy when relating in particular to a kind of sub picture element global estimation that can reduce under the frequency domain condition also improved the method for picture steadiness, belongs to technical field of image processing.

Background technology

GLOBAL MOTION ESTIMATION TECHNOLOGY can be used for obtaining the mass motion situation between adjacent two two field pictures of video sequence.It is a sport technique segment the most key in the image stabilization process.Overall motion estimation has influenced the performance performance of image stabilization system from following two aspects.At first, the accuracy of overall motion estimation and precision have directly determined the image stabilization quality of system.Secondly, because the amount of calculation of overall motion estimation part generally accounts for more than 90% of system-computed amount, so the speed of overall motion estimation has determined the processing capability in real time of steady picture system.

Referring to shown in Figure 1, overall motion estimation is made up of whole picture element global estimation and sub picture element global estimation two parts.Whole picture element global estimation is based on the Direct Sampling image, and the global motion vector of the whole Pixel-level between image is estimated; The sub picture element global estimation then at first estimates the value of non-Direct Sampling point (sub-pix point) in the image by interpolation, based on this, further estimate the sub picture element global motion vector on the basis of known whole picture element global motion vector.Relevant experiment shows that the sub picture element global estimation can be described the global motion situation between image more exactly, effectively improves the motion compensation effect.Therefore the sub picture element global estimation becomes the research focus in current many applications.

The sub picture element global estimation can be divided into time domain and two kinds of implementations of frequency domain.At application number is 200410073837.5, and the applying date is in the Chinese invention patent application "-kind of fast sub-picture element movement estimating method " on September 3rd, 2004, discloses a kind of fast sub-picture element movement estimating method based on time domain.Compare with this time domain implementation method, have advantages such as applied widely, that antijamming capability is strong, thereby be more suitable for image stabilization and use based on the global motion estimating method of frequency domain.But in the specific implementation, the frequency domain global motion estimating method need obtain the global motion situation between image in the spatial domain by the phase relation that compares adjacent two two field picture frequency spectrums in the video sequence according to the translation feature of Fourier transform.Have the Fourier transform process between spatial domain and the frequency domain in this process, amount of calculation is along with the variation of image size increases very fast.For example carry out 1/2 ^kDuring the sub picture element global estimation of pixel precision, the image size after the interpolation becomes 4 of former figure ^kDoubly, corresponding storage overhead will increase by 4 ^k-1 times, and computing cost approaches 16 of whole pixel situation when adopting discrete Fourier transform (DFT) ^kDoubly, even adopt fast fourier transformation algorithm, amount of calculation also will reach 8 of original amount of calculation ^kDoubly.For the frequency domain global motion estimating method, this sub-pix extended mode not only computational complexity improves, and along with the increase of sub-pixel precision, storage overhead also exponentially level increases, and therefore is difficult to satisfy the actual demand of the limited applied environment of storage and computational resource.On the other hand, when the global motion estimating method of implementing based on frequency domain, the image border content change is apparent in view to the interference of frequency domain overall motion estimation, therefore must be overcome targetedly.

Summary of the invention

Primary and foremost purpose of the present invention provides the frequency domain fast sub picture element global motion estimating method that is used for image stabilization.The frequency domain phase change information that this method can utilize the motion of image sub picture element global to produce is not being carried out under the situation of image interpolation, by directly calculating image sub picture element global motion vector.

Another object of the present invention is to provide a kind of can effectively suppress the method for image border content change to frequency domain interference that overall motion estimation causes.This method can improve the accuracy of frequency domain overall motion estimation, especially the accuracy of frequency domain sub picture element global estimation.

For achieving the above object, the present invention adopts following technical scheme.

A kind of frequency domain fast sub picture element global motion estimating method that is used for image stabilization is characterized in that comprising the steps:

(1) the original crosspower spectrum of adjacent two two field pictures in the calculating video sequence;

(2) the maximum interference coefficient of acquisition each Frequency point row, column direction correspondence under the designated precision scope, then that the amplitude of each Frequency point in the former frame image spectrum is corresponding with the row, column direction respectively maximum interference multiplication, obtain the size of the maximum interference component that each Frequency point can overcome under the designated precision scope, minimum value and empirical value in the maximum interference component of each Frequency point are compared, judge whether this Frequency point is the Frequency point that is subject to disturb;

(3) suppress for being judged as the Frequency point that is subject to disturb, obtain to disturb the crosspower spectrum after suppressing;

(4) crosspower spectrum after described interference is suppressed carries out inverse-Fourier transform, generates the distribution of movement function, and the peak-peak in the searching moving distribution function obtains whole picture element global motion vector according to the coordinate of maximal peak point;

(5) judge and note in the described distribution of movement function several times peak value adjacent on the row, column direction, calculate the sub picture element global motion vector according to the corresponding relation of primary and secondary peak Distribution in sub picture element global motion and the distribution of movement function with the peak-peak point coordinates;

(6) the sub picture element global motion vector that calculates is carried out approximate processing, obtain the sub picture element global motion vector under the designated precision.

Wherein, it is relevant that the size of described empirical value and the grain details of image are enriched the degree forward, between 200～400.

Described step (5) is divided into two steps:

At first, according to the peak-peak that search in the step (4) obtains, note respectively in the horizontal direction with vertical direction on the functional value of the most contiguous three coordinate points of maximal peak point;

Secondly, according to the peak-peak that obtains and note in the horizontal direction with vertical direction on the functional value of the most contiguous three coordinate points of maximal peak point, calculate the sub picture element global motion vector under the corresponding arbitrary accuracy respectively.

Calculating and storage overhead when the frequency domain fast sub picture element global motion estimating method that is used for image stabilization provided by the present invention can effectively reduce traditional frequency domain sub picture element global estimation, and have interference suppressioning effect preferably for the image border content change, improved the accuracy of frequency domain sub picture element global estimation.Relevant test result shows that this method can both obtain effect preferably for the sub picture element global estimation of all kinds of scene images in the image stabilization.

Description of drawings

The present invention is further illustrated below in conjunction with the drawings and specific embodiments.

Fig. 1 is the pixel distribution schematic diagram of the sub picture element global motion of two dimension;

Fig. 2 is the flow chart that is used for the frequency domain fast sub picture element global motion estimating method of image stabilization provided by the present invention;

Fig. 3 is sub picture element global when motion motion distribution function peak Distribution schematic diagram of one dimension.

Embodiment

Under frequency domain condition, a kind of good sub picture element global motion estimating method must be considered the influence of precision, accuracy, operational efficiency and the storage overhead of estimation simultaneously, can effectively overcome image border content change adverse effect simultaneously.For this reason, solution thinking of the present invention is: according to the corresponding relation of sub picture element global motion between the image of spatial domain with frequency domain intermediate frequency spectrum situation of change, on the basis of the whole picture element global estimation of frequency domain, utilize its middle result, by calculating direct acquisition sub picture element global motion vector, realized frequency domain sub picture element global estimation fast and accurately.

Below in conjunction with the description of drawings specific implementation of the present invention.Fig. 2 is the flow chart that is used for the frequency domain fast sub picture element global motion estimating method of image stabilization provided by the present invention.This method comprises following step: at first, calculate the crosspower spectrum function S between adjacent two two field pictures in the video sequence ₀, and, the crosspower spectrum function is disturbed inhibition according to the form of expression of image border interference that content change produces, obtain comprising the crosspower spectrum function S of more accurate global motion information ₁To the crosspower spectrum function S that newly obtains ₁Carry out inverse-Fourier transform, generate distribution of movement function I, search for its maximal peak point and obtain whole picture element global motion vector; Secondly, judge and note among the distribution of movement function I the several minor peaks (be preferably three time) adjacent on row (row) direction, just can calculate the sub picture element global motion vector according to the corresponding relation of primary and secondary peak Distribution in sub picture element global motion and the distribution of movement function with the peak-peak point coordinates; At last, the sub picture element global motion vector that calculates is carried out approximate processing, obtain the sub picture element global motion vector under the designated precision.

The step of the whole picture element global motion vector of above-mentioned acquisition specifically comprises three following sub-steps: at first, calculate the original crosspower spectrum of adjacent two two field pictures in the video sequence according to the definition of crosspower spectrum function; Then, the information of different frequency point in the crosspower spectrum is screened and suppress according to selected accuracy rating, acquisition has the more crosspower spectrum of high reliability; At last, the new crosspower spectrum that obtains after disturb suppressing is carried out inverse-Fourier transform, generate the distribution of movement function, the peak-peak in the searching moving distribution function obtains whole picture element global motion vector according to the coordinate of maximal peak point.

Concrete calculation procedure is as follows:

(1) calculating of original crosspower spectrum

Crosspower spectrum has characterized the correlation between the two two field picture frequency spectrums.Each Frequency point all includes global motion information between two two field pictures in the crosspower spectrum in theory.When calculating crosspower spectrum, at first adopt formula (1) and formula (2) to calculate the frequency spectrum F that two frames are treated estimated image ₁(u, v), F ₂(u v), calculates according to formula (3) then and determines original crosspower spectrum function S ₀(u, v).

F ₁(u，v)＝F(f ₁(x，y)) (1)

F ₂(u，v)＝F(f ₂(x，y)) (2)

F in formula (1), (2) ₁(x, y), f ₂(x y) represents that respectively two frames treat estimated image, F ₁(u, v), F ₂(u represents then that v) two frames treat the frequency spectrum of estimated image correspondence, and symbol F represents Fourier transform.In order to improve the speed of overall motion estimation, use fast fourier transform to replace common discrete Fourier transform (DFT) usually in actual applications, the image size is taken as 2 integral number power.

$S_0(u,v)=\frac{F_2(u,v)}{F_1(u,v)}---\left(3\right)$

S in the formula (3) ₀(u v) is two frames and treats original crosspower spectrum between estimated image.

(2) interference of crosspower spectrum suppresses

In the ideal case, the global motion information that each Frequency point comprised in the crosspower spectrum all should be accurately, but in the actual conditions, because the influence of factors such as image border content change can cause the distortion of information in the crosspower spectrum.The reliability of information has determined the accuracy of frequency domain overall motion estimation in the crosspower spectrum, therefore suppresses above-mentioned factor to the interference that crosspower spectrum produced, and improves the reliability of information in the crosspower spectrum, is the key that realizes accurate frequency domain overall motion estimation.

The different frequency point is subjected to the influence degree difference of image border content change, in order to improve the accuracy of overall motion estimation, the accuracy that the present invention is directed to each Frequency point information in the crosspower spectrum is estimated, and suppress wherein to be subject to the Frequency point of interference effect according to the specified accuracy scope, obtain new, comprise the crosspower spectrum of global motion information more accurately.

When each Frequency point information was estimated in to crosspower spectrum, the present invention at first used formula (4), (5) to obtain the maximum interference coefficients R of each Frequency point row, column direction correspondence under the designated precision scope respectively ₁(u), R ₂(v), this coefficient has been described in picture material and has been changed under the most significant situation of interference effect that is produced the mould value of image spectrum and the ratio between the interference components; Utilize formula (6), (7) maximum interference multiplication that the amplitude of each Frequency point in the former frame image spectrum is corresponding with the row, column direction respectively then, obtain the big or small V of the maximum interference component that each Frequency point can overcome under the designated precision scope ₁(u), V ₂(v); Utilize the V of formula (8) at last with each Frequency point ₁(u) and V ₂(minimum value v) and an empirical value compare, and judge whether this Frequency point is interfered easily.Be judged as the Frequency point that is subject to disturb, then need it is suppressed.

$R_1\left(u\right)=\sin (\frac{2\mathrm{\πu}}{m\·n}\·\frac{1}{S})---\left(4\right)$

$R_2\left(v\right)=\sin (\frac{2\mathrm{\πv}}{m\·n}\·\frac{1}{S})---\left(5\right)$

The ranks number of m, n presentation video in formula (4), (5), u, v represent the frequency values of row, column direction respectively,

Expression designated precision scope (as 1/2 pixel, 1/4 pixel, 1/8 pixel).

V ₁(u)＝|F ₁(u，v)|·R ₁(u) (6)

V ₂(v)＝|F ₁(u，v)|·R ₂(v) (7)

In formula (6), (7) | F ₁(u, v) | the frequency spectrum of piece image is in Frequency point (u, the mould value size of v) locating before the expression.

$S_1(u,v)=\left\{\begin{array}{cc}{S}_0(u,v)& \mathrm{if}\min \{V_1\left(u\right),V_2\left(v\right)\}\&GreaterEqual;\mathrm{<figure-callout\; id="0"\; label="Threshold"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">Threshold</figure-callout>}\\ 0& \mathrm{if}\min \{V_1\left(u\right),V_2\left(v\right)\}<\mathrm{Threshold}\end{array}\right.---\left(8\right)$

Constant Threshold is an empirical value in the formula (8), and the presentation video edge content changes the interference components size that causes.By obtaining disturb suppressing experiment under a large amount of different images content change situations, when the value of thresholding Threshold was between 200～400, interference suppressioning effect was better.Can select bigger slightly threshold value at the grain details abundant image, and select less threshold value effect better for the less image of grain details.

Compare with the whole picture element global motion estimation techniques of existing frequency domain, technique scheme can effectively overcome the influence that the image border content change is caused the overall motion estimation accuracy, especially to the adverse effect of follow-up sub picture element global estimation accuracy.

(3) whole picture element global motion vector obtains

Obtain new crosspower spectrum after above original crosspower spectrum function being disturbed inhibition, newer crosspower spectrum is carried out inverse-Fourier transform and obtain distribution of movement function I.Maximal peak point P in the searching moving distribution function ₀, obtain whole picture element global motion vector according to its coordinate position.Formula (9) is whole picture element global motion vector (m ₁, m ₂) computing formula.

(m ₁，m ₂)＝arg?max(I(x，y)) (9)

Further be presented in each sub-steps of calculating the sub picture element global motion vector under the arbitrary accuracy below.Different with traditional frequency domain sub picture element global motion estimation techniques is, among the present invention, calculate and store the problem that burden is increased sharply at adopting conventional method to cause, sub picture element global motion vector calculating method under the arbitrary accuracy is divided into two steps: at first, the maximal peak point P that rapid middle search obtains according to previous step ₀, note the functional value of three coordinate points adjacent respectively with this peak point row, column direction coordinate position; Secondly, according to the peak-peak that obtains and note with peak-peak level, the adjacent several times peak value in vertical direction position, calculate the sub picture element global motion vector under its corresponding arbitrary accuracy respectively.

Concrete implementation step is as follows:

(1) record minor peaks

Referring to shown in Figure 3, make maximal peak point P among the distribution of movement function I ₀Coordinate be (x ₀, y ₀).Record maximal peak point P ₀The functional value of three coordinate points that and arranged on left and right sides is the most contiguous is designated as P respectively ₁, P ₂, P ₃Record maximal peak point P ₀The functional value of three coordinate points that upper and lower both sides are the most contiguous is designated as P respectively ₄, P ₅, P ₆Formula (10), (11), (12), (13) are the formula of choosing of above line, column direction minor peaks.

$\begin{array}{cc}\left\{\begin{array}{c}{\mathrm{<figure-callout\; id="1"\; label="P"\; filenames="C200710099576E00121.png,C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_1=I(x_0,y_0-2)\\ {\mathrm{<figure-callout\; id="2"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_2=I(x_0,y_0-1)\\ {\mathrm{<figure-callout\; id="3"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_3=I(x_0,y_0+1)\end{array}\right.& \mathrm{if\; I}(x_0,y_0-1)>I(x_0,y_0+1)\end{array}---\left(10\right)$

$\begin{array}{cc}\left\{\begin{array}{c}{\mathrm{<figure-callout\; id="1"\; label="P"\; filenames="C200710099576E00121.png,C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_1=I(x_0,y_0+2)\\ P_2=I(x_0,y_0+1)\\ P_3=I(x_0,y_0-1)\end{array}\right.& \mathrm{if\; I}(x_0,y_0-1)\&le;I(x_0,y_0+1)\end{array}---\left(11\right)$

$\begin{array}{cc}\left\{\begin{array}{c}{\mathrm{<figure-callout\; id="4"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_4=I(x_0-2,y_0)\\ P_5=I(x_0-1,y_0)\\ P_6=I(x_0+1,y_0)\end{array}\right.& \mathrm{if\; I}(x_0-1,y_0)>I(x_0+1,y_0)\end{array}---\left(12\right)$

$\begin{array}{cc}\left\{\begin{array}{c}{\mathrm{<figure-callout\; id="4"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_4=I(x_0+2,y_0)\\ {\mathrm{<figure-callout\; id="5"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_5=I(x_0+1,y_0)\\ P_6=I(x_0-1,y_0)\end{array}\right.& \mathrm{if\; I}(x_0-1,y_0)\&le;I(x_0+1,y_0)\end{array}---\left(13\right)$

(2) sub picture element global motion vector computation

According to the peak-peak P that obtains ₀And note with peak-peak level, the adjacent minor peaks P in vertical direction position ₁, P ₂, P ₃And P ₄, P ₅, P ₆, utilize formula (14), (15) to calculate sub picture element global motion vector m on the row, column direction ₁And m ₂

${\mathrm{<figure-callout\; id="1"\; label="m"\; filenames="C200710099576E00121.png,C200710099576E00131.png"\; state="\{\{state\}\}">m</figure-callout>}}_1=\frac{P_2-{\mathrm{<figure-callout\; id="3"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_3+{2P}_1}{{\mathrm{<figure-callout\; id="0"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_0+{\mathrm{<figure-callout\; id="1"\; label="P"\; filenames="C200710099576E00121.png,C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_1+{\mathrm{<figure-callout\; id="2"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_2+P_3}---\left(14\right)$

${\mathrm{<figure-callout\; id="2"\; label="m"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">m</figure-callout>}}_2=\frac{P_5-P_6+{2P}_4}{{\mathrm{<figure-callout\; id="0"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_0+{\mathrm{<figure-callout\; id="4"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_4+{\mathrm{<figure-callout\; id="5"\; label="P"\; filenames="C200710099576E00131.png"\; state="\{\{state\}\}">P</figure-callout>}}_5+P_6}---\left(15\right)$

Of particular note, in the above-mentioned sub picture element global motion vector computation formula, employed minor peaks is three adjacent minor peaks.This is under the condition of Catmull-Rom three rank interpolation methods, obtains by the corresponding relation derivation between motion of contrast images sub picture element global and the spectral change.Catmull-Rom three rank interpolation are sub-pix image interpolation method comparatively commonly used, the better situation of change of represent images texture.But, also can use other ripe interpolation method.Under the prerequisite of using other interpolation method, selected minor peaks also can be other minor peaks.

Above-mentioned sub picture element global motion vector calculating method can be on the basis of the whole picture element global estimation of frequency domain, utilize its middle result, by calculating the sub picture element global motion vector between direct acquisition image, calculating that causes because of image interpolation and the increase of storing burden have been avoided.

Below the method that how to obtain the sub picture element global motion vector under the designated precision is launched detailed explanation.

The calculation procedure of front has calculated the sub picture element global motion vector under the arbitrary accuracy, but sub-pixel precision is not high more good more in actual applications, also needs to take all factors into consideration factors such as effect and computing cost.The general normal sub-pixel precision that adopts has 1/2,1/4 or 1/8 pixel.Therefore, in order to satisfy requirement of actual application, need carry out approximate processing to the sub picture element global motion vector that calculates under the arbitrary accuracy.When specifying sub-pixel precision is 1/2 ^KThe time, can calculate sub picture element global motion vector under institute's precision prescribed according to formula (16), (17).

m ₁′＝round(m ₁×2 ^K)/2 ^K (16)

m ₂′＝round(m ₂×2 ^K)/2 ^K (17)

Wherein round represents to round up.Obtain the sub picture element global motion vector under the designated precision thus.

More than disclosed only be instantiation of the present invention, according to thought provided by the invention, those skilled in the art can think and variation, all should fall within the scope of protection of the present invention.

Claims (3)

1. a frequency domain fast sub picture element global motion estimating method that is used for image stabilization is characterized in that comprising the steps:

(1) the original crosspower spectrum of adjacent two two field pictures in the calculating video sequence;

(2) the maximum interference coefficient of acquisition each Frequency point row, column direction correspondence under the designated precision scope, then that the amplitude of each Frequency point in the former frame image spectrum is corresponding with the row, column direction respectively maximum interference multiplication, obtain the size of the maximum interference component that each Frequency point can overcome under the designated precision scope, minimum value and empirical value in the maximum interference component of each Frequency point are compared, judge whether this Frequency point is the Frequency point that is subject to disturb;

(3) suppress for being judged as the Frequency point that is subject to disturb, obtain to disturb the crosspower spectrum after suppressing;

(4) crosspower spectrum after described interference is suppressed carries out inverse-Fourier transform, generates the distribution of movement function, and the peak-peak in the searching moving distribution function obtains whole picture element global motion vector according to the coordinate of maximal peak point;

(5) judge and note in the described distribution of movement function several times peak value adjacent on the row, column direction, calculate the sub picture element global motion vector according to the corresponding relation of primary and secondary peak Distribution in sub picture element global motion and the distribution of movement function with the peak-peak point coordinates;

(6) the sub picture element global motion vector that calculates is carried out approximate processing, obtain the sub picture element global motion vector under the designated precision.

2. the frequency domain fast sub picture element global motion estimating method that is used for image stabilization as claimed in claim 1 is characterized in that:

It is relevant that the size of described empirical value and the grain details of image are enriched the degree forward, between 200～400.

3. the frequency domain fast sub picture element global motion estimating method that is used for image stabilization as claimed in claim 1 is characterized in that:

Described step (5) is divided into two steps:

At first, according to the peak-peak that search in the step (4) obtains, note respectively in the horizontal direction with vertical direction on the functional value of the most contiguous three coordinate points of maximal peak point;

Secondly, according to the peak-peak that obtains and note in the horizontal direction with vertical direction on the functional value of the most contiguous three coordinate points of maximal peak point, calculate the sub picture element global motion vector under the corresponding arbitrary accuracy respectively.

Images