CN1933554A - Anti-fluttering method and apparatus for hand-held digital camera equipment - Google Patents

Abstract

A method for preventing hand digital video camera device from flutter includes making movement estimation of image sequence, judging whether flutter can be occurred or not and judging whether flutter compensation should be carried out or not, estimating out current frame flutter proof transform coefficient by utilizing each local movement vector of image current frame if flutter is occurred then carrying out flutter proof transform on current frame and combining current frame with reference frame to form conversion frame for carrying out coding continuously.

Description

A kind of anti-fluttering method and device that is used for hand-held digital camera equipment

Technical field

The present invention relates to the hand-held digital camera equipment image processing method, be specifically related to the video anti-fluttering method of hand-held digital camera equipment.The invention still further relates to the video anti-shake apparatus of hand-held digital camera equipment.

Background technology

Progress and development along with science and technology, hand-held digital camera equipment more and more enters people's life, yet because the natural cause of organization of human body, the inevitable light exercise of elbow wrist joint brings shake all can for the result of shooting, in addition because the restriction of self storing of handheld device generally all needs real-time video compression coding.This invention is exactly that to have utilized the motion estimation module of video compression coding to carry out anti-shake, with the anti-shake efficient that combines with video coding and not only reduced anti-shake cost but also improved video compression.

The anti-shake system of present stage mainly is divided into two big classes: the anti-shake and electronic flutter-proof of optics.Optics is anti-shake to be by the shake of the built-in instrument induction camera of camera lens, reaches anti-shake effect by the position of adjusting lens in the camera lens again.The anti-shake technology of optics is generally used in the high-end digital camera and Digital Video.Electronic flutter-proof is to come image is handled by electronically, to alleviate the influence of shake to imaging.At present, electronic flutter-proof mainly contains three kinds of enforcement means, improves ISO photosensitivity/electronic image compression/BSS pattern automatically.The anti-shake principle of CCD (Charge Coupled Device charge coupled cell) is exactly that CCD is placed on the support that can move up and down, detect earlier whether shake is arranged, because use gyrosensor, the detection of shake and other companies are basic identical.Sensor goes out the signals such as direction, speed, amount of movement of shake then, calculating the CCD amount of movement that can be enough to offset shake through handling.Electronic flutter-proof is generally used in the digital camera and Digital Video of low side owing to cheap.

This shows that optics is anti-shake respectively to have pluses and minuses with two kinds of technology of electronic flutter-proof.The anti-shake technology of optics anti-shake effective can make full use of the collection pixel of CCD, but need the lens movement compensation arrangement, so cost is higher relatively.Electronic flutter-proof need not the lens movement compensation arrangement, and is cheap, realizes simply, still reduces the utilance of CCD; Adopt the CCD compensation method then to need extra CCD mobile device, also to a certain degree improved cost.No matter be the anti-shake or electronic flutter-proof of optics, most methods all need the transducer-jitter compensation gyroscope of a motion to come the direction and the intensity of perceive motion, thereby adopt optical means and electronic method to compensate.

Summary of the invention

The purpose of this invention is to provide that a kind of cost is low, CCD (Charge Coupled Device charge coupled cell) utilance height, realize being easy to be used for the anti-fluttering method of hand-held digital camera equipment.Purpose of the present invention also provides a kind of video anti-shake apparatus of hand-held digital camera equipment.

In order to achieve the above object, the present invention takes following scheme: 1) at first image sequence is carried out estimation, shake the judgement of generation then, if the ratio that the non-zero efficient motion-vector of present frame accounts for the total movement vector greater than certain threshold value then think shake has taken place, need be carried out jitter compensation; Otherwise keep original image and do not compensate; When 2) shake took place, promptly each local motion vector of the present frame by image was estimated the affine transformation coefficient of present frame and is carried out affine transformation and the local motion filtering; 3) carry out anti-shake to present frame or affine transformation and be merged into reference frame and proceed coding after the transform frame again.

Wherein, described estimation comprises:

1) motion estimation module is input as i two field picture and i-1 two field picture as the reference image, is output as the motion vector of i two field picture with respect to the i-1 two field picture;

2) method for estimating adopts based on the method for estimating in the video-frequency compression method, is that unit carries out estimation with piece and macro block promptly, obtains the motion vector of each macro block in the image or piece by estimation.

Wherein, the conversion coefficient calculating of described present frame and affine transformation and local motion filtering operation comprises:

＜1〉the vector matrix MV that at first estimation is obtained converts paired pixel coordinate to, the current point coordinates of present frame be (X1, Y1), motion vector be MV (MVx1, MVy1), then the pixel of corresponding former frame be (Xref1 Yref1) has:

Xref1＝X1+MVx1；

Yref1＝Y1+MVy1；

＜2〉secondly find the solution the shake conversion coefficient, when shake takes place camera, can think to be similar between two frame images affine transformation has taken place, thereby have:

Xref1＝a*X1+b*Y1+e；

Yref1＝c*X1+d*Y1+f；

Only rotate with the shake of translation and can be reduced to when not considering the shake of lens zoom when thinking:

Xref1＝a*X1+b*Y1+e；

Yref1＝-b*X1+a*Y1+f；

We can set up matrix equation to the vector matrix that is obtained by estimation in conjunction with the coordinate of this point:

$\left[\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="Xref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xref</figure-callout>}1\\ \mathrm{Yref}1\\ \mathrm{<figure-callout\; id="2"\; label="Xref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Xref</figure-callout>}2\\ \mathrm{Yref}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]=\left[\begin{array}{cc}a& b\\ -b& a\end{array}\right]*\left[\begin{array}{c}X1\\ Y1\\ X2\\ \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{<figure-callout\; id="1"\; label="Xn\; Y"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xn</figure-callout>}& \\ Y1\end{array}\right]+\left[\begin{array}{c}e\\ f\end{array}\right]$

Conversion is as follows:

$\left[\begin{array}{cccc}X1& \mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Y</figure-callout>}1& 1& 0\\ Y1& -\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">X</figure-callout>}1& 0& 1\\ X2& \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2& 1& 0\\ Y2& -\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">X</figure-callout>}2& 0& 1\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \mathrm{Xn}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& -\mathrm{Xn}& 0& 1\end{array}\right]*\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]=\left[\begin{array}{c}\mathrm{Xref}1\\ \mathrm{Yref}1\\ \mathrm{Xref}2\\ \mathrm{Yref}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]$

Can regard following form as:

AX＝b

Wherein:

$A=\left[\begin{array}{cccc}X1& \mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Y</figure-callout>}1& 1& 0\\ Y1& -\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">X</figure-callout>}1& 0& 1\\ X2& \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2& 1& 0\\ Y2& -\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">X</figure-callout>}2& 0& 1\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \mathrm{Xn}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& -\mathrm{Xn}& 0& 1\end{array}\right]$ $b=\left[\begin{array}{c}\mathrm{Xref}1\\ \mathrm{Yref}1\\ \mathrm{Xref}2\\ \mathrm{Yref}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]$ $X=\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]$

Find the solution and obtain (X):

X＝[A ^TA] ^-1A ^Tb

Can obtain conversion coefficient (a b e f)

＜3〉once more, the filtering local motion, the pixel that the conversion coefficient that utilization calculates obtains the motion vector matrix to recomputate obtain (Xrefi ', Yrefi ') with corresponding (Xrefi, Yrefi) relative error, for any one error of coordinate greater than the point of a predetermined number pixel to thinking the position at local motion piece place, thereby remove; Utilize the remaining n ' individual then to carrying out step＜2 again〉in computing obtain relatively accurate global change's coefficient.

Wherein, described motion compensation comprises: utilize global change's coefficient that present frame is carried out conversion, promptly use affine transformation and interpolation with the point transformation in the present frame on new transform frame, excision is positioned at the outer point of image coordinate, with the some polishing on the reference frame, this moment obtains the transform frame of present frame and the error of reference frame has only local motion and eliminated overall shake for the point that does not have.

The anti-shake apparatus of hand-held digital camera equipment, comprise motion estimation unit, coding unit, entropy coding unit, motion compensation units, buffer unit, change over switch, this switch is used for the information of present frame motion estimation result is sent into coding unit, or sends into conversion coefficient and calculate unit, affine transformation unit; Conversion coefficient calculates the unit, is used to find the solution the affine transformation coefficient of present frame, and its end is connected with motion estimation unit by change over switch, and the other end is connected with the affine transformation unit; The affine transformation unit is used to eliminate the shake that present frame may exist, and it connects described coding unit by change over switch; Local motion filtering unit is used for judging whether present frame shake has taken place and eliminated the local motion that the overall situation is shaken, and it connects the affine transformation unit respectively and conversion coefficient calculates the unit.

The anti-shake apparatus of hand-held digital camera equipment, comprise motion estimation unit one, coding unit, entropy coding unit, motion compensation units, buffer unit, also set up a motion estimation unit two, one end of this unit is connected with the affine transformation unit, and the other end is connected with coding unit by change over switch; Conversion coefficient calculates the unit, is used to find the solution the affine transformation coefficient of present frame, and its end is connected with motion estimation unit one, and the other end is connected with the affine transformation unit; The affine transformation unit is used to eliminate the shake that present frame may exist, and it connects the motion estimation unit two of setting up; Local motion filtering unit is used for judging whether present frame shake has taken place and eliminated the local motion that the overall situation is shaken, and it connects the affine transformation unit respectively and conversion coefficient calculates the unit.

1) compared with the prior art, owing to adopt above scheme, the present invention need not external motion sensor, the jitter compensation gyroscope, realize the estimation of shake, the compensation of dither image by digital image processing method fully, efficiently solve translation, the shake that takes place in rotation even the zooming transform process.2) realize simple, CCD utilance height and cheap.3) the present invention can carry out anti-shake processing for hand-held picture pick-up device when encoding, especially to the anti-shake processing of rotation occurring.

Description of drawings

Fig. 1 is the structural representation that the present invention has only a motion estimation unit;

Fig. 2 is the structural representation of two motion estimation unit of the present invention;

Relation and motion vector schematic diagram between Fig. 3 present frame that is the present invention when shake takes place and the reference frame;

Fig. 4 is the structural representation after present frame affine transformation of the present invention and the interpolation;

Fig. 5 is the transform frame schematic diagram that present frame after the conversion of the present invention and reference frame are merged into;

Fig. 6 is a reference frame schematic diagram of the present invention;

Fig. 7 be the present invention shake generation present frame wherein the personage advance forward and have the local motion schematic diagram;

Fig. 8 is the transform frame schematic diagram after the present invention merges.

Among the figure: K1, K2, change over switch; 1, the position 1 of K1, K2, K3; 2, the position 2 of K1, K2, K3; 3, motion estimation unit; 4, conversion coefficient calculates the unit; 5, local motion filtering unit; 6, affine unit; 7, entropy coding unit; 8, coding unit; 9, motion compensation units; 10, buffer unit; The motion estimation unit of 11, setting up.

Embodiment

Following examples are used to illustrate the present invention, but are not used for limiting the scope of the invention.

As shown in Figure 1, the anti-fluttering method that the present invention proposes need not to add the motion compensation transducer and obtains motion state, but introduced the method for estimating of video coding technique, wherein, the shake of image is equivalent to take place affine transformation, thereby calculates the conversion coefficient of the overall situation shake of every frame by method for estimating.Promptly pass through each local motion vector (MV of present frame ₁, MV ₂... MV _N), the affine transformation coefficient of estimation present frame (a, b, c, d, e, f).Present frame is carried out anti-shake (affine) conversion and is merged into transform frame with reference frame proceeding coding afterwards again.

Embodiment one: as shown in Figure 1, this device adopts increases conversion coefficient calculating 6, one local motion filterings unit 5,4, one affine transformation unit, unit and two change over switch K1, K2 in video encoder.When anti-shake apparatus was worked, two K switch 1, K2 placed position 1 earlier, and earlier carry out estimation to present frame this moment.The motion vector that the 4 pairs of estimation in conversion coefficient calculating unit obtain carries out the calculating of affine transformation coefficient.Affine transformation unit 6 utilizes the conversion coefficient that obtains to carry out affine transformation.The local motion that exists in the shake of the local motion filtering unit 5 filterings overall situation.Dotted line is represented what change over switch K1, K2 were controlled by local motion filtering unit among Fig. 1.

Judge that in the result of motion estimation unit 3 ratio that efficient motion-vector accounts for (not comprising 0 motion vector) the total movement vector less than certain threshold value, then thinks not have overall situation shake or the excessive scope that has exceeded shake of amount of exercise.Anti-shake processing is no longer carried out in the follow-up work of encoding normally etc. this moment.This threshold value generally chooses 2/3.When not having overall situation shake, two change over switch K1, K2 are placed position 2.In this case the result of estimation is directly carried out other coding works of video encoder.

When motion estimation unit is exported ratio that non-0 efficient motion-vector accounts for whole motion vector number greater than certain threshold value, think to have overall situation shake (this threshold value generally chooses 2/3), when do not exist in the local motion filtering unit error of coordinate greater than the point of a predetermined number pixel to the time think and do not have local motion.Then directly carry out present frame to be merged into transform frame with reference frame after the affine transformation this moment, then two change over switch K1, K2 placed position 2 to utilize transform frame to carry out the complete video coding process of video encoder.

Then the motion vector behind the filtering local motion carries out the calculating of conversion coefficient when having overall situation shake and local motion is arranged, the conversion coefficient that the present frame utilization is tried to achieve carries out being merged into transform frame with reference frame after the affine transformation then, then two K switch 1, K2 is placed position 2 to utilize transform frame to carry out the complete video coding process of video encoder.

Wherein, described motion estimation unit 3 comprises:

Motion estimation unit 3 is input as i two field picture and i-1 two field picture as the reference image.Be output as the motion vector of i two field picture with respect to the i-1 two field picture.

Method for estimating can adopt based on the method for estimating in the video-frequency compression method, is that unit carries out estimation with piece and macro block.At this moment can obtain the motion vector of each macro block in the image or piece by estimation.

The present invention adopt MPEG and H.26X the motion estimation unit in the video encoding standard be applied directly to motion estimation unit 3 of the present invention.These methods can be easy to realize, directly use.

Because present MPEG and H.26X wait motion estimation unit is all arranged in the video encoding standard, the present invention can directly adopt the motion estimation unit in the video encoder that meets these standards.So the present invention can be fused in the video encoder easily.

The function that described conversion coefficient calculates unit 4 comprises:

＜1〉the vector matrix MV that at first estimation is obtained converts paired pixel coordinate to, the current point coordinates of present frame be (X1, Y1), motion vector be MV (MVx1, MVy1), then the pixel of corresponding former frame be (Xref1 Yref1) has:

Xref1＝X1+MVx1；

Yref1＝Y1+MVy1；

＜2〉secondly find the solution the shake conversion coefficient, when shake takes place camera, can think to be similar between two frame images affine transformation has taken place, thereby have:

Xref1＝a*X1+b*Y1+e；

Yref1＝c*X1+d*Y1+f；

Only rotate with the shake of translation and can be reduced to when not considering the shake of lens zoom when thinking:

Xref1＝a*X1+b*Y1+e；

Yref1＝-b*X1+a*Y1+f；

We can set up matrix equation to the vector matrix that is obtained by estimation in conjunction with the coordinate of this point:

$\left[\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="Xref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xref</figure-callout>}1\\ \mathrm{<figure-callout\; id="1"\; label="Yref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Yref</figure-callout>}1\\ \mathrm{<figure-callout\; id="2"\; label="Xref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Xref</figure-callout>}2\\ \mathrm{<figure-callout\; id="2"\; label="Yref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Yref</figure-callout>}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]=\left[\begin{array}{cc}a& b\\ -b& a\end{array}\right]*\left[\begin{array}{c}X1\\ Y1\\ X2\\ \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2\\ \&CenterDot;\\ \&CenterDot;\\ X2\\ \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2\end{array}\right]+\left[\begin{array}{c}e\\ f\end{array}\right]$

Conversion is as follows:

$\left[\begin{array}{cccc}X1& \mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Y</figure-callout>}1& 1& 0\\ Y1& -\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">X</figure-callout>}1& 0& 1\\ X2& \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2& 1& 0\\ Y2& -\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">X</figure-callout>}2& 0& 1\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \mathrm{<figure-callout\; id="1"\; label="Xn\; Yn"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xn</figure-callout>}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& \mathrm{Xn}& 0& 1\end{array}\right]*\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]=\left[\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="Xref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xref</figure-callout>}1\\ \mathrm{<figure-callout\; id="1"\; label="Yref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Yref</figure-callout>}1\\ \mathrm{<figure-callout\; id="2"\; label="Xref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Xref</figure-callout>}2\\ \mathrm{<figure-callout\; id="2"\; label="Yref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Yref</figure-callout>}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]$

Can regard following form as:

AX＝b

Wherein:

$A=\left[\begin{array}{cccc}X1& \mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Y</figure-callout>}1& 1& 0\\ Y1& -\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">X</figure-callout>}1& 0& 1\\ X2& \mathrm{<figure-callout\; id="2"\; label="Y"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Y</figure-callout>}2& 1& 0\\ Y2& -\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">X</figure-callout>}2& 0& 1\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \&CenterDot;& \&CenterDot;& \&CenterDot;& \&CenterDot;\\ \mathrm{<figure-callout\; id="1"\; label="Xn\; Yn"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xn</figure-callout>}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& -\mathrm{Xn}& 0& 1\end{array}\right]$ $b=\left[\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="Xref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Xref</figure-callout>}1\\ \mathrm{<figure-callout\; id="1"\; label="Yref"\; filenames="A20051013593200181.png,A20051013593200191.png"\; state="\{\{state\}\}">Yref</figure-callout>}1\\ \mathrm{<figure-callout\; id="2"\; label="Xref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Xref</figure-callout>}2\\ \mathrm{<figure-callout\; id="2"\; label="Yref"\; filenames="A20051013593200181.png"\; state="\{\{state\}\}">Yref</figure-callout>}2\\ \&CenterDot;\\ \&CenterDot;\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]$ $X=\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]$

Find the solution and obtain (X):

X＝[A ^TA] ^-1A ^Tb

Can obtain conversion coefficient (a b e f)

The function of described affine transformation unit 6 comprises:

1) when motion estimation unit is exported ratio that non-0 efficient motion-vector accounts for whole motion vector number greater than certain threshold value, thinks to have overall situation shake (this threshold value generally chooses 2/3).When existing under the overall jitter conditions, utilize pixel that conversion coefficient that conversion coefficient calculates unit 4 outputs obtains the motion vector matrix that the calculating of carrying out affine transformation (Xrefi ', Yrefi ') is outputed to local motion filtering unit 5.

2) referring to Fig. 3-Fig. 5, as seen from Figure 3, when taking place, shake both existed skew on level and the vertical direction also to have the deviation of rotation between present frame and the reference frame, and we can regard it as, and affine transformation has taken place.Thereby utilize corresponding motion vector to ask for the conversion coefficient of global motion.Utilize global change's coefficient that present frame is carried out conversion, promptly with affine transformation and interpolation the point in the present frame is carried out the conversion (see figure 4), the method for described interpolation can be with neighbor interpolation method, linear interpolation method, cubic B-spline interpolation method etc.; Excision is positioned at the outer point of image coordinate, generates the transform frame (see figure 5) for the point that does not have with the some polishing on the reference frame, carries out the video coding operation.Equally, Fig. 6-Fig. 8 has described the jitter elimination when having local motion, wherein Fig. 6 represents reference frame, Fig. 7 is for existing the present frame of overall situation shake and local motion (pedestrian moves), and Fig. 8 is that excision is positioned at the outer point of image coordinate and merges the transform frame that obtains with reference frame after utilizing the conversion coefficient that obtains behind the filtering local motion to carry out affine transformation and interpolation.Obtaining the transform frame of present frame and the error of reference frame this moment has only local motion and has eliminated a large amount of overall situation shakes.So because very little corresponding the diminishing of residual error of proceeding to encode and obtaining of error of transform frame and reference frame, thereby also improved compression efficiency.

The function of described local motion filtering unit 5 is as follows:

1) when existing under the overall jitter conditions, utilize affine transformation unit 6 output (Xrefi ', Yrefi ') with corresponding (Xrefi, Yrefi) relative error, because the unit of coordinate is exactly a pixel, so for any one error of coordinate greater than the point of a Δ number pixel to thinking the position at local motion piece place, thereby remove.Utilize the remaining individual point of n ' right then, carry out above-mentioned conversion coefficient again and calculate unit 4 steps＜2〉in computing obtain relatively accurate global change's coefficient.Wherein Δ is chosen minimum movement and is estimated 1/2 of block size, and the motion estimation block of for example having selected 8*8 for use then Δ is chosen for 8*1/2=4.

2) when non-0 efficient motion-vector behind the filtering local motion vector accounts for the ratio of non-0 efficient motion-vector that total movement estimates output less than certain threshold value, then think not have overall situation shake or the excessive scope that has exceeded shake of amount of exercise.Controlling two change over switch K1, K2 this moment places position 2 follow-up work such as encode normally no longer to carry out anti-shake processing.This threshold value generally chooses 2/3.

In the device shown in Figure 1 owing to have only a motion estimation unit, anti-shake processing and video coding all need to use, though still need two switches to switch so saved device, in order to better meet in fast processing, we have proposed device as shown in Figure 2, and this device adopts and set up 11, one conversion coefficients calculating of motion estimation unit unit 4 before video encoder, 6, one local motion filterings unit, an affine transformation unit 5 and a change over switch K3.Though increased a motion estimation unit 11 than embodiment one this moment, reduced by a switch.Described motion estimation unit 11 1 ends of setting up are connected with affine transformation unit 6, and the other end is connected with coding unit by change over switch K3; Conversion coefficient calculates unit 4, is used to find the solution the affine transformation coefficient of present frame, and its end is connected with motion estimation unit 3, and the other end is connected with affine transformation unit 6; Affine transformation unit 6 is used to eliminate the shake that present frame may exist, and it links to each other with the motion estimation unit 11 of setting up; Local motion filtering unit 5 is used for judging whether present frame shake has taken place and eliminated the local motion that the overall situation is shaken, and it calculates unit 4 with affine transformation unit 6 and conversion coefficient respectively and links to each other.Dotted line is represented what change over switch K3 was controlled by local motion filtering unit among Fig. 2.

When anti-shake apparatus was worked, K switch 3 placed position 2 earlier, and earlier carry out estimation to present frame this moment.The motion vector that the 4 pairs of estimation in conversion coefficient calculating unit obtain carries out the calculating of affine transformation coefficient.Affine transformation unit 6 utilizes the conversion coefficient that obtains to carry out affine transformation.The local motion that exists in the shake of the local motion filtering unit 5 filterings overall situation.

Judge that in the result of the motion estimation unit 11 of setting up ratio that efficient motion-vector accounts for (not comprising 0 motion vector) the total movement vector less than certain threshold value, then thinks not have overall situation shake or the excessive scope that has exceeded shake of amount of exercise.Anti-shake processing is no longer carried out in the follow-up work of encoding normally etc. this moment.This threshold value generally chooses 2/3.When not having overall situation shake, K switch 3 is placed position 1.In this case the result of estimation is directly carried out other coding works of video encoder.

When motion estimation unit is exported ratio that non-0 efficient motion-vector accounts for whole motion vector number greater than certain threshold value, think to have overall situation shake (this threshold value generally chooses 2/3), when do not exist in the local motion filtering unit error of coordinate greater than the point of a predetermined number pixel to the time think and do not have local motion.Then directly carry out present frame to be merged into transform frame with reference frame after the affine transformation this moment, and K switch 3 remains on position 2, utilizes transform frame to carry out video coding process complete in the video encoder then.

Then the motion vector behind the filtering local motion carries out the calculating of conversion coefficient when having overall situation shake and local motion is arranged, the conversion coefficient that the present frame utilization is tried to achieve carries out being merged into transform frame with reference frame after the affine transformation then, K switch 3 remains on position 2, utilizes transform frame to carry out video coding process complete in the video encoder.

Wherein, the function of described motion estimation unit 3 is identical with embodiment one; The function of the described motion estimation unit of setting up 11 is identical with motion estimation unit 3 among the embodiment one.

Wherein, to calculate the function of unit 4 identical with embodiment one for described conversion coefficient.

The function of described affine transformation unit 6 is identical with embodiment one.

The function of described local motion filtering unit 5 is as follows:

1) utilize affine transformation unit 6 output (Xrefi ', Yrefi ') with corresponding (Xrefi, Yrefi) relative error, because the unit of coordinate is exactly a pixel, so for any one error of coordinate greater than the point of a Δ number pixel to thinking the position at local motion piece place, thereby remove.Utilize the remaining individual point of n ' right then, carry out above-mentioned conversion coefficient again and calculate unit step＜2〉in computing obtain relatively accurate global change's coefficient.Wherein Δ is chosen minimum movement and is estimated 1/2 of block size, and the motion estimation block of for example having selected 8*8 for use then Δ is chosen for 8*1/2=4.

2) when non-0 efficient motion-vector behind the filtering local motion vector accounts for the ratio of non-0 efficient motion-vector that total movement estimates output less than certain threshold value, then think not have overall situation shake or the excessive scope that has exceeded shake of amount of exercise.This moment, control switch K3 placed position 1 follow-up work such as encode normally no longer to carry out anti-shake processing.This threshold value generally chooses 2/3.

Claims (6)

1, a kind of anti-fluttering method that is used for hand-held digital camera equipment, it is characterized in that comprising: 1) at first image sequence is carried out estimation, shake the judgement of generation then, if the ratio that the non-zero efficient motion-vector of present frame accounts for the total movement vector greater than certain threshold value then think shake has taken place, need be carried out jitter compensation; Otherwise keep original image and do not compensate; When 2) shake took place, promptly each local motion vector of the present frame by image was estimated the affine transformation coefficient of present frame and is carried out affine transformation and the local motion filtering; 3) carry out anti-shake to present frame or affine transformation and be merged into reference frame and proceed coding after the transform frame again.

2, a kind of anti-fluttering method that is used for hand-held digital camera equipment as claimed in claim 1 is characterized in that:

Described method for estimating adopts based on the method for estimating in the video-frequency compression method, is that unit carries out estimation with piece and macro block promptly, obtains the motion vector of each macro block in the image or piece by estimation.

3, a kind of anti-fluttering method that is used for hand-held digital camera equipment as claimed in claim 1 is characterized in that the conversion coefficient calculating of described present frame and affine transformation and local motion filtering operation comprises:

＜1〉the vector matrix MV that at first estimation is obtained converts paired pixel coordinate to, the current point coordinates of present frame be (X1, Y1), motion vector be MV (MVx1, MVy1), then the pixel of corresponding former frame be (Xref1 Yref1) has:

Xref1＝X1+MVx1；

Yref1＝Y1+MVy1；

＜2〉secondly find the solution the shake conversion coefficient, when shake takes place camera, can think to be similar between two frame images affine transformation has taken place, thereby have:

Xref1＝a*X1+b*Y1+e；

Yref1＝c*X1+d*Y1+f；

Only rotate with the shake of translation and can be reduced to when not considering the shake of lens zoom when thinking:

Xref1＝a*X1+b*Y1+e；

Yref1＝-b*X1+a*Y1+f；

We can set up matrix equation to the vector matrix that is obtained by estimation in conjunction with the coordinate of this point:

$\left[\begin{array}{c}\mathrm{Xref}1\\ \mathrm{Yref}1\\ \mathrm{Xref}2\\ \mathrm{Yref}2\\ .\\ .\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]=\left[\begin{array}{cc}a& b\\ -b& a\end{array}\right]*\left[\begin{array}{c}X1\\ Y1\\ X2\\ Y2\\ .\\ .\\ \mathrm{Xn}\\ Y1\end{array}\right]+\left[\begin{array}{c}e\\ f\end{array}\right]$

Conversion is as follows:

$\left[\begin{array}{cccc}X1& Y1& 1& 0\\ Y1& -X1& 0& 1\\ X2& Y2& 1& 0\\ Y2& -X2& 0& 1\\ .& .& .& .\\ .& .& .& .\\ \mathrm{Xn}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& -\mathrm{Xn}& 0& 1\end{array}\right]*\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]=\left[\begin{array}{c}\mathrm{Xref}1\\ \mathrm{Yref}1\\ \mathrm{Xref}2\\ \mathrm{Yref}2\\ .\\ .\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]$

Can regard following form as:

AX＝b

Wherein:

$A=\left[\begin{array}{cccc}X1& Y1& 1& 0\\ Y1& -X1& 0& 1\\ X2& Y2& 1& 0\\ Y2& -X2& 0& 1\\ .& .& .& .\\ .& .& .& .\\ \mathrm{Xn}& \mathrm{Yn}& 1& 0\\ \mathrm{Yn}& -\mathrm{Xn}& 0& 1\end{array}\right]b=\left[\begin{array}{c}\mathrm{Xref}1\\ \mathrm{Yref}1\\ \mathrm{Xref}2\\ \mathrm{Yref}2\\ .\\ .\\ \mathrm{Xrefn}\\ \mathrm{Yrefn}\end{array}\right]X=\left[\begin{array}{c}a\\ b\\ e\\ f\end{array}\right]$

Find the solution and obtain (X):

X＝[A ^TA] ^-1A ^Tb

Can obtain conversion coefficient (a b e f)

＜3〉once more, the filtering local motion, the pixel that the conversion coefficient that utilization calculates obtains the motion vector matrix to recomputate obtain (Xrefi ', Yrefi ') with corresponding (Xrefi, Yrefi) relative error, for any one error of coordinate greater than the point of a predetermined number pixel to thinking the position at local motion piece place, thereby remove; Utilize the remaining n ' individual then to carrying out step＜2 again〉in computing obtain relatively accurate global change's coefficient.

＜4〉last, judge when the non-0 efficient motion-vector number behind the filtering local motion vector accounts for ratio that total movement estimates the non-0 efficient motion-vector number exported less than certain threshold value, then think not have overall situation shake or the excessive scope that has exceeded shake of amount of exercise, directly utilize this moment the output result of the motion estimation module follow-up work of encoding normally no longer to carry out anti-shake processing.

4, a kind of anti-fluttering method that is used for hand-held digital camera equipment as claimed in claim 1, it is characterized in that described motion compensation comprises: utilize global change's coefficient that present frame is carried out conversion, promptly use affine transformation and interpolation with the point transformation in the present frame on new transform frame, excision is positioned at the outer point of image coordinate,, obtain the transform frame of present frame and the error of reference frame this moment and have only the error seldom of local motion generation and eliminated the Errors Catastrophic that overall shake produces with the some polishing on the reference frame for the point that does not have.

5, a kind of anti-shake apparatus of hand-held digital camera equipment, comprise motion estimation unit, coding unit, entropy coding unit, motion compensation units, buffer unit, it is characterized in that also comprising: change over switch, conversion coefficient calculate unit, affine transformation unit, local motion filtering unit, wherein, change over switch, be used for the information of present frame motion estimation result is sent into coding unit, or send into conversion coefficient and calculate unit, affine transformation unit; Conversion coefficient calculates the unit, is used to find the solution the affine transformation coefficient of present frame, and its end is connected with motion estimation unit by change over switch, and the other end is connected with the affine transformation unit; The affine transformation unit is used to eliminate the shake that present frame may exist, and it connects described coding unit by change over switch; Local motion filtering unit is used for judging whether present frame shake has taken place and eliminated the local motion that the overall situation is shaken, and it connects the affine transformation unit respectively and conversion coefficient calculates the unit.

6, a kind of anti-shake apparatus of hand-held digital camera equipment, comprise motion estimation unit one, coding unit, entropy coding unit, motion compensation units, buffer unit, it is characterized in that also comprising: set up a motion estimation unit two, one end of this unit is connected with the affine transformation unit, and the other end is connected with coding unit by change over switch; Conversion coefficient calculates the unit, is used to find the solution the affine transformation coefficient of present frame, and its end is connected with motion estimation unit one, and the other end is connected with the affine transformation unit; The affine transformation unit is used to eliminate the shake that present frame may exist, and it connects the motion estimation unit of setting up; Local motion filtering unit is used for judging whether present frame shake has taken place and eliminated the local motion that the overall situation is shaken, and it connects the affine transformation unit respectively and conversion coefficient calculates the unit.

Images