KR100856474B1 - Stabilizing method for shaking video image - Google Patents

Abstract

A method for correcting a shaking video image is provided to correct a shaking video image by using a stabilized correction parameter which is obtained by a speedy estimation. A method for correcting a shaking vide image, includes the step of forming a phase correlation space by obtaining a phase correlation for a reference frame and a present frame of a reference vide image by using a phase correlation method. The phase correlation is based on a shift property of Fourier transform. The method further includes the steps of selecting a peculiar pixel from the reference video image(S2), calculating a motion vector for the selected peculiar pixel by using an optical flow method, obtaining an average direction and an average size of vectors from the calculated motion vector of the peculiar pixel(S4), establishing a search area in the phase correlation space which is processed for a single area from the obtained average direction information and the size information in relation to the motion vector(S5), obtaining a maximum peak value from the established search area and determining its location as a correction parameter(S7), and applying the determined correction parameter to the current frame and outputting a stabilized frame(S8).

Description

Stabilizing method for shaking video image}

The present invention relates to an image shake correction method, and more particularly, to an image stabilization method for an image shake phenomenon due to an external environment and a shake phenomenon of an image having a long zoom.

Recently, with the development of digital image processing technology, various methods for the digital image stabilization (DIS) system have been studied. Among them, the vibration of the image caused by the unstable camera fixing device or the vibration of the hand shake, the amplification of amplification when the remote object is enlarged by optical or digital zoom, and the shaking phenomenon that occurs in an unstable environment such as the inside of a vehicle It is one of the important research topics for digital image stabilization as a major factor that degrades the quality of video in both subjective and objective aspects.

In particular, for outdoor video surveillance systems such as outdoor surveillance systems or road traffic monitoring, the cameras are secured to building walls, overpasses, bridges or dedicated supports. However, the camera fixing device may be shaken by various external situations. In the external situation applied to the camera fixing device, the shaking caused by severe wind is the main cause, and especially in the case of bridges, a large amount of traffic is the main cause. As a result, it will have a characteristic that is always loose.

In addition, in the case of a camera having a high magnification zoom in lens, aperiodic fine shaking may occur even in an environment in which a remote image is monitored in real time at a high magnification. Vibration in the remote monitoring environment not only causes visual fatigue of the monitor, but also interferes with stable monitoring of the target target.

Conventional correction techniques have realized shake detection by an angular velocity sensor by control of a mechanical imaging optical system. This method is unsuitable for the trend of products requiring small size and light weight due to the increase of correction mechanism.

On the other hand, the image stabilization system detects the amount of vibration only by digital signal processing of the image, identifies the image characteristic, and compensates for the vibration phenomenon by the LSI that adaptively controls the system. Phase correlation and optical flow are used as representative shake correction methods currently used.

The phase correlation method shows excellent performance in estimating the rapidly changing discontinuous shaking and the amount of transition of the whole frame, while the optical flow method shows excellent performance in the continuous changing slowly and pixel-by-pixel motion.

In general, a shake correction algorithm obtains a shake correction parameter by estimating frame moving vectors (FMVs) from local motion vectors (LMVs) between two consecutive frames. Motion estimation and compensation techniques are also the basic techniques of video compression, and in the case of motion estimation, a block matching algorithm (BMA), a gradient matching algorithm (GMA), and phase correlation are used. Here, the phase correlation method is suitable for estimating FMV, while the optical flow method is suitable for estimating LMV.

A fast Fourier transform algorithm is used to obtain a stabilized image using phase correlation. This algorithm obtains the maximum power of peak value from the spatial domain result of fast inverse Fourier transform by taking a certain area of 2 ^N × 2 ^N in the original image and obtaining the fast Fourier transformed spectrum and the spectrum for the reference image. The searched position is determined by the correction parameter corresponding to the transition component.

However, when the phase correlation is applied by taking only a single region, misinterpretation occurs intermittently. The reason for the erroneous estimation is that, unlike the phase correlation theory, it is difficult to obtain a single spike peak in the resultant space of phase correlation due to noise, etc. in the real image. Incorrect image correction due to such a false estimation does not meet the purpose of stabilization of a shake image to prevent visual fatigue of the shake of the image and requires a precise estimation method.

In order to solve this problem, a method of determining a position of the largest peak value in each candidate peak obtained from each phase correlation space by taking a plurality of candidate regions instead of a single region is generally used.

However, when applying this method, stable estimation is realized, rather than taking a single image and analyzing it, but it is not a complete countermeasure against false estimation. Furthermore, this method has to search for the maximum value of the peak value in the entire spatial region formed by performing Fourier and Inverse Fourier transforms from multiple regions taken from the original image, which is a delay factor in a system requiring real time processing.

Even when the optical flow method is used, a large processing time is required when calculating the optical flow for all image pixels, which is not suitable for real time processing. The technical problem of the optical flow method is the problem of how to estimate the optical flow as quickly as possible.

Accordingly, an object of the present invention is to provide a method for correcting image shake by using a stable correction parameter obtained by fast estimation.

The above object is obtained by forming a phase correlation space by obtaining a phase correlation between a reference frame and a current frame of a reference image by a phase correlation method, wherein the phase correlation is based on a shift property of a Fourier transform. ; Selecting a singular pixel from the reference image; Calculating a motion vector for the selected singular pixel using optical flow method; Obtaining an average direction of the vectors and an average size of the vectors from the calculated motion vectors of the singular pixels; Setting a search region in the phase correlation space processed for a single region from the obtained average direction information and size information with respect to the motion vector; Determining a maximum peak value in the set search region and determining a position as a correction parameter; And outputting the stabilized frame by applying the determined correction parameter to the current frame.

Preferably, the method of selecting a singular pixel comprises: extracting edge information of the reference image and setting the value as a mask; And arranging the amount of change in pixel intensity of the current image with respect to the mask area in order of size, and selecting all pixels having the brightest value among pixels having the same intensity change among the results.

Further, preferably, the selection of the singular pixel can be selected only in the first frame and only when the reference frame is changed.

As described above, according to the present invention, there are various advantages.

First, the large shake correction is achieved by reducing the computation amount by mixing the phase correlation method suitable for estimating the motion vector (FMV) of the entire frame and the optical flow method suitable for estimating the motion vector (LMV) of the local pixel for image stabilization. While following well, it is effective for accurate following of fine correction of small shake. Therefore, it can be used for image stabilization of fence guard video surveillance system of armistice lines, military, police facilities, and far magnified images such as coast guards.

Compared with the conventional stabilization using only the phase correlation method, it can be implemented with a calculation amount of 1/10 practically.

In addition, since the camera or lens that is firmly fixed is very robust and its shake is very fine, it is difficult to detect by any physical means, but even in this case, the image that is zoomed in over 1000 times is still shaken. According to the present invention, the image stabilization of the image is a powerful solution to the method of the present invention by digital signal processing.

Hereinafter, with reference to the accompanying drawings will be described an embodiment of the present invention.

In general, camera shake correction uses a hardware-dependent method using a sensor as a seat. However, in the present invention, in order to stabilize a video image, an image caused by camera shake due to an external cause or a remote zoom image may be shaken. It is processed by software through image processing technique. Specifically, the present invention provides a method for estimating a local pixel motion vector (LMV) using an optical flow method including a block matching algorithm (BMA) and a gradient matching algorithm (GMA) and a frame motion vector (FMV) using phase correlation. Hybrid method using a method of estimating the fast FMV estimate and correct the image with the obtained correction parameters.

Accordingly, a stabilized output video having a constant focus is obtained as if the camera is not shaken at a certain target in the output video.

1 is a flowchart illustrating a correction method according to the present invention.

First, a phase correlation space is formed by obtaining a phase correlation between a reference frame, which is a reference image for image stabilization, and a current frame by a phase correlation method (step S1). In this process, the multi-domain processing that is prepared and applied as a countermeasure against the conventional phase correlation method is unnecessary, and only a single domain processing is performed.

Correction parameter estimation by phase correlation is an algorithm based on the shift property of the Fourier transform. The transition characteristic of the Fourier transform is that the transition of the two functions on the coordinates is transformed into a linear phase difference in the Fourier region.

That is, if f _k and f _{k +1} are two-dimensional images shifted by t ( t _x , t _y ) on the two-dimensional plane, f _k ( x , y ) and f _{k +1} ( x , y ) are represented by equation (1). Has the same relationship as

By the Fourier transition characteristic, the Fourier transform of these two functions is represented by equation (2).

Since F _k ( u + v ) and F _{k + 1} ( u , v ) are complex numbers, we obtain a cross power spectrum (CPS) to separate the transition components of the image.

Where F ^* _{k + 1} ( u , v ) is the conjugate complex number of F _{k +1} ( u , v ).

Inverse Fourier Transformation of Equation (3) to obtain only the transition components yields a transition value that makes an impulse on t ( t _x , t _y ) on the two-dimensional coordinate axis as in Equation (4).

Subsequently, a specific pixel is selected in the reference image (step S2).

The method of selecting singular pixels extracts edge information of the reference image and sets the value as a mask. Then, the amount of change in pixel intensity of the current image is aligned with respect to the mask area, and all pixels having the brightest value among the pixels having the same intensity change among the resulting alignment values are selected to form a singular pixel.

This singular pixel selection procedure is not a procedure performed every frame, but is selected only in the first frame and is performed only when the reference frame is changed. In addition, unlike the representative point matching method (for example, Korean Patent No. 94465) based on the existing predetermined arrangement, the arrangement is not determined and different singular pixels are selected according to the image content of the reference frame. In this process, the edge information is used only to construct a mask for forming a singular pixel. This method is more robust to position change in terms of reliability than the motion vector detection method from the existing edge information.

Next, as shown in FIG. 2, the motion vector for the selected singular pixel is quickly obtained using the optical flow method (step S3).

A method of estimating a motion vector by applying an optical flow method in a moving picture may be represented by movement of a luminance pattern formed by a moving object. The optical flow defines a square sum of brightness in a fixed size window between a current frame and a previous frame as a matching criterion for luminance, and the displacement between frames is a method of estimating the flow of luminance in such a manner that the square sum obtained is the minimum. In general, the optical flow calculation method proposed by Lucas and Kanade is used a lot.

Specifically, in this method, the luminance of a point ( x, y ) of an image at time t is called I ( x, y, t ), and there are as many transitions as δx , δy , δt at time t + δ t . If so, the luminance is represented by I ( x + δx , y + δy , t + δt ) and has the same relationship as in Equation (5).

Assuming that the change in intensity I ( x, y, t ) is very small for continuous images, it can be expressed by the following equation (6) having a Taylor series.

Where H.O.T. Since is a very small value of the higher order term and can be ignored,

이 된다.

Becomes

, v =

, I _x =

, I _y =

, I _t =

이며 u와 v는 광류 벡터 구성성분들로 속도벡터(velocity vector)에 해당하며, 식(7)과 같이 표현된다.Where u =

, v =

, I _x =

, I _y =

, I _t =

U and v are the components of the light flow vector, corresponding to the velocity vector, and are represented by Eq. (7).

Lucas and Kanade calculated u and v on the assumption that the motion vectors for the pixels in small region B were the same.

That is, the square error E of the optical flow equation for the region B is defined as Equation (8),

The partial differential equations for x and y in equation (8) are as shown in equations (9) and (10).

Computing equations (9) and (10) with a 2x2 matrix like equation (11) yields u and v .

When the 2x2 determinant of Equation (10) is interpreted as a linear system, the flow of light flow appears by representing the velocity vectors u and v on the x and y axes for the pixels moved by the 2x2 matrix.

3 is a block diagram of an optical flow model proposed by Lucas and Kanade to obtain an optical flow.

In this model, pre-processing is performed using a Gaussian filter for smoothing preprocessing to reduce errors that may occur during subsequent gradient calculation of the input video image. The smoothed input image is extended in three dimensions with respect to the time and space axes to perform a gradient operation.

FIG. 4 illustrates an example of an operation of inclination with respect to the I _x axis during the operation of inclination with respect to the I _x axis, the I _y axis, and the I _t axis in three dimensions. In addition, FIG. 5 shows an example of an inclination process with respect to the I _y axis, and FIG. 6 shows an example of an inclination process with respect to the I _t axis.

The gradient process uses a 1x5 kernel like this:

1x5 kernel = {-1 8 0 -8 1}

By setting the middle frame of the input frame, we use the 0.75 proposed by Lucas and Kanade as weights for the previous and subsequent frames. In this way I _x , I _y, is rescued by calculating a three-dimensional gradient of _{I t I x, I y,} I t, I 2 x, I 2 y, I x I y, t and I _{t y} I.

Next, referring to FIG. 7, the search area is determined by obtaining the average direction of the vectors and the average size of the vectors from the motion vectors of the singular pixels (indicated by the red dots in FIG. 7) calculated in step S3 (steps S4 and S5). ).

Then, the search area is set by the size obtained in the average direction obtained from the center of the phase-correlated space processed for the single area in step 1 from the obtained average direction information and size information. At this time, the size of the search area is set in proportion to the obtained size information. This method is proposed as a method of reducing the search area for quick searching (step S6).

Then, the maximum peak value is obtained in the set search area and the position is determined as the correction parameter (step S7).

The maximum peak value in the search region represents the amount of transition due to the shaking of the current video image frame compared to the reference video image.

Subsequently, it is output as a stabilized frame by applying the correction parameter determined to the current frame (step S8).

That is, in all cases, the transition of the image position due to camera shake occurs in the same size up, down, left, and right in all pixels, and thus the correction parameter is applied to the entire pixel of the corresponding image frame to compensate. The shake-free image placed at the same position as the frame is reconstructed.

As described above, according to the hybrid method using the phase correlation and the optical flow method proposed in the present invention, it is not necessary to take multiple regions and analyze them as a countermeasure for obtaining a stable correction parameter by the phase correlation. It is possible to apply stable phase correlation. Therefore, the intermittent false estimation pointed out as a disadvantage of image stabilization by phase correlation can be completely solved. In addition, fast and accurate frame motion vector transition component estimation is possible in that only the single region is used and the search region is limited only by the motion vector of the singular pixel initially selected for the reference frame by the optical flow method.

1 is a flowchart illustrating a correction method according to the present invention.

Fig. 2 is a photograph illustrating the estimation of the direction and magnitude of the motion vector by optical flow analysis.

3 is a block diagram of an optical flow model proposed by Lucas and Kanade to obtain an optical flow.

4 shows an example of the slope association process for I _x (2.2).

5 shows an example of a tilt process with respect to the I _y axis.

6 shows an example of a tilt process with respect to the I _t axis.

7 is a diagram illustrating setting of a search section.

Claims (3)

Forming a phase correlation space by obtaining a phase correlation between a reference frame of the reference image and a current frame by a phase correlation method, wherein the phase correlation is based on a shift property of a Fourier transform; Selecting a singular pixel from the reference image; Calculating a motion vector for the selected singular pixel using optical flow method; Obtaining an average direction of the vectors and an average size of the vectors from the calculated motion vectors of the singular pixels; Setting a search region in the phase correlation space processed for a single region from the obtained average direction information and size information with respect to the motion vector; Determining a maximum peak value in the set search region and determining a position as a correction parameter; And And applying the determined correction parameter to the current frame and outputting the stabilized frame. The method according to claim 1, The method of selecting the singular pixel may include: Extracting edge information of the reference image and setting the value as a mask; And And arranging the amount of change in pixel intensity of the current image with respect to the mask area in order of size, and selecting all pixels having the brightest value among the pixels having the same intensity change among the results. . The method according to claim 2, The selection of the singular pixel is selected only in the first frame and is performed only when the reference frame is changed.

Images