KR20070068639A - Apparatus for detecting and correcting motion by hand in moving pictures - Google Patents

Abstract

An apparatus for detecting and correcting a motion of an image caused by hand trembling is provided to calculate a motion of an image and accumulated motions and provide information required for correction to an encoder to detect and correct a motion caused by hand trembling without decreasing an encoding speed. An apparatus for detecting a motion of an image includes an estimation unit(110), a computation unit(120), and a determination unit(130). The estimation unit obtains horizontal and vertical motion vectors to estimate a motion of an input image. The computation unit calculates accumulated motions in vertical and horizontal directions by using the motion vectors. The determination unit determines that the motion of the input image is caused by hand trembling when the accumulated motions are smaller than a predetermined threshold value.

Description

Apparatus for detecting and correcting motion by hand in moving pictures}

1 is a structural diagram of an embodiment of an image stabilizer for a video according to the present invention;

Figure 2a is an exemplary view of a horizontal line input to the device of the present invention,

Figure 2b is an example of the longitudinal line input to the apparatus of the present invention,

3A is an exemplary view for explaining that the motion predictor of FIG. 1 predicts a motion in a vertical direction;

3B is an exemplary view for explaining that the motion predictor of FIG. 1 predicts a motion in a horizontal direction;

4A and 4B are exemplary views for explaining that the motion predictor of FIG. 1 predicts motion in the vertical and horizontal directions;

FIG. 5 is a detailed structural diagram of an embodiment of the cumulative motion calculation unit of FIG. 1; FIG.

6 is an exemplary view for explaining a correction process performed by the correction unit of FIG. 1.

<Explanation of symbols for the main parts of the drawings>

110: motion estimation unit 120: cumulative motion amount calculation unit

130: Image stabilization unit 140: Correction unit

The present invention relates to a camera shake detection and correction device, and more particularly to a camera shake detection and correction device used in a camcorder or camera.

In general, a block matching algorithm is most widely known as a method for detecting motion in image stabilization. In order to use the block matching algorithm, a memory for storing a previous frame is required. In order not to add such a memory, a memory of a video encoder must be used. Therefore, in order not to affect the encoding speed of the encoder, it is a task to find the global motion vector required for image stabilization fairly accurately and as quickly as possible.

As a prior art, Korean Patent Laid-Open Publication No. 1999-0011577 (the overall motion detection apparatus and method for image stabilization), Korean Patent Laid-Open Publication No. 2005-0023549 (image stabilization camera phone) and Korean Patent Publication No. 2001-0096969 ( Disclosed is a video signal transmission method and apparatus of a video terminal.

Korean Patent Laid-Open Publication No. 1999-0011577 has proposed a method for detecting only the entire motion without the effect of partial motion. When using this method, even if there is only one small and moving object in the image, The direction of the moving object is detected as a whole movement.

However, the basic definition of image stabilization is not to find the whole movement by using the direction of the moving object in the image, but on the contrary, by using an object without motion (that is, a background such as a tree, a desk, a chair, etc.) Since the entire motion is to be found, there is a problem that the motion due to the hand shake can not be found by the influence of the partial motion by the above-mentioned patent.

In addition, Patent Publication No. 2005-0023549 discloses a method of compensating motion by obtaining a motion vector using a motion sensor, adjusting a starting point of an input image image, or directly modulating a driving pulse of the sensor.

However, the above-mentioned Patent Publication No. 2005-0023549 detects a motion vector as a motion when a certain threshold is exceeded, and thus cannot accurately determine whether the motion is a hand shake or a deliberate motion (panning). However, there is a problem in that it is impossible to detect a motion due to hand shake, and since an encoder memory is used, there is a problem influencing the encoding speed.

On the other hand, Korean Patent Laid-Open No. 2001-0096969 proposes a method for improving the quality of an image transmitted to a counterpart by compensating for the shaking of an image caused by a camera movement.

However, the above-mentioned Patent Publication No. 2001-0096969 also has a problem in that it is impossible to accurately detect whether the movement is caused by hand shaking or panning, and thus, it is impossible to detect the movement due to hand shaking. However, there is a problem that affects the encoding speed.

The present invention has been proposed to solve the above problems, and by calculating the total motion and cumulative motion independently of the encoder with respect to the input image data, and providing the encoder with the information necessary for correction, the camera shakes without loss of encoding speed. It is an object of the present invention to provide an image stabilization and correction apparatus for detecting and correcting a motion by a motion picture.

In order to achieve the above object, according to a preferred embodiment of the present invention, a prediction unit for estimating the motion of the input image to obtain a motion vector in the vertical / horizontal direction; A cumulative motion amount calculator for calculating the cumulative motion amount in the vertical / horizontal direction using the motion vector obtained by the predictor; And a determination unit configured to determine the hand shake when the cumulative motion amount calculated by the cumulative motion amount calculation unit is smaller than a predetermined threshold value.

At this time, the correction unit for performing the shake correction by adjusting the offset of the image by the amount of cumulative movement may be further included.

Here, the prediction unit divides the image into a predetermined number of regions in a horizontal direction, estimates motion by taking a median value of the motion vectors of each region, and divides the image into a predetermined number of regions in a vertical direction, The motion is predicted by taking the median of the motion vectors of the regions.

The cumulative movement amount calculating unit may further include a delay unit for delaying an input signal and multiplying a gain; And an operation unit for outputting an accumulated motion amount by calculating an input signal and an output of the delay unit, wherein the gain is substantially 0.95.

The determining unit may determine panning when the cumulative motion amount calculated by the cumulative motion calculation unit is greater than a predetermined threshold.

In addition, according to another embodiment of the present invention, in the image processor of the image processing apparatus for detecting camera shake, obtain a motion vector in the vertical / horizontal direction to predict the motion of the input image, by using the motion vector A detector configured to calculate a cumulative motion amount in the vertical / horizontal direction and determine that the calculated motion is less than a predetermined threshold value as a hand shake; And a correction unit configured to perform image stabilization by adjusting an offset of the image by an amount of accumulated motion.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. First, in adding reference numerals to the components of each drawing, it should be noted that the same components as possible, even if displayed on the other drawings have the same number as possible. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a structural diagram of an embodiment of a video image stabilizer according to the present invention.

As shown in the figure, the correction apparatus of the present invention, the motion estimation unit 110, the cumulative motion calculation unit 120, the hand shake determination unit 130 for determining whether the hand shake or panning and the motion caused by the hand shake It is configured to include a correction unit 140 for correction.

On the other hand, Figures 2a and 2b is an exemplary view of the horizontal and vertical lines input to the apparatus of the present invention.

As shown in the figure, the device of the present invention obtains an average value of pixel values in the horizontal and vertical directions required for predicting a motion vector in the vertical / horizontal direction, and the average value is input. For example, when the size of an image to be encoded is VGA (640 * 480), the average value of pixel values of 240 lines in the horizontal direction and 540 lines in the vertical direction is obtained as shown in FIG. 2. For reference, in order to quickly calculate a motion vector, data can be obtained by extracting data from only half of the entire image. However, the type of data input to the apparatus of the present invention is not limited thereto.

The motion predictor 110 of FIG. 1 is responsible for predicting the motion in the vertical / horizontal direction. Hereinafter, the operation of the motion predictor 110 will be described in detail with reference to FIG. 3.

FIG. 3A is an exemplary view for explaining that the motion predictor of FIG. 1 predicts a motion in a vertical direction.

를 구한다. 현재 영상에서 구한 벡터

를 이전 영상에서 구한 벡터

과 탐색 영역 내에서 비교하여 가장 유사한 벡터를 찾는다. 이때 유사도 측정을 위해 SAD(sum of absolute difference)를 이용하였으며, 그 수식은 다음과 같다.Referring to FIG. 3A, the motion predictor of the present invention divides an image into three regions in a horizontal direction and adds pixel values in the horizontal direction in each region.

Obtain Vector obtained from the current image

Obtained from previous image

In the search region to find the most similar vector. At this time, the sum of absolute difference (SAD) was used to measure the similarity, and the formula is as follows.

로 길이가 N인 벡터이며,

는 탐색 영역 안에서의 최대 범위를 나타낸다. 각 영역에서의 움직임 벡터는 탐색 영역 안에서 SAD가 가장 적은 값을 가질 때의 위치를 찾음으로써 구하게 된다. 즉, 다음 수학식과 같다. here,

Is a vector of length N,

Indicates the maximum range within the search area. The motion vector in each region is obtained by finding the position when SAD has the smallest value in the search region. That is, it is as follows.

The motion vectors obtained in each region are used to define one entire motion vector in the vertical direction. The equation is as follows.

That is, the motion estimation unit 110 of the present invention divides the horizontal and vertical areas into three and takes the median vlaue of the motion vectors found in each local area, so that the size of the moving object exceeds half of the image size. If you don't, you can use the surrounding background (that is, an object without motion) to find a fairly accurate overall motion.

3B is an exemplary view for explaining that the motion predictor of FIG. 1 predicts a motion in a horizontal direction.

를 구한다. 현재 영상에서 구한 벡터

를 이전 영상에서 구한

와 탐색 영역 내에서 비교하여 가장 유사한 벡터를 찾음으로써, 최종적으로 수평 방향으로 하나의 전체 움직임 벡터

를 구하게 된다. 그 중간 과정은 수직 방향의 움직임 예측 과정과 동일하므로 그 상세한 설명은 생략하기로 한다.The motion predictor 110 of the present invention divides an image into three regions in the vertical direction, and then adds pixel values in the vertical direction in each region.

Obtain Vector obtained from the current image

From the previous video

And compare within the search area to find the most similar vector, so that finally one full motion vector in the horizontal direction

Will be obtained. Since the intermediate process is the same as the vertical motion prediction process, a detailed description thereof will be omitted.

That is, when an average value of pixel values in the vertical and horizontal directions is input as shown in FIGS. 2A and 2B, the motion predictor 110 predicts a motion vector in the vertical / horizontal direction. 4A and 4B are exemplary views for explaining that the motion predictor of FIG. 1 predicts motion in the vertical and horizontal directions.

As shown in the figure, the search area is ± 30. The method of predicting the motion vector is as described above, and in the application example of the present invention, the horizontal motion vector can be obtained by down sampling to 1/2 to reduce the amount of calculation.

The cumulative motion calculation unit 120 of FIG. 1 is responsible for calculating the cumulative motion amount in the vertical / horizontal direction using the motion vector in the vertical / horizontal direction obtained by the motion predictor 110. This will be described with reference to FIG. 5.

FIG. 5 is a detailed structural diagram of an embodiment of the cumulative movement amount calculator of FIG. 1. FIG.

는 다음 수학식과 같다.As shown in the figure, the cumulative motion calculation unit 120 of the present invention includes a calculation unit 310 and the delay unit 320, the input is

Is as shown in the following equation.

를 곱하여 연산부(310)에 제공한다. 본 발명의 바람직한 일실시예에 따르면,

는 0.95이다. 연산부(310)는 지연부(320)에서 입력신호

와 지연부(320)에서 입력된 신호를 덧셈 연산하는 기능을 담당한다. 연산부(310)의 출력인

는 다음의 수학식과 같으 며,

이다.Delay unit 320 delays the input signal and gain

Multiply by and provide it to the operation unit 310. According to a preferred embodiment of the present invention,

Is 0.95. The calculation unit 310 is an input signal from the delay unit 320

And it is responsible for the addition operation of the signal input from the delay unit (320). Output of the operation unit 310

Is the same as

to be.

Meanwhile, the hand shake determination unit 130 of FIG. 1 considers the hand shake when the cumulative motion amount calculated by the cumulative motion calculation unit 120 is smaller than a predetermined threshold value, and transmits the hand shake to the corrector 140. However, if the cumulative amount of motion is greater than the threshold, it is regarded as intentional panning. For example, the hand shake determiner 130 may consider whether the hand shake or the panning is when the cumulative movement amount obtained from 10 consecutive images is less than 20.

6 illustrates a correction process of the correction unit 140. 6 is an exemplary view for explaining a correction process performed by the correction unit of FIG. 1.

As shown in the figure, when the cumulative motion amount is within the stabilization area, the offset ('X_offset' and 'Y_offset' in FIG. 6) of the image is adjusted by the cumulative motion amount, and an appropriate area is read from the frame memory and encoded. By performing the above, it is possible to obtain an image in which the shake is corrected. If the cumulative motion amount is out of the stable region in the middle of the shake correction, the cumulative motion amount is initialized and calculated. In this case, in order to remove the discontinuity of the image due to the mode switching, the image may be sequentially returned to its original position by sequentially moving the three images successively by a motion vector corresponding to 1/4 of the last cumulative movement amount.

The present invention described above is not limited to the above-described embodiments and the accompanying drawings, and various substitutions, modifications, and changes can be made in the art without departing from the technical spirit of the present invention. It will be clear to those of ordinary knowledge.

As described above, the present invention divides the horizontal and vertical regions into three and takes the median vlaue of the motion vectors found in each local region, so that when the size of the moving object does not exceed half of the image size, It has the effect of using the surrounding background (the object that is not moving) to find a fairly accurate total motion.

In addition, the present invention calculates the total motion and cumulative motion amount through the fast operation with the image data input in the middle of the CSP process independently of the encoder to provide the encoder with the information necessary for image stabilization, so that no loss in encoding speed This has the effect of allowing a fairly accurate calibration without giving.

In addition, the present invention has the effect of distinguishing panning and hand shake to detect only the case of hand shake, thereby enabling accurate hand shake correction.

Claims (8)

A prediction unit for predicting a motion of an input image by obtaining a motion vector in a vertical / horizontal direction; A cumulative motion amount calculator for calculating the cumulative motion amount in the vertical / horizontal direction using the motion vector obtained by the predictor; And And a determination unit for determining a hand shake when the cumulative motion amount calculated by the cumulative motion amount calculation unit is smaller than a predetermined threshold value. The apparatus of claim 1, further comprising a correction unit configured to perform image stabilization by adjusting an offset of an image by an amount of cumulative movement. The image stabilizer of claim 1, wherein the predictor divides the image into a predetermined number of regions in a horizontal direction, and estimates motion by taking an intermediate value of a motion vector of each region. The image stabilizer of claim 1, wherein the prediction unit divides the image into a predetermined number of regions in a vertical direction, and estimates motion by taking an intermediate value of the motion vectors of each region. The apparatus of claim 1, wherein the cumulative movement amount calculating unit comprises: a delay unit for delaying an input signal and multiplying a gain; And And a computing unit for outputting a cumulative movement amount by calculating an input signal and an output of the delay unit. The camera shake detection apparatus according to claim 5, wherein the gain is substantially 0.95. The apparatus of claim 1, wherein the determination unit determines panning when the cumulative motion amount calculated by the cumulative motion amount calculation unit is greater than a predetermined threshold value. An image processor of an image processing apparatus for detecting camera shake, When the motion vector in the vertical / horizontal direction is calculated, the motion of the input image is predicted, and the cumulative motion amount in the vertical / horizontal direction is calculated using the motion vector thus obtained, and the calculated motion is smaller than the predetermined threshold. A detection unit for determining by hand shake; And An image processor including a correction unit for performing image stabilization by adjusting the offset of the image by the amount of cumulative movement.

Images