KR101012481B1 - Apparatus and Method for Correcting Moving Image Wavering - Google Patents

Abstract

Disclosed is a video stabilization device and method. The video image stabilization apparatus may further include: a selector configured to select at least one reference macroblock from an input image of a first frame and to select at least one search region corresponding to at least one reference macroblock from an input image of a second frame; An operation unit that performs a statistical operation by mapping the reference macroblock of the first frame to each of the search blocks included in the search region of the second frame and using the result value of the statistical operation to correct the output image of the second frame. And a vector calculator for calculating a motion vector. Therefore, the amount of computation can be significantly reduced while effectively correcting image distortion due to camera shake.

Description

Video stabilization device and method {Apparatus and Method for Correcting Moving Image Wavering}

The present invention relates to a video stabilization (Wavering) correction apparatus and method, and more particularly, to a video stabilization technology that can significantly reduce the amount of calculation while effectively correcting the distortion of the image due to hand shake.

In general, a digital imaging apparatus such as a digital camcorder or a digital camera continuously captures a still image of 15 to 30 frames per second and displays the same through a display means such as a liquid crystal display (LCD). As a result, a moving image is shown to the user.

When shooting video using a digital imaging device, unavoidable movement or tremor (e.g. panning and tilting) of the user's body in contact with the digital imaging device is inevitably involved, so unless a fixed means such as a tripod is used, It is difficult to maintain a physically stable shooting state. Therefore, such instability of the physical photographing state may cause distortion of the captured image.

In addition, in the case of a digital image photographing apparatus using a complementary metal oxide semiconductor module (CMOS) module, such as a mobile terminal, distortion of an image may occur due to the slight shaking generated during the rolling shutter.

The distortion of the image may be further intensified by instability due to the motion and shaking of the user's body and the digital imaging apparatus moving along the moving line of the subject, and as the number of frames taken per second decreases, interpolation due to interframe movement or vibration As the distance increases, the instability of the image experienced by the user becomes greater.

Therefore, recently, various techniques for image stabilization of a digital image photographing apparatus have been studied. For example, conventional techniques for image stabilization may be classified into an optical image stabilization technique and a non-optical image stabilization technique.

The optical image stabilization technique is a technique for applying a device capable of overcoming a shake or shake inherently with respect to an image input through a CMOS module in order to prevent a video shake. For example, the optical image stabilization may be performed by applying a gyro sensor or the like that can physically move the center of the lens of the digital imaging apparatus in the opposite direction to the movement when the lens of the digital imaging apparatus moves during the video recording. will be.

However, since the optical image stabilization should be provided with an additional device for the corresponding function in the digital imaging apparatus, there is a problem that the product production cost and unit cost increase, and the mobile pursuing small size and light weight by increasing the size of the product There is a limit to the effective application to the terminal.

On the other hand, non-optical image stabilization technology is a method for correcting the image taken at the hardware or software level of the chip state. For example, by calculating a motion vector of the photographed image in which the motion is generated and correcting the image by adjusting the offset of the photographed image as much as the motion vector.

Conventional non-optical moving image stabilization techniques have been developed to reduce the amount of computation and motion estimation in order to highlight the speed and the reduction of hardware. However, the method of reducing the amount of computation and the algorithm-based motion estimation technique cause errors in the motion estimation results or increase the time required for computation, thereby degrading its performance.

Conventional techniques for VLSI (Very Large Scale Integration) of motion estimation have shown somewhat improved results due to the high speed and integration of the system. However, as the range of the search area increases, the data of the search area to be processed gradually increases. The demand for fast speed and small footprint hardware for processing is increasing.

On the other hand, although the degree of integration for a small area is improved due to the rapid development of hardware, the use of local memory for the search area storage and the calculation according to the motion estimation is an inevitable problem in a multimedia system that processes a large amount of data. In addition, the memory used for motion estimation has a large area as a unit of frame, and a hardware area is inevitably increased as a buffer is used to increase arithmetic processing speed for a large amount of data.

The technical problem to be solved by the present invention is to provide a video image stabilization device and method that can significantly reduce the amount of calculation while effectively correcting the distortion of the image due to hand shake.

In order to solve this technical problem, the present invention provides a video image stabilization apparatus in one aspect. The video image stabilizer selects at least one reference macroblock from an input image of a first frame and selects at least one search region corresponding to the at least one reference macroblock from an input image of a second frame. Wealth; An operation unit configured to perform a statistical operation by mapping the reference macroblock of the first frame to each of the search blocks included in the search region of the second frame; And a vector calculator configured to calculate a motion vector for correcting the output image of the second frame by using the result of the statistical operation.

The at least one search area may be a plurality of areas spaced apart from each other by a specific distance from the center of the input image of the second frame. The selector may store information of a reference macroblock and a search region of each frame in a memory. For example, the selector may store pixel data of a reference macroblock of the first frame and pixel data of a search area of the second frame in a memory.

The operation unit may include a first input unit which loads pixel data of a reference macroblock of the first frame stored in the memory; A second input unit which loads pixel data of a search area of the second frame stored in the memory; And calculating the operation matrix by performing the statistical operation while moving the reference macroblock of the first frame in a vertical or horizontal direction so as to correspond to each search block included in the search region of the second frame. It may include a statistical operation unit for storing a matrix in the memory.

The video image stabilizer is configured to calculate pixel data of a specific macroblock of the first frame and pixel data of a macroblock of the second frame located at the same coordinates as a specific macroblock of the first frame and perform the operation. The controller may further include a controller configured to control the statistical operation by the calculator based on the received information.

The controller is configured to load pixel data of the reference macroblock of the first frame and pixel data of the reference macroblock of the second frame, and to convert the pixel between the reference macroblock of the first frame and the reference macroblock of the second frame. A difference calculation unit for calculating a difference value of each pixel for each pixel; A position information calculator for calculating position information of a pixel whose difference value is equal to or greater than a preset reference value; And a target controller configured to control the calculator to perform the statistical operation on only pixels corresponding to the calculated position information among pixels of the reference macroblock of the first frame.

The control unit may load pixel data of the reference macroblock of the first frame and pixel data of the reference macroblock of the second frame, and load the reference macroblock of the first frame and the reference macroblock of the second frame. A difference calculation unit that calculates a difference value of the pixel value for each pixel; A position information calculator for calculating position information of pixels whose difference value is equal to or greater than a first reference value; A ratio calculator configured to calculate a ratio of a pixel in which the difference value is equal to or greater than a first reference value preset in the reference macroblock; And when the calculated ratio is less than a second preset reference value, controlling the calculation unit to perform the statistical operation on only pixels corresponding to the position information among pixels of the reference macroblock of the first frame, The controller may include a controller configured to control the calculator to perform the statistical operation on all pixels of the reference macroblock of the first frame when the ratio is equal to or greater than a preset second reference value.

The video image stabilization apparatus receives downsized images of the first frame and the second frame from the outside, and converts each pixel value of the downsized image into a gray level value to input an image of the first frame and The apparatus may further include an input processor configured to generate input images of the second frame, respectively.

The input processor may include an image receiver configured to receive images of the first frame and the second frame from the image photographing unit, respectively; An input scaler for lowering the resolution of the received image of the first frame and the second frame; It may include an information converter for converting each pixel value of the image processed by the input scaler to a gray level value.

The video image stabilization apparatus may further include an output scaler unit which receives images of the first frame and the second frame from the outside, and downsizes the size to the size of the output image. The video image stabilization apparatus may further include a correcting unit configured to correct an output image of the second frame downsized by the output scaler using the calculated motion vector.

The vector calculating unit detects a point having a minimum value in the calculation matrix calculated through the statistical operation, calculates position information of the matching block among the search blocks included in the search region of the second frame, and calculates the position of the matched block. The motion vector may be calculated based on the location information.

The vector calculator may use a spiral search method for searching for a point value in a spiral direction from a center point of the arithmetic matrix to detect a point having the minimum value in the arithmetic matrix.

On the other hand, in order to solve the above technical problem of the present invention provides a video stabilization method in another aspect. The image stabilization method may further include selecting at least one reference macroblock from an input image of a first frame; Selecting at least one search region corresponding to the at least one reference macroblock from an input image of a second frame; Performing a statistical operation by mapping the reference macroblock of the first frame to each of the search blocks included in the search region of the second frame; And calculating a motion vector for correcting the output image of the second frame by using the result of the statistical operation.

The video stabilization method may further include loading pixel data of a reference macroblock of the first frame; Loading a reference macroblock of the second frame located at the same coordinates as the reference macroblock of the first frame; Computing pixel data of the reference macroblock of the loaded first frame and the reference macroblock of the loaded second frame; And controlling the statistical operation based on the calculated information.

The calculating may include calculating a difference value of a pixel value for each pixel between the reference macroblock of the first frame and the reference macroblock of the second frame; Comparing the calculated difference value with a first reference value; And calculating position information of a pixel whose calculated difference value is greater than or equal to a first reference value. In this case, the controlling of the statistical operation may include controlling the statistical operation to be performed only on a pixel corresponding to the position information among pixels of the reference macroblock of the first frame.

The calculating may include calculating a ratio of the calculated difference value to a reference macroblock of a pixel having a first reference value or more; The method may further include comparing the calculated ratio with a second reference value. In this case, the controlling of the statistical operation may include controlling the statistical operation using at least one of the result information of comparing the calculated ratio with the second reference value and the calculated position information. It may be.

As described above, according to the video stabilization technology according to the present invention has an advantage that the amount of calculation can be significantly reduced while effectively correcting the distortion of the image due to the hand shake.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. In the preferred embodiment of the present invention described below, specific technical terms are used for clarity of content. However, the invention is not limited to the particular term selected, and it is to be understood that each specific term includes all technical synonyms that operate in a similar manner to achieve a similar purpose.

<First Embodiment>

1 is a block diagram showing the configuration of a digital image photographing apparatus including a video image stabilizer according to a first embodiment of the present invention.

As illustrated in FIG. 1, the video image stabilizer 100 according to the first exemplary embodiment of the present invention may be provided in the digital image capturing apparatus 1. The digital image capturing apparatus 1 may be, for example, a digital camcorder, a digital camera having an image capturing function, or a mobile terminal.

The video image stabilizer 100 may be linked with the image capturing unit 10, the memory 20, the display unit 30, and the like of the digital image capturing apparatus 1. The image capturing unit 10 may be an apparatus capable of capturing an image and converting the image into an electrical signal. For example, the image capturing unit 10 may be a CMOS module or an optical imaging sensor. The memory 20 may refer to various storage means capable of storing digital data. In addition, the display unit 30 is a device capable of displaying an image on a screen, for example, may mean an LCD panel.

The video image stabilization apparatus 100 continuously receives a frame-by-frame image, that is, a frame image, from the image capturing unit 10, and removes the distortion caused by hand shake in the received frame image. The corrected frame image may be corrected by using an improved and efficient process, and then the corrected frame image may be transmitted to the display unit 30.

FIG. 2 is an exemplary diagram for exemplarily explaining the above-described concept of hand shake distortion, and FIG. 3 is a graph for explaining motion estimation capable of correcting the distortion caused by hand shake.

As shown in FIG. 2, the actual movement of the moving object is captured by the digital image capturing apparatus 1 including the image capturing unit 10, and then displayed to the user through the display unit 30. In this case, the image displayed through the display unit 30 reflects the shaking of the image capturing unit 10 caused by the camera shake as well as the movement of the moving object.

Accordingly, as shown in FIG. 3, the digital image capturing apparatus 1 uses the video image stabilization apparatus 100 to reflect a correction using motion estimation or the like on an image including the moving object. Can be displayed.

FIG. 4 is a block diagram showing the detailed configuration of the video image stabilizer 100 shown in FIG. 1, and shows the configuration of the video image stabilizer 100 according to the first preferred embodiment of the present invention.

As shown in FIG. 4, the video image stabilizer 100 includes an input processor 110, an output scaler 120, a selector 130, a calculator 140, a vector calculator 150, and a corrector ( 160), and the like. Each of the above components may perform a unique function and may interwork with the memory 20 for a corresponding operation.

Hereinafter, the components of the video image stabilizer 100 will be described in detail.

1. Input processing unit 110

The input processor 110 processes a frame image sequentially received from the image capturing unit 10 to generate an input image of each frame, and transmits the generated input image of each frame to the selection unit 130. can do. For example, the input processor 110 reduces the scale of the image by reducing the resolution of the received frame image, and also inputs the input to the selector 130 by converting each pixel value of the reduced image into a gray level value. An image can be generated.

FIG. 5 is a block diagram showing the configuration of the input processing unit 110 shown in FIG. 4.

As illustrated in FIG. 5, the input processor 110 may include an image receiver 112, an input scaler 114, an information converter 116, and the like.

The image receiving unit 112 may receive a frame image from the image capturing unit 10. For example, the image receiving unit 112 starts to receive from the image of the first frame of the video photographed from the image capturing unit 10, and the N-1 (N is an integer) th frame (for example, for convenience of understanding below). For example, an image of the previous frame, an image of the Nth frame (hereinafter, referred to as a current frame for easy understanding) may be continuously received. In this case, the received frame image may be, for example, a high resolution image composed of 1024 x 768 pixels. The pixel value of each pixel may include YCbCr data.

The input scaler 114 may downsize the frame image received by the image receiver 112. For example, the input scaler 114 reduces the resolution of images of each frame received by the image receiving unit 112, for example, the image of the previous frame, the image of the current frame, and the like to an image having 512 x 384 pixels. Can be downsized

The ratio of reducing the size of the frame image may be various application examples according to the embodiment. Preferably, it is preferable to reduce the size to about 50% to 70% of the received frame image in order to prevent the calculation of the motion vector and the reduction of the correction effect therefrom while increasing the computational efficiency.

The information converter 116 may perform a function of converting each pixel value of the downsized image by the input scaler 114 into a gray level value. Each pixel of the frame image received by the image receiving unit 112 has information such as brightness, saturation, color, etc. When calculating the motion vector, only the brightness information can sufficiently calculate the motion vector. Accordingly, the information converter 112 converts the value of each pixel to a gray level value.

When calculating the high resolution frame image photographed by the image capturing unit 10, since the number of pixels to be calculated and the information amount of each pixel is too large, the use of the memory 20 is increased and the computational efficiency is increased. Will also be lowered. However, when the operation is performed using the input image processed by the input processor 110, the number of pixels is reduced, and the amount of information of each pixel is also reduced, thereby reducing the use of the memory 20 and increasing the computational efficiency. . In practice, the difference between the case where the output image is corrected using the motion vector calculated using the original image and the case where the output image is corrected using the motion vector calculated using the processed input image is experimentally compared. The result is almost no difference between humans.

2. Output scaler unit 120

The output scaler 120 may perform a function of downsizing the frame images sequentially received from the image capturing unit 10 to match the size of a predetermined output image. For example, the output scaler 120 may downsize the image of the previous frame, the image of the current frame, etc. received from the image capturing unit 10 to a size required by the display unit 30.

Preferably, the size of the output image may be smaller than the size of the input image. For example, if the input image downsized by the input scaler 114 is, for example, a 512 x 384 image, the output image downsized by the output scaler 120 may have a size of 320 x 240, for example.

The output scaler 120 may provide a user interface for setting the size of the output image, and may resize the output image according to the size set by the user.

3. Selection unit 130

The selector 130 may perform a function of selecting at least one reference macroblock and a search region corresponding to each reference macroblock from the input image of each frame received from the input processor 110. The reference macroblock of each frame may be a specific macroblock in the search area of the frame.

The motion of the image due to camera shake during image correction should be distinguished from the case where the photographed subject moves. Therefore, in order to generate an accurate and accurate correction image, only distortion caused by hand movement is corrected without moving the subject. In general, when capturing a moving image, the subject is mostly focused on the center portion of the image. Therefore, for accurate correction, it is preferable to calculate a motion vector based on a background part where the subject is less likely to exist.

Accordingly, the selector 130 may select a plurality of areas spaced apart from each other by a specific distance from the center of the input image of each frame as a search area for calculating a motion vector.

The location of the search area and the reference macroblock may be set by the user. For example, the selector 130 may provide a user interface for setting the coordinates of the search area or the reference macroblock, and select the search area and the reference macroblock according to the set coordinates. The location of the search area and the reference macroblock may be determined in advance according to hardware resources or arithmetic processing capability of the digital image capturing apparatus 1 equipped with the video image stabilizer 100.

6 is an exemplary diagram for describing a search region and a reference macroblock selected by the selector 130.

As illustrated in FIG. 6, the selector 130 may select four search areas spaced apart from each other by a specific distance from the center of the input image IA of each frame, for example, search areas SE1, SE2, SE3, and SE4. have. Each search area may be, for example, an area of 144 x 120 size.

The selector 130 may select the search areas to be included in the size of the output image OA of each frame. For example, if the input image IA is an image of 512 x 384 size and the output image OA is 360 x 240 size, the search regions, for example, the search regions SE1, SE2, SE3, SE4 are selected to be included in the 360 x 240. It is desirable to.

The selector 130 may select macroblocks of specific coordinates, for example, macroblocks D1, D2, D3, and D4, as reference macroblocks in each search region. In the example shown in FIG. 6, the reference macroblock is set to, for example, a 96 × 72 size block.

As such, when the input image of the previous frame is received, the selector 130 selects the search area and the reference macroblock from the input image of the previous frame and stores the searched area and the reference macroblock in the memory 20, respectively, and then receives the input image of the current frame. If a search region and a reference macroblock are selected from the input image of the current frame, the search region and the reference macroblock may be respectively stored in the memory 20. In this case, the memory 20 may include a plurality of line memories. For example, pixel data such as the search area and the reference macroblock may be stored in each line memory.

Meanwhile, in the present invention, the reference macroblock of the previous frame and the search region of the current frame are used to calculate the motion vector of the current frame. This is described through the configuration of the operation unit 140 and the vector calculation unit 150 to be described below.

4. The calculating unit 140

In order to calculate information to be used to calculate a motion vector for correcting the current frame, the operation unit 140 may select a reference macroblock selected from the input image of the previous frame (that is, the reference macroblock of the previous frame) and the input image of the current frame. Statistical operations are performed using the selected search area (that is, the search area of the current frame), and a calculation matrix is calculated. For example, the calculator 140 may perform a statistical operation while mapping the reference macroblock of the previous frame to each search block included in the search region of the current frame.

The statistical operation may be performed using a variety of calculation techniques, such as sum of absolute difference (SAD), sum of squared difference (SSD), mean squared error (MSE), mean absolute error (MAE) or mean absolute (MAD) distortion). The statistical calculation may be selected in consideration of the device specification of the digital image capturing apparatus 1 equipped with the video image stabilization apparatus 100. Preferably, the statistical calculation may be performed using only sum and difference calculations. Can be used.

FIG. 7 is a block diagram showing the configuration of the calculating unit 140 shown in FIG.

As illustrated in FIG. 7, the operation unit 140 may include a first input unit 142, a second input unit 144, a statistical operation unit 146, and the like.

In the procedure for correcting the current frame, the first input unit 142 loads the pixel data of the reference macroblock of the previous frame stored in the memory 20 from the memory 20. In addition, the second input unit 144 loads the pixel data of the search area of the current frame stored in the memory 20 from the memory 20.

The statistical operation unit 146 performs a statistical operation (eg, a SAD operation) while moving the reference macroblock of the previous frame in a vertical or horizontal direction so as to correspond to each search block included in the search region of the current frame. A matrix (e.g., a SAD math matrix, etc.) can be calculated.

Hereinafter, the operation of the operation unit 140 will be described in more detail. First, for the sake of understanding, the reference macroblock is assumed to be a macroblock having a size of 5 × 5, and the search area is assumed to be an area having a size of 9 × 9. However, this is not a limitation, and the size of the macroblock may be variously selected according to the embodiment, such as 96 x 72, and likewise, the size of the search area may be variously selected according to the embodiment, such as 144 x 120. .

In addition, even if a plurality of reference macroblocks and a search region are selected, since each statistical operation is the same, a statistical operation using one reference macroblock and a search region will be described as an example.

Table 1 shows a macroblock having a size of 5 × 5 as a reference macroblock of a previous frame. m11, m12,... . m55 represents pixels in a reference macroblock.

Table 2 shows a search area having a size of 9 × 9 as a search area of the current frame. s11, s12,... s99 represents pixels in the search area.

Table 3 shows a process of performing a statistical operation using the reference macroblock of the previous frame shown in Table 1 and the search region of the current frame shown in Table 2.

As shown in Table 3, the calculation unit 140 moves the reference macroblock of the previous frame horizontally or vertically by one line in the search area to correspond to the search macro blocks in the search area. You can perform the operation. In this case, the search block may mean a block corresponding to each movement of the reference macroblock in the search area. Thus, the search block has the same size as the reference macroblock.

The range in which the reference macroblock is movable within the search area is 5 lines in the horizontal direction and 5 lines in the vertical direction. Thus, when the statistical calculation unit completes the calculation, 25 calculation result values as shown in Table 4 below, namely 5 An arithmetic matrix with x 5 points can be calculated. In this case, each point of the operation matrix may mean a result of performing a statistical operation, for example, an SAD operation, using pixel data of a reference macroblock and pixel data of a search block corresponding thereto.

Similarly, if the size of the search region of the current frame is 144 x 120 and the size of the reference macroblock of the previous frame is 96 x 72, the reference macroblock is 49 lines horizontally and vertically in the search region. Can move to 49 lines. Therefore, the result value of the statistical calculation by the calculation unit 140 is a calculation matrix having a size of 49 x 49.

When the calculation matrix is calculated through this process, the calculation unit 140 may store the calculation matrix in the memory 20. For example, the arithmetic matrix may be stored in a line memory.

5. Vector calculation unit 150

The vector calculator 150 calculates a motion vector for correcting the output image of the current frame by using a result of the statistical calculation calculated by the calculator 140, that is, a calculation matrix. The motion vector may mean a matched block most similar to the reference macroblock among the search blocks included in the search region and a relative vector value of the reference macroblock.

The vector calculator 150 may load a calculation matrix stored in the memory 20 by the calculator 140 from the memory 20 and detect a point having a minimum value in the loaded calculation matrix. The search block corresponding to the detected point may be regarded as a matching block, the position information of the matching block is calculated, and a motion vector may be calculated based on the calculated position information.

The vector calculator 150 may use a spiral search method that searches for a point value in a spiral direction from a center point of the arithmetic matrix in order to detect a point having a minimum value in the arithmetic matrix. .

8 is an exemplary diagram for describing a process of performing a spiral search performed by the vector calculator 150.

As shown in FIG. 8, the vector calculator 150 widens the rotation radius while rotating clockwise (or counterclockwise) from the center point C1 of the arithmetic matrix to find a point having a minimum value in the arithmetic matrix. The search can be performed by a thin route.

On the other hand, if there are a plurality of points having the minimum value, the vector calculator 150 may calculate the location information of the matching block by detecting a point located nearest to the center point C1. For example, B1 shown in FIG. 8 represents a point having a minimum value, and A1 and A2 represent points having the same value as the B1. That is, B1, A1, and A2 have the same minimum value. In this case, the vector calculator 150 may detect B1 closest to the center point C1.

6. Correction unit 160

The corrector 160 may perform a function of correcting an output image of the current frame downsized by the output scaler 120 using the motion vector calculated by the vector calculator 150.

For example, the correction unit 160 determines whether the characteristic of the motion vector is the intended motion of the user by using a motion vector integration (MVI), or the like. By adjusting the offset of the output image, the correction may be performed by reflecting the motion vector on the output image.

The configuration of the video image stabilizer 100 according to the first embodiment of the present invention has been described above. Hereinafter, the process of performing the correction using the video image stabilization apparatus 100 will be described in a methodological manner.

FIG. 9 is a flowchart illustrating a video stabilization method according to a first exemplary embodiment of the present invention, focusing on steps related to a procedure for correcting an image of a current frame. As mentioned above, to correct the image of the current frame, information of the previous frame and information of the current frame are used.

As illustrated in FIG. 9, the video image stabilization apparatus 100 may first process the frame images sequentially received from the image capturing unit to generate an input image of each frame (S1).

For example, when the image stabilization apparatus 100 receives an image of a previous frame from the image capturing unit 10, the video image stabilization apparatus 100 may downsize the received image of the previous frame and convert each pixel value into gray level information. . Subsequently, when the image of the current frame is received from the image capturing unit 10, the video image stabilization apparatus 100 may downsize the received image of the current frame and convert each pixel value into gray level information. That is, the video image stabilizer 100 may convert a high resolution color image into a downsized black and white image (that is, an input image of each frame) through the input processing step (S1).

Subsequently, the video image stabilizer 100 selects a search area and a reference macroblock from the input image of each processed frame (step S2).

For example, the video image stabilizer 100 selects a search region and a reference macroblock from an input image of a previous frame and stores pixel data of the search region and the reference macroblock of the selected previous frame in the memory 20, respectively. Subsequently, the video image stabilizer 100 may select a search region and a reference macroblock from an input image of the current frame and store pixel data of the search region and the reference macroblock of the selected current frame in the memory 20. Among the stored information, pixel data of a reference macroblock of a previous frame, pixel data of a search region of a current frame, and the like are used as information for calculating a motion vector of the current frame.

Subsequently, the video image stabilization apparatus 100 may perform a statistical operation by mapping the reference macroblock of the previous frame to each of the search blocks included in the search region of the current frame (S3).

For example, the video image stabilizer 100 may load pixel data of a reference macroblock of a previous frame from the memory 20 and load pixel data of a search block of the current frame from the memory. When the data is loaded, the video image stabilization apparatus 100 may calculate a calculation matrix by performing a statistical operation while matching the reference macroblock of the previous frame to each of the search blocks included in the search region of the current frame. . The computed matrix can be stored in a memory.

Next, the video image stabilization apparatus 100 may calculate a motion vector using the result of the statistical operation (step S4). For example, the video image stabilizer 100 may load the arithmetic matrix stored in the memory 20 and detect a point having a minimum value in the loaded arithmetic matrix. The search block corresponding to the detected point may be regarded as a matching block, the position information of the matching block is calculated, and the motion vector may be calculated using the calculated position information.

As described above, the video image stabilizer 100 may use a spiral search method to detect a point having a minimum value in the arithmetic matrix. In addition, when there are a plurality of points having the minimum value as a result of the search, the point closest to the center point among the plurality of points may be detected.

When the motion vector is calculated, the video image stabilization apparatus 100 corrects the output image using the calculated motion vector (step S5). For example, the video image stabilization apparatus determines whether or not the characteristic of the calculated motion vector is the intended motion of the user, and in the case of unintentional movement, i.e., shaking, etc., adjusts the offset of the output image of the current frame to adjust the The output image can be corrected. In this case, the output image of the current frame may be an image previously resized by the video image stabilizer 100.

The video stabilization apparatus 100 and the method according to the first embodiment of the present invention have been described in detail. Hereinafter, as another preferred embodiment of the present invention will be described with respect to the second embodiment that can further reduce the amount of calculation in the statistical calculation.

&Lt; Embodiment 2 >

10 is a block diagram showing the configuration of a video image stabilizer according to a second preferred embodiment of the present invention.

As shown in FIG. 10, the image stabilizer 101 according to the second exemplary embodiment of the present invention includes the input processor 110, the selector 130, the calculator 140, and the vector calculator 150 described above. The controller 180 may further include a controller 180 along with the output scaler 120 and the corrector 160.

The controller 180 calculates the difference value of the pixel value for each pixel between two frames, for example, macroblocks of the same position included in the previous frame and the current frame, for each pixel, and performs a statistical operation by the calculator based on the calculated result. You can perform the function to control.

For example, the controller 180 calculates a difference value of a pixel value for each pixel between a specific macroblock in a search region of a previous frame and a macroblock having the same position as a specific macroblock of the previous frame in the current frame, and the difference A pixel having a value greater than or equal to a specific reference value may be detected, and the operation unit 140 may be controlled to perform a statistical operation on only the detected pixel.

FIG. 11 is a block diagram showing the configuration of such a controller, and FIG. 12 is a flowchart for explaining the operation of the controller 180 shown in FIG.

As illustrated in FIG. 11, the controller 180 may include a difference calculator 182, a location information calculator 184, and a target controller 186.

11 to 12, the difference operation unit 182 loads pixel data of a specific macroblock in a search region of a previous frame and pixel data of a macroblock having the same position as the specific macroblock in a current frame from a memory, respectively. The difference value of the pixel value between each pixel can be calculated (step S11).

Preferably, the specific macroblock may mean a reference macroblock. For example, the difference operation unit 182 loads the pixel data of the reference macroblock of the previous frame and the pixel data of the reference macroblock of the current frame from the memory and corresponds to the corresponding pixel in each pixel value of the reference macroblock of the previous frame. After subtracting the pixel value of the current frame, an absolute value may be calculated.

In the second embodiment, it is assumed that the input image of the previous frame and the current frame is converted into a black and white image by the input processor 110 as in the first embodiment described above, and it is assumed that each pixel value is a gray level value. Shall be. Therefore, in this case, the difference value may mean a difference value of gray levels.

The position information calculating unit 184 compares the difference value of the pixel value for each pixel calculated by the difference calculating unit 182 with a preset reference value and calculates the position information of the pixel whose difference value is greater than or equal to the set reference value ( Step: S12).

The target controller 186 may control the calculator 140 to selectively perform statistical calculation according to the pixel based on the calculated position information (step S13). For example, the target controller 186 stores the calculated position information in the memory 20, and targets only pixels corresponding to the stored position information among the pixels of the reference macroblock of the previous frame when performing a statistical operation. As such, the operation unit 140 may be controlled to perform a statistical operation.

Hereinafter, an operation example of the controller 180 having such a configuration will be described in detail with reference to Tables 5 to 8 below.

Table 5 shows an example of the reference macroblock of the previous frame. As shown in Table 5, it is assumed that the reference macroblock is a 5 × 5 size macroblock, and each pixel value represents a gray level value. In this case, the gray level is exemplified as representing a level of 0 to 255, that is, 8 bits of brightness information.

Table 6 shows the reference macroblock of the current frame corresponding to the reference macroblock of the previous frame shown in Table 5. The reference macroblock of the previous frame and the reference macroblock of the current frame are located at the same coordinates in the input image of the frame.

The controller 180 loads respective pieces of information shown in Tables 5 to 6 from the memory 20, for example, each pixel value of the macroblocks shown in Table 6 at each pixel value of the macroblocks shown in Table 5, respectively. By subtracting and taking the absolute value, the difference value of the gray level can be calculated for each pixel. Then, a matrix as shown in Table 7 below may be generated.

As shown in Table 7, the controller 20 may calculate a difference value between gray levels for each pixel of the reference macroblock of the previous frame and the reference macroblock of the current frame.

The controller 20 compares the calculated difference values for each pixel with a preset reference value and detects position information of pixels having a difference value greater than or equal to the reference value. That is, if the difference value is larger than the reference value, the controller 20 considers that there is a change in the image in the pixel, and if the difference value is smaller than the reference value, the controller 20 considers that there is no change in the image in the pixel.

For example, if the reference value is set to 10, the controller 20 compares each difference value shown in Table 7 with a reference value of 10, and for pixels having a difference value equal to or greater than the reference value, pixels having a '1' difference value equal to or greater than the reference value Field can be given a '0'. Then, the result value of the matrix form as shown in Table 8 below can be calculated.

As shown in Table 8, correspond to the coordinates of (2,3), (3,3), (4,2), (4,3), (4,4) and (5, 4) in the reference macroblock. Pixels correspond to pixels determined to have a change in image.

The controller 180 stores the position information of the type shown in Table 8 in the memory 20, for example, a line memory, and only the pixels corresponding to the position information when the statistical operation is performed by the operation unit 140. You can control the statistical operations to be performed on the target.

For example, the operation unit 140 may perform a statistical operation by moving the reference macroblock of the previous frame horizontally or vertically by one line in the search region and corresponding the reference macroblocks to the search blocks in the search region, respectively. Can be.

In this case, the controller 180 may control the calculator 140 to perform a statistical operation using only the pixel value of the reference macroblock and the pixel value of the corresponding search block with respect to the pixel corresponding to the position information. According to the control of the controller 180, the calculation matrix calculated by the calculator 140 may have statistical calculation values only for pixels corresponding to the position information, and the remaining pixels may be set to, for example, '0'.

For example, assuming that the control unit 180 controls the statistical calculation for calculating the calculation matrix shown in Table 4 according to the positional information shown in Table 8, (2,3), (3) Only SAD23, SAD33, SAD42, SAD43, SAD44, and SAD53 values corresponding to the coordinates of (3), (4,2), (4,3), (4,4), and (5, 4) will be calculated. The values of will be zero.

In the above, the video image stabilization apparatus 101 and the related method according to the second exemplary embodiment of the present invention have been described. Hereinafter, another embodiment in which the operation of the controller is modified in the second embodiment will be described.

Third Embodiment

First, the image stabilization apparatus according to the third exemplary embodiment of the present invention is the same as the configuration of the second exemplary embodiment shown in FIG. 10, but the function of the controller is modified. In the third embodiment, the control unit calculates, for each pixel, a difference value of a pixel value between the reference macroblock of the previous frame and the macroblock of the current frame, and calculates a ratio of pixels whose difference value is greater than or equal to the first reference value in the reference macroblock. Can be calculated. The operation unit is substantially controlled only when the calculated ratio is equal to or greater than the second reference value.

FIG. 13 is a block diagram illustrating a configuration of a controller included in the video image stabilizer according to the third exemplary embodiment of the present invention, and FIG. 14 is a flowchart for describing an operation of the controller illustrated in FIG. 13.

As illustrated in FIG. 13, the controller 280 may include a difference calculator 282, a ratio calculator 286, a location information calculator 284, a target controller 288, and the like.

13 to 14, the difference operation unit 282 loads pixel data of the reference macroblock of the previous frame and pixel data of the reference macroblock in the current frame from the memory 20, respectively, so that the pixel values between each pixel may be changed. The difference value can be calculated (step S21).

For example, the difference operation unit 282 loads the pixel data of the reference macroblock of the previous frame and the pixel data of the reference macroblock of the current frame from the memory 20 and at each pixel value of the reference macroblock of the previous frame. The absolute value may be calculated after subtracting the pixel value of the current frame corresponding to the pixel. The difference value calculated at this time may be, for example, a difference value such as a gray level.

The position information calculator 284 compares the difference value of the pixel value for each pixel calculated by the difference calculator 282 with a first reference value which is set in advance, and calculates the position information of the pixel whose difference value is equal to or greater than the first reference value. It calculates (step: S22).

The ratio calculator 286 may calculate a ratio occupied by a pixel whose difference value is greater than or equal to a first reference value in the reference macroblock (step S23). For example, in the above-described example shown in Table 8, since the total number of pixels of the macroblock is 25 and there are 6 pixels having a difference value equal to or greater than a specific reference value, the ratio calculator 286 determines that the difference value is equal to or greater than the first reference value. The ratio of can be calculated as '24% '.

The target controller 288 compares the calculated ratio with a second preset reference value (step S24). In this case, if the calculated ratio is less than the preset second reference value, the target controller 288 may control the operation unit 140 to perform a statistical operation based on the calculated position information (step: S25). . For example, the target controller 288 stores the calculated position information in the memory 20, and targets only pixels corresponding to the stored position information among pixels of the reference macroblock of the previous frame when performing a statistical operation. As such, the operation unit 140 may be controlled to perform a statistical operation.

On the other hand, as a result of comparing the calculated ratio with the second reference value, if the ratio to be calculated is greater than or equal to the preset second reference value, the target controller 288 is included in the reference matrix of the previous frame when performing the statistical operation. In operation S27, a statistical operation may be performed on all pixels.

As described above, the embodiments in which the control unit calculates the pixel value of the macroblock of the previous frame and the macroblock of the current frame having the same position have been described in the second and third embodiments. According to the operation of the controller, the amount of calculation can be drastically reduced, but the pixels with actual shaking can be accurately identified and reflected in the statistical calculation.

Meanwhile, in order to control a statistical operation for correction of the current frame, the pixel value of the macroblock of the previous frame (ie, the N-1 frame) and the preceding frame (ie, the N-2 frame) of the previous frame may be calculated. For example, the controller calculates the difference value of the pixel value between the reference macroblock of the preceding frame and the reference macroblock of the previous frame for each pixel, and detects the position information of the pixels whose difference value is greater than or equal to the first reference value. In the statistical operation of the current frame, the statistical operation may be controlled based on the position information. In this case, a control scheme may be determined by calculating a ratio of pixels having the difference value equal to or greater than a first reference value and comparing the result with a second reference value.

Although the present invention has been described above with reference to its preferred embodiments, those skilled in the art will variously modify the present invention without departing from the spirit and scope of the invention as set forth in the claims below. And can be practiced with modification. Accordingly, modifications of the embodiments of the present invention will not depart from the scope of the present invention.

The following drawings, which are attached to this specification, illustrate exemplary embodiments of the present invention, and together with the detailed description of the present invention serve to further understand the technical spirit of the present invention, the present invention includes matters described in such drawings. It should not be construed as limited to.

1 is a block diagram showing the configuration of a digital image photographing apparatus including a video image stabilizer according to a first embodiment of the present invention.

2 is an exemplary diagram for exemplarily explaining the concept of the shake distortion mentioned above.

3 is a graph for explaining motion estimation capable of correcting such distortion due to camera shake.

FIG. 4 is a block diagram showing the detailed configuration of the video image stabilizer shown in FIG.

FIG. 5 is a block diagram showing a configuration of an input processing unit shown in FIG. 4.

6 is an exemplary diagram for describing a search region and a reference macroblock selected by the selection unit.

FIG. 7 is a block diagram showing the configuration of the calculation unit shown in FIG. 4.

8 is an exemplary diagram for describing a process of performing a spiral search performed by a vector calculator.

9 is a flowchart illustrating a video stabilization method according to a first embodiment of the present invention.

10 is a block diagram showing the configuration of a video image stabilizer according to a second preferred embodiment of the present invention.

11 is a block diagram showing the configuration of a control unit included in the video image stabilizer according to the second embodiment of the present invention.

FIG. 12 is a flowchart for describing an operation of the controller illustrated in FIG. 11.

FIG. 13 is a block diagram showing a configuration of a control unit included in the video image stabilizer according to the third embodiment of the present invention.

FIG. 14 is a flowchart for describing an operation of the controller illustrated in FIG. 13.

<Description of the symbols for the main parts of the drawings>

10: video recording unit 20: memory

30: display unit 100: video stabilization device

110: input processing unit 120: output scaler

130: selection unit 140: arithmetic unit

150: vector calculation unit 160: correction unit

180, 280: control unit

Claims (20)

A selection unit selecting at least one reference macroblock from an input image of a first frame and selecting at least one search region corresponding to the at least one reference macroblock from an input image of a second frame; An operation unit configured to perform a statistical operation by mapping the reference macroblock of the first frame to each of the search blocks included in the search region of the second frame; A vector calculator configured to calculate a motion vector for correcting the output image of the second frame by using the result of the statistical operation; An output scaler unit for receiving an image in the second frame from the outside and downsizing the size of the received image of the second frame to a size of an output image; And And a correction unit configured to correct an output image of the second frame downsized by the output scaler by using the calculated motion vector. The apparatus of claim 1, wherein the at least one search region is a plurality of regions spaced apart from each other by a specific distance from a center of the input image of the second frame. The method of claim 1, wherein the reference macroblock of the first frame is included in a search region in the first frame, wherein the search region of the first frame and the search region of the second frame are located at the same coordinates in the input image. Video stabilization device, characterized in that. The apparatus of claim 1, wherein the selector stores pixel data of a reference macroblock of the first frame and pixel data of a search region of the second frame in a memory. The method of claim 4, wherein the operation unit, A first input unit which loads pixel data of a reference macroblock of the first frame stored in the memory; A second input unit which loads pixel data of a search area of the second frame stored in the memory; And Compute the calculation matrix by performing the statistical operation while moving the reference macroblock of the first frame in the vertical or horizontal direction to correspond to each search block included in the search region of the second frame, and calculates the calculation matrix Video stabilization apparatus comprising a statistical operation unit for storing in the memory. The method of claim 1, wherein the pixel data of the reference macroblock of the first frame and the pixel data of the reference macroblock in the second frame located at the same coordinates as the reference macroblock of the first frame are calculated, and And a controller which controls the statistical calculation by the calculator based on the calculated information. The method of claim 6, wherein the control unit, The pixel data of the reference macroblock of the first frame and the pixel data of the reference macroblock of the second frame are loaded, and a difference value of the pixel value between the reference macroblock of the first frame and the reference macroblock of the second frame is loaded. A difference calculation unit for calculating a pixel for each pixel; A position information calculator for calculating position information of a pixel whose difference value is equal to or greater than a preset reference value; And And a target controller configured to control the calculator to perform the statistical operation on only the pixels corresponding to the calculated position information among the pixels of the reference macroblock of the first frame. The method of claim 6, wherein the control unit, The pixel data of the reference macroblock of the first frame and the pixel data of the reference macroblock of the second frame are loaded, and a difference value of the pixel value between the reference macroblock of the first frame and the reference macroblock of the second frame is loaded. A difference calculation unit for calculating a pixel for each pixel; A position information calculator for calculating position information of a pixel whose difference value is equal to or greater than a first reference value; A ratio calculator configured to calculate a ratio of a pixel in which the difference value is equal to or greater than a first reference value preset in the reference macroblock; And When the calculated ratio is less than a second preset reference value, the operation unit is controlled to perform the statistical operation on only pixels corresponding to the position information among pixels of the reference macroblock of the first frame, and the ratio And a controller configured to control the operation unit to perform the statistical operation on all pixels of the reference macroblock of the first frame when the preset second reference value is equal to or greater than the second reference value. The method of claim 1, wherein the pixel data of the reference macroblock of the first frame and the pixel data of the reference macroblock of the previous frame of the first frame located at the same coordinates as the reference macroblock of the first frame are calculated. And a controller configured to control the statistical calculation by the calculator based on the calculated information. The input image and the input image of the first frame according to claim 1, wherein an image of the first frame and the second frame is received and downsized from the outside, and each pixel value of the downsized image is converted into a gray level value. The image stabilization apparatus further comprises an input processing unit for generating an input image of the second frame, respectively. The method of claim 10, wherein the input processing unit, An image receiver configured to receive images of the first frame and the second frame from the image photographing unit, respectively; An input scaler which lowers the resolution of the received image of the first frame and the second frame; And an information converter configured to convert each pixel value of the image processed by the input scaler into a gray level value. The image stabilization method of claim 1, wherein the output scaler receives an image of the first frame from an external source and downsizes the size of the received image of the first frame to the size of the output image. Device. delete The method of claim 1, wherein the vector calculating unit detects a point having a minimum value in a calculation matrix calculated through the statistical operation, and calculates position information of a matching block among search blocks included in a search region of the second frame. And the motion vector is calculated based on the calculated position information of the matching block. 15. The method of claim 14, wherein the vector calculator uses a spiral search method for searching for values of points in a spiral direction from a center point of the calculation matrix to detect a point having the minimum value in the calculation matrix. Video stabilization device, characterized in that. Selecting at least one reference macroblock from an input image of the first frame; Selecting at least one search region corresponding to the at least one reference macroblock from an input image of a second frame; Performing a statistical operation by mapping the reference macroblock of the first frame to each of the search blocks included in the search region of the second frame; Calculating a motion vector for correcting the output image of the second frame by using the result of the statistical operation; Downsizing the size of the image in the second frame received from the outside to the size of the output image; And And correcting an output image of the down-sized second frame by using the calculated motion vector. 17. The method of claim 16, further comprising: loading pixel data of a reference macroblock of the first frame; Loading a reference macroblock in the second frame located at the same coordinates as the reference macroblock of the first frame; Computing pixel data of the reference macroblock of the loaded first frame and the reference macroblock of the loaded second frame; And And further comprising controlling the statistical operation based on the calculated information. 18. The method of claim 17, wherein the calculating comprises: Calculating a difference value of a pixel value for each pixel between the reference macroblock of the first frame and the reference macroblock of the second frame; Comparing the calculated difference value with a first reference value; And And calculating position information of a pixel whose calculated difference value is greater than or equal to a first reference value. 19. The method of claim 18, wherein controlling the statistical operation comprises: And controlling the statistical operation to be performed on only pixels corresponding to the position information among pixels of the reference macroblock of the first frame. 19. The method of claim 18, wherein said calculating comprises: Calculating a ratio of the calculated difference value to a reference macroblock of a pixel having a first reference value or more; Comparing the calculated ratio with a second reference value; Controlling the statistical operation, And controlling the statistical operation using at least one of the result information of comparing the calculated ratio with the second reference value and the calculated positional information.

Images