KR20200010527A - Apparatus and method for stabilizing image - Google Patents

Abstract

The present invention relates to a device for stabilizing an image which robustly operates in a dynamic environment and a method thereof. The device for stabilizing an image comprises: a foreground image generation part for generating a foreground image including a background and a dynamic object from a predetermined reference image and a difference image of a current image; a feature point selection part which removes the feature points corresponding to the dynamic object in the foreground image from the feature points of the reference image and selects the feature points corresponding to the background; a motion calculation part which tracks the selected feature point of the reference image from an inputted next image and calculates a motion between the reference image and the next image by finding and using a corresponding point; and an image correction part for generating a stabilized image by converting the next image using the motion.

Description

Apparatus and method for stabilizing image}

The present invention relates to an image stabilization apparatus and method that robustly operates in a dynamic environment.

Recently, as the use of multimedia devices increases, the demand for image enhancement techniques for digital images photographed in various environments is increasing. Image quality improvement techniques include image blur, noise reduction, and image stabilization technology, and are widely used in digital cameras, smartphones, home cameras or camcorders, industrial surveillance cameras, broadcast cameras, and military imaging equipment. Apply. Early photographic devices only produced images by digitizing existing analog images. However, recently used photographing apparatus has been applied a variety of pre-processing and post-processing correction technology it is possible to obtain a digital image of clearer and clearer picture than the conventional analog image.

The most commonly used technique of digital image correction is image stabilization. When the user takes a picture with the shooting device or shoots while moving, camera shake is generated.In the case of a camera installed on a vehicle such as a car, airplane or helicopter, it may be caused by various environmental factors such as mechanical vibration or friction with the ground. Shake will occur. In addition, as the magnification of the zoom lens is increased, there is a problem that the screen shake is severe even with slight movement. Image stabilization technology is a technology that can obtain a clear and clear image when the image pickup device shakes when shooting, it is applied to remove the unwanted shake occurring in the captured image.

Conventional image stabilization technology extracts feature points from reference video, finds corresponding points using feature point tracking from current video, calculates motion between reference video and current video, converts current video to calculated motion, and generates stable video. Create In the image stabilization technology, the image is stabilized using the motion of the background image, and thus the motion of the image is calculated using the feature points extracted from the background image. However, when a feature point extracted on a dynamic object is used to calculate the motion of an image, an error that cannot be ignored can be generated when calculating the motion of the image, and as a result, image stabilization fails.

Japanese Unexamined Patent Publication No. 2011-003057

The technical problem to be solved by the present invention is to provide an image stabilization apparatus and method that robustly operates in a dynamic environment by removing feature points on a dynamic object using a foreground image.

According to an aspect of the present invention, there is provided an apparatus for stabilizing an image, comprising: a foreground image generation unit generating a foreground image including a background and a dynamic object from a preset reference image and a difference image of a current image; A feature point selector which removes feature points corresponding to a dynamic object of the foreground image from feature points of the reference image and selects feature points corresponding to the background; A motion calculation unit for tracking a selected feature point of the reference image from the input next image and calculating a motion amount between the reference image and the next image using the corresponding point; And an image corrector configured to generate the stabilized image by converting the next image using the motion amount.

In the present invention, the foreground image generating unit, characterized in that to remove the noise by applying a blob filter (blob filter) to the difference image.

In the present invention, the delay unit for delaying the foreground image by one frame to match the synchronization of the foreground image and the next image; characterized in that it further comprises.

According to an aspect of the present invention, there is provided a method of stabilizing an image, comprising: a foreground image generation step of generating a foreground image including a background and a dynamic object from a preset reference image and a difference image of a current image; A feature point selecting step of removing feature points corresponding to a dynamic object of the foreground image from feature points of the reference image and selecting feature points corresponding to the background; A motion calculating step of tracking a selected feature point of the reference image from an input next image to find a corresponding point, and calculating a motion amount between the reference image and the next image using the corresponding point; And an image correction step of generating a stabilized image by converting the next image by using the motion amount.

In the present disclosure, the generating of the foreground image may include removing noise by applying blob filtering to the difference image.

In the present invention, the delay step of delaying the foreground image by one frame to match the synchronization of the foreground image and the next image; characterized in that it further comprises.

As described above, according to the present invention, the existing image stabilization technique fails to stabilize the image when the dynamic object occupies a large portion of the image. You can stabilize the image. For example, it can be used for image stabilization in an environment where there are many dynamic objects in the image, such as a person or a car.

In addition, the conventional method for generating the foreground image is slow because of the computational background modeling method, but the image stabilization technique according to the present embodiment uses the difference between the reference image and the stabilized current image. You can create a foreground image. This foreground image can be applied to other image analysis algorithms.

In addition, by using the image stabilization technology according to the present embodiment, it can be used for image registration in a fixed camera, a moving camera, a PTZ camera, which can be used for motion detection, object tracking, object detection, object recognition, and the like.

Furthermore, by using the image stabilization technique according to the present embodiment, it is possible to robustly find the corresponding point of the two images in the dynamic environment.

1 is a block diagram showing the configuration of an image stabilization apparatus according to an embodiment of the present invention.
2 is a diagram illustrating feature point extraction according to an embodiment of the present invention.
3 is a diagram illustrating image stabilization using an arbitrary image according to an embodiment of the present invention.
4 is a flowchart illustrating an operation of an image stabilization method according to an exemplary embodiment.

Advantages and features of the present invention, and methods for achieving them will be apparent with reference to the embodiments described in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments set forth below, but may be embodied in many different forms and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. . The embodiments set forth below are provided to make the disclosure of the present invention complete, and to fully inform the scope of the invention to those skilled in the art. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof. Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components will be given the same reference numerals and redundant description thereof will be omitted. Let's do it.

1 is a block diagram showing the configuration of an image stabilization apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the image stabilization apparatus 10 may include a camera 100, an image processor 110, a feature point extractor 120, a foreground image generator 130, a delay unit 140, and a feature point selector 150. ), A motion calculator 160, and an image corrector 170.

The camera 100 refers to a camera photographing a subject using, for example, a complementary metal-oxide semiconductor (COMS) module or a charge coupled device (CCD) module. An image (or video) input to the same subject may be used. It is provided to the COMS module or CCD module through the lens, the COMS module or CCD module converts the optical signal of the subject passing through the lens into an electrical signal (shooting signal) and outputs. Furthermore, the camera 100 may be a thermal camera. A thermal camera is a camera that detects a temperature difference between an object and its surrounding background by radiant energy emitted from each object, and converts it into an electrical signal to image it.

The image processor 110 may reduce noise with respect to an image signal output from the camera 100, perform gamma correction, color filter array interpolation, color matrix, and color correction ( Image signal processing for image quality improvement such as color correction and color enhancement is performed. In addition, the image processor 110 may generate an image file by compressing the image data generated by the image signal processing to improve the image quality, or may restore the image data from the image file. The compressed format of the image includes a reversible format or an irreversible format. In addition, the image processing unit 110 may also perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, image effect processing, and the like. Face recognition, scene recognition processing, and the like can be performed by the image recognition processing. For example, luminance level adjustment, color correction, contrast adjustment, outline enhancement adjustment, screen division processing, character image generation and the like, and image synthesis processing can be performed.

The feature point extractor 120 detects feature points from the reference image, the current image (the feedback stabilized current image), and the next image, which are processed and output from the image processor 110. These feature points may be represented by coordinates, respectively. The image has various feature elements, and some of these various feature elements may be extracted as feature points according to a user's needs. In order to detect feature points from an image, a Harris corner detection method, a scale invariant feature transform (SIFT) algorithm, a speeded up robust feature (SURF) algorithm, or the like may be used.

The feature point extractor 120 may include a first feature point extractor 121 for extracting feature points from a stabilized current image, a second feature point extractor 122 for extracting feature points from a reference image, and a feature point for extracting feature points from a next image. Three feature point extractor 123 may be included. Here, the reference image may represent an image having the least shaking among images included in the image data generated by photographing an object, and an image having the least shaking may be set as the reference image.

2 is a diagram illustrating feature point extraction according to an embodiment of the present invention. Referring to FIG. 2, FIG. 2A shows an arbitrary image (current image, reference image, next image), FIG. 2B shows a reduced image, and FIG. 2C shows an edge image for the reduced image. The feature point extractor 120 may extract the block as a feature point when the number of edges existing in an arbitrary block (eg, 3 × 3 block) in the edge image exceeds a threshold. Here, in order to remove a feature point existing on an edge having a high probability of failing feature point tracking and to select a feature point at which an edge intersects or is located at a corner, a threshold value may be changed by adjusting a width and a width of a block. 2D illustrates feature points extracted because the number of edges present in a block in the edge image exceeds a threshold.

The foreground image generator 130 generates a foreground image including a background and a dynamic object from the difference image between the reference image and the current image. The foreground image generating unit 130 converts the reference image and the current image into a gray scale image, respectively, and resizes and generates a foreground image including a background and a dynamic object from the difference image. 3A shows an arbitrary reference image, FIG. 3B shows an arbitrary stabilized current image, and FIG. 3C shows a foreground image including a background and a dynamic object, which is a difference image between the reference image and the stabilized current image. Is shown. The noise may be removed by applying a blob filter such as a median filter to the difference image. Here, the reason for obtaining the difference image using the reference image and the stabilized current image is that, first, if a stable image is successfully obtained, the difference image is theoretically aligned because the reference image and the stabilized current image are aligned with each other. Represents a dynamic object. Second, although a conventional background modeling technique may be used, since the processing speed is slow, the present embodiment uses the difference image of the reference image and the stabilized current image in consideration of the high speed processing.

The delay unit 140 delays the foreground image by one frame in order to synchronize the generated foreground image with the next image.

The feature point selector 150 removes the feature points corresponding to the dynamic object from the foreground image delayed by one frame from the feature points of the reference image, and selects the feature points corresponding to the background. FIG. 3D illustrates a reference image including feature points, and FIG. 3E illustrates removing feature points corresponding to a dynamic object of the foreground image from the reference image and selecting feature points corresponding to the background of the foreground image from the reference image. Is shown. As such, the feature point selector 150 removes the feature points corresponding to the dynamic object from the foreground image from the feature points of the reference image and selects the feature points corresponding to the background.

The motion calculator 160 tracks selected feature points of the reference image from the inputted next image to find a corresponding point, and calculates a motion between the reference image and the next image using the corresponding point. FIG. 3E shows a reference image including the selected feature point, and FIG. 3F shows a next image including the feature point. The motion calculator 160 obtains a corresponding point by tracking a selected feature point of the reference image from the input next image. Herein, acquiring a corresponding point may refer to a task of finding where the feature points selected in the reference image are located in the next image. The correspondence point acquisition method may use a known kanade-lucas-tomasi (KLT) tracker algorithm, a taylor expansion, and a hessian matrix. When the corresponding point is obtained as described above, the motion calculator 160 calculates a motion amount indicating how much the next image moves relative to the reference image using the corresponding point.

The image corrector 170 converts the next image using the calculated motion amount to generate a stabilized image.

Next, referring to FIG. 4, it is a flowchart illustrating the operation of an image stabilization method according to an exemplary embodiment. The image stabilization method according to the present invention may be performed in the image stabilization apparatus with the help of peripheral components as shown in FIG. 1. In the following description, portions overlapping with the description of FIGS. 1 to 3 will be omitted.

The image stabilization apparatus generates a foreground image including a background and a dynamic object from the difference image of the reference image and the current image (S100). The image stabilization apparatus converts the reference image and the current image into a gray scale image, respectively, and resizes and generates a foreground image including a background and a dynamic object from the difference image. Here, the reason for obtaining the difference image using the reference image and the stabilized current image is that, first, if a stable image is successfully obtained, the difference image is theoretically aligned because the reference image and the stabilized current image are aligned with each other. Represents a dynamic object. Second, although an existing background modeling technique may be used, since the processing speed is slow, the present embodiment uses the difference image of the reference image and the stabilized current image in consideration of high-speed processing.

When the foreground image is generated, the image stabilization apparatus delays the foreground image by one frame to match the synchronization of the generated foreground image with the next image (S200).

Then, the image stabilization apparatus removes the feature points corresponding to the dynamic object from the foreground image delayed by one frame from the feature points of the reference image, and selects the feature points corresponding to the background (S300).

The image stabilization apparatus tracks selected feature points of the reference image from the input next image, finds a corresponding point, and calculates a motion between the reference image and the next image using the corresponding point (S400). In this case, searching for a corresponding point may be an operation of finding where the selected feature points in the reference image are located in the next image. When the corresponding point is obtained as described above, the motion calculator 160 calculates a motion amount indicating how much the next image moves relative to the reference image using the corresponding point.

The image stabilization apparatus generates the stabilized image by converting the next image using the calculated motion amount (S500).

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar indicating terminology may correspond to both the singular and the plural. In addition, in the present invention, when the range is described, it includes the invention to which the individual values belonging to the range are applied (when there is no description thereof), and each individual value constituting the range is described in the detailed description of the invention. Same as

If the steps constituting the method according to the invention are not explicitly stated or contrary to the steps, the steps may be performed in a suitable order. The present invention is not necessarily limited to the description order of the above steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for describing the present invention in detail and is not limited by the claims, and thus the scope of the present invention is limited by the examples or exemplary terms. It doesn't happen. Also, one of ordinary skill in the art appreciates that various modifications, combinations and changes can be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all the scope equivalent to or equivalent to the scope of the claims as well as the claims to be described below are within the scope of the spirit of the present invention. Will belong to.

100: camera 110: image processing unit
120: feature point extractor 130: foreground image generator
140: delay unit 150: feature point selector
160: motion calculation unit 170: image correction unit

Claims (5)

A feature point extracting unit extracting feature points of the first image;
A feature point selector that removes feature points corresponding to a dynamic object among the feature points of the first image and selects feature points corresponding to a background;
In the second image, the feature points selected in the first image are traced to obtain corresponding points corresponding to the selected feature points of the first image in the second image, and the second image is obtained using the corresponding points in the second image. A motion calculator configured to calculate a motion of the motion controller; And
And an image corrector configured to generate a stabilized image correcting the movement of the second image. The method of claim 1,
And a foreground image generator which generates a foreground image from a difference image of a third image that is a previous image of the first image and the second image.
The feature point selector removes feature points corresponding to the dynamic object of the foreground image among the feature points of the first image. The method of claim 2,
The foreground image generating unit generates the foreground image from the difference image after converting and resizing the first image and the third image into a gray scale image, respectively. The method of claim 1,
And a camera for acquiring the first image and the second image. Smartphone comprising the image stabilization device of any one of claims 1 to 4.

Images