KR102213201B1 - Apparatus and method for stabilizing image - Google Patents

Abstract

The present invention relates to an image stabilization apparatus and method that operates robustly in a dynamic environment. The image stabilization device includes a foreground image generator that generates a foreground image including a background and a dynamic object from a difference image between a preset reference image and a current image, and removes feature points corresponding to the dynamic object among the foreground images from the feature points of the reference image. And, a feature point selector that selects feature points corresponding to the background, a motion calculator that tracks the selected feature point of the reference image from the input next image to find a corresponding point, and calculates a motion between the reference image and the next image using the corresponding point, and motion It includes an image correction unit for generating a stabilized image by converting the next image by using.

Description

Apparatus and method for stabilizing image}

The present invention relates to an image stabilization apparatus and method that operates robustly in a dynamic environment.

Recently, as the use of multimedia devices increases, the demand for image enhancement technology for digital images photographed in various environments is also increasing. Image quality improvement technologies include image blur removal, noise removal, and image stabilization technologies, and are widely used in shooting devices such as digital cameras, smartphones, home cameras or camcorders, industrial surveillance cameras, broadcasting cameras, and military video recording devices. Apply. Early photographing devices only digitized existing analog images to create images. However, recently used photographing apparatuses have applied various pre-processing and post-processing correction technologies to obtain digital images with clearer and clearer quality than conventional analog images.

Among the digital image correction technologies, the most commonly used technology is image stabilization. Shaking occurs when a user takes a picture with a photographic device or when filming while moving, and in the case of a camera installed in a vehicle, such as a car, airplane, or helicopter, it is caused by various environmental factors such as mechanical vibration or friction with the ground. There is no shaking. In addition, as the magnification of the zoom lens increases, there is a problem that the screen shake becomes severe even with slight movement. The image stabilization technology is a technology capable of obtaining a clear and clear image when the photographing device shakes during photographing, and is applied to remove unwanted shake that occurs in the photographed image.

Existing image stabilization technology extracts a feature point from a reference image, finds a corresponding point using feature point tracking in the current image, calculates the motion between the reference image and the current image, and converts the current image into the calculated motion to obtain a stable image. Generate. In this image stabilization technique, since the image is stabilized by using the motion of the background image, the motion of the image is calculated using the feature points extracted from the background image. However, when a feature point extracted on a dynamic object is used for motion calculation of an image, an error that cannot be ignored may occur in the motion calculation of an image, and as a result, it becomes a factor in which image stabilization fails.

Japanese Laid-Open Patent Publication No. 2011-003057

A technical problem to be solved by the present invention is to provide an image stabilization apparatus and method that operates robustly even in a dynamic environment by removing feature points on a dynamic object using a foreground image.

An image stabilization apparatus according to an embodiment for solving a technical problem to be achieved by the present invention includes: a foreground image generator configured to generate a foreground image including a background and a dynamic object from a difference image between a preset reference image and a current image; A feature point selection unit for removing feature points corresponding to the dynamic object in the foreground image from feature points of the reference image and selecting feature points corresponding to the background; A motion calculator configured to find a corresponding point by tracking the selected feature point of the reference image from the input next image, and calculate a motion amount between the reference image and the next image using the corresponding point; And an image correction unit for generating a stabilized image by converting the next image using the amount of motion.

In the present invention, the foreground image generator is characterized in that noise is removed by applying a blob filter to the difference image.

In the present invention, it characterized in that it further comprises a delay unit for delaying the foreground image by one frame in order to match the synchronization between the foreground image and the next image.

An image stabilization method according to an embodiment for solving a technical problem to be achieved by the present invention includes a foreground image generation step of generating a foreground image including a background and a dynamic object from a difference image between a preset reference image and a current image; A feature point selection step of removing feature points corresponding to the dynamic object in the foreground image from feature points of the reference image and selecting feature points corresponding to the background; A motion calculation step of tracking a selected feature point of the reference image in the input next image to find a corresponding point, and calculating a motion amount between the reference image and the next image using the corresponding point; And an image correction step of generating a stabilized image by converting the next image using the amount of motion.

In the present invention, the foreground image generation step includes removing noise by applying blob filtering to the difference image.

In the present invention, it characterized in that it further comprises a delay step of delaying the foreground image by one frame in order to match the synchronization between the foreground image and the next image.

As described above, according to the present invention, the existing image stabilization technology fails to stabilize the image when the proportion occupied by the dynamic object in the image increases, but the image stabilization technology according to the present embodiment fails even when the proportion occupied by the dynamic object in the image is large. You can stabilize the video. For example, it can be used for image stabilization in an environment where there are many dynamic objects in the image, such as a person or a car.

In addition, the conventional method of generating the foreground image is slow because the background modeling technique with a large amount of computation is used, but the image stabilization technology according to the present embodiment uses the difference image between the reference image and the stabilized current image. You can create a foreground image. This foreground image can be applied to other image analysis algorithms.

In addition, the image stabilization technology according to the present embodiment can be used for image matching in a fixed camera, a moving camera, and a PTZ camera, which can be used for motion detection, object tracking, object detection, and object recognition.

Furthermore, by using the image stabilization technology according to the present embodiment, it is possible to find a correspondence point between two images in a dynamic environment.

1 is a block diagram showing the configuration of an image stabilization apparatus according to an embodiment of the present invention.
2 is a diagram illustrating feature point extraction according to an embodiment of the present invention.
3 is a diagram illustrating image stabilization using an arbitrary image according to an embodiment of the present invention.
4 is a flowchart showing an operation of an image stabilization method according to an embodiment of the present invention.

Advantages and features of the present invention, and a method of achieving them will become apparent with reference to embodiments described in detail together with the accompanying drawings. However, it should be understood that the present invention is not limited to the embodiments presented below, but may be implemented in a variety of different forms, and includes all transformations, equivalents, or substitutes included in the spirit and scope of the present invention. . The embodiments presented below are provided to complete the disclosure of the present invention, and to fully inform a person of ordinary skill in the art to which the present invention belongs. In describing the present invention, when it is determined that a detailed description of a related known technology may obscure the subject matter of the present invention, a detailed description thereof will be omitted.

The terms used in the present application are used only to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance. Terms such as first and second may be used to describe various elements, but the elements should not be limited by the terms. These terms are used only for the purpose of distinguishing one component from another component.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings, and in the description with reference to the accompanying drawings, identical or corresponding components are assigned the same reference numbers, and redundant descriptions thereof are omitted. I will do it.

1 is a block diagram showing the configuration of an image stabilization apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the image stabilization apparatus 10 includes a camera 100, an image processing unit 110, a feature point extraction unit 120, a foreground image generation unit 130, a delay unit 140, and a feature point selection unit 150. ), a motion calculation unit 160 and an image correction unit 170.

The camera 100 refers to a camera that photographs a subject using, for example, a complementary metal-oxide semiconductor (COMS) module or a charge coupled device (CCD) module, and an image (or video) input to the same subject is It is provided as a COMS module or a CCD module through a lens, and the COMS module or CCD module converts an optical signal of a subject passing through the lens into an electrical signal (photographing signal) and outputs it. Furthermore, the camera 100 may be a thermal camera. A thermal camera refers to a camera that detects a temperature difference between an object and its surrounding background by radiant energy emitted from each object, converts it into an electrical signal, and image.

The image processing unit 110 reduces noise with respect to the image signal output from the camera 100, gamma correction, color filter array interpolation, color matrix, color correction ( Image signal processing for image quality improvement, such as color correction and color enhancement, is performed. In addition, the image processing unit 110 may generate an image file by compressing image data generated by processing an image signal for image quality improvement, or may restore image data from the image file. The video compression format includes a reversible format or an irreversible format. In addition, the image processing unit 110 may functionally perform color processing, blur processing, edge enhancement processing, image analysis processing, image recognition processing, and image effect processing. Face recognition, scene recognition processing, and the like can be performed by image recognition processing. For example, luminance level adjustment, color correction, contrast adjustment, outline enhancement adjustment, screen division processing, character image generation and image synthesis processing, and the like can be performed.

The feature point extracting unit 120 detects feature points from a reference image, a current image (feedback stabilized current image) and a next image that are image-processed and output from the image processing unit 110. Each of these feature points can be displayed as coordinates. The image includes various feature elements, and some of these various feature elements may be extracted as feature points according to the needs of the user. To detect feature points from an image, a Harris corner detection method, a scale invariant feature transform (SIFT) algorithm, a speeded up robust feature (SURF) algorithm, and the like can be used.

The feature point extracting unit 120 includes a first feature point extracting unit 121 for extracting feature points from the stabilized current image, a second feature point extracting unit 122 for extracting feature points from a reference image, and a second feature point extracting unit 122 for extracting feature points from the next image. 3 may include a feature point extraction unit 123. Here, the reference image represents an image having the least shaking among images included in image data generated by photographing an object, and an image having the least shaking may be set as the reference image.

2 is a diagram illustrating feature point extraction according to an embodiment of the present invention. Referring to FIG. 2, FIG. 2A shows an arbitrary image (current image, reference image, and next image), FIG. 2B shows a reduced image, and FIG. 2C shows an edge image for a reduced image. The feature point extracting unit 120 may extract the corresponding block as a feature point when the number of edges existing in an arbitrary block (eg, 3×3 block) in the edge image exceeds a threshold value. Here, in order to remove a feature point existing on an edge with a high probability of failing to track the feature point, and select a feature point at which the edge intersects or at a corner, the threshold value can be changed by adjusting the width and width of the block. FIG. 2D shows feature points extracted because the number of edges in a block in the edge image exceeds a threshold value.

The foreground image generator 130 generates a foreground image including a background and a dynamic object from a difference image between the reference image and the current image. The foreground image generation unit 130 converts the reference image and the current image into gray scale images, respectively, resizes, and then generates a foreground image including a background and a dynamic object from the difference image. 3A shows a random reference image, FIG. 3B shows a random stabilized current image, and FIG. 3C shows a foreground image including a background and a dynamic object, which is a difference between the reference image and the stabilized current image. Is shown. Here, noise may be removed by applying a blob filter such as a median filter to the difference image. Here, the reason for obtaining the difference image using the reference image and the stabilized current image is: First, if a successful and stable image is obtained, the difference image is theoretically because the reference image and the stabilized current image are aligned with each other. Represents a dynamic object. Second, although an existing background modeling technique may be used, since the processing speed is slow, the present embodiment uses the difference image between the reference image and the stabilized current image in consideration of high-speed processing.

The delay unit 140 delays the foreground image by one frame in order to synchronize the generated foreground image with the next image.

The feature point selection unit 150 removes feature points corresponding to the dynamic object among the foreground image delayed by one frame from the feature points of the reference image, and selects feature points corresponding to the background. Fig. 3D shows a reference image including feature points, and Fig. 3E is a diagram in which feature points corresponding to the dynamic object of the foreground image are removed from the reference image, and feature points corresponding to the background of the foreground image are selected from the reference image. Is shown. In this way, the feature point selection unit 150 removes feature points corresponding to the dynamic object among the foreground image from the feature points of the reference image, and selects feature points corresponding to the background.

The motion calculator 160 finds a corresponding point by tracking the selected feature point of the reference image from the input next image, and calculates a motion between the reference image and the next image using the corresponding point. FIG. 3E shows a reference image including the selected feature point, and FIG. 3F shows a next image including the feature point. The motion calculator 160 acquires a corresponding point by tracking the selected feature point of the reference image from the input next image. Here, the acquisition of the corresponding point may be referred to as an operation of finding where the feature points selected from the reference image are located in the next image. A method of obtaining a corresponding point may use a known Kanade-lucas-tomasi (KLT) tracker algorithm, a taylor expansion, and a Hessian matrix. When the corresponding point is obtained in this way, the motion calculation unit 160 calculates a motion amount indicating how much of the next image relative to the reference image has moved using the corresponding point.

The image correction unit 170 generates a stabilized image by converting the next image using the calculated amount of motion.

Next, referring to FIG. 4 is a flowchart showing an operation of an image stabilization method according to an exemplary embodiment of the present invention. The image stabilization method according to the present invention may be performed in an image stabilization apparatus with the help of surrounding elements as shown in FIG. 1. In the following description, portions overlapping with the descriptions of FIGS. 1 to 3 will be omitted.

The image stabilization apparatus generates a foreground image including a background and a dynamic object from the difference image between the reference image and the current image (S100). The image stabilization apparatus converts the reference image and the current image into gray scale images, respectively, resizes, and then generates a foreground image including a background and a dynamic object from the difference image. Here, the reason for obtaining the difference image using the reference image and the stabilized current image is: First, if a successful and stable image is obtained, the difference image is theoretically because the reference image and the stabilized current image are aligned with each other. Represents a dynamic object. Second, although an existing background modeling technique may be used, since the processing speed is slow, the present embodiment uses the difference image between the reference image and the stabilized current image in consideration of high-speed processing.

When the foreground image is generated, the image stabilization apparatus delays the foreground image by one frame in order to synchronize the generated foreground image with the next image (S200).

Then, the image stabilization apparatus removes the feature points corresponding to the dynamic object among the foreground image delayed by one frame from the feature points of the reference image, and selects the feature points corresponding to the background (S300).

The image stabilization apparatus finds a corresponding point by tracking the selected feature point of the reference image from the input next image, and calculates a motion between the reference image and the next image using the corresponding point (S400). Here, finding the corresponding point can be said to be an operation of finding where the feature points selected in the reference image are located in the next image. When the corresponding point is obtained in this way, the motion calculation unit 160 calculates a motion amount indicating how much of the next image relative to the reference image has moved using the corresponding point.

The image stabilization apparatus generates a stabilized image by converting the next image using the calculated motion amount (S500).

In the specification of the present invention (especially in the claims), the use of the term "above" and a reference term similar thereto may correspond to both the singular and the plural. In addition, when a range is described in the present invention, the invention to which an individual value falling within the range is applied (unless otherwise stated), and each individual value constituting the range is described in the detailed description of the invention. Same as

If there is no explicit order or contrary to the steps constituting the method according to the present invention, the steps may be performed in a suitable order. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or illustrative terms (for example, etc.) in the present invention is merely for describing the present invention in detail, and the scope of the present invention is limited by the above examples or illustrative terms unless limited by the claims. It does not become. In addition, those skilled in the art can recognize that various modifications, combinations, and changes may be configured according to design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention is limited to the above-described embodiments and should not be determined, and all ranges equivalent to or equivalently changed from the claims to be described later as well as the claims to be described later are the scope of the spirit of the present invention. It will be said to belong to.

100: camera 110: image processing unit
120: feature point extraction unit 130: foreground image generation unit
140: delay unit 150: feature point selection unit
160: motion calculation unit 170: image correction unit

Claims (6)

delete A feature point extraction unit for extracting feature points of the first image;
A feature point selection unit for removing feature points corresponding to the dynamic object among feature points of the first image and selecting feature points corresponding to the background;
By tracking selected feature points of the first image in a second image, corresponding points corresponding to the selected feature points of the first image are obtained in the second image, and corresponding points of the second image are A motion calculator for calculating a motion;
An image correction unit generating a stabilized image obtained by correcting the motion of the second image; And
A foreground image generator for generating a foreground image from a difference image between the first image and a third image that is a previous image of the second image; further comprising,
The feature point selector removes feature points corresponding to the dynamic object of the foreground image among feature points of the first image. The method of claim 2,
The foreground image generating unit generates the foreground image from a difference image after converting and resizing the first image and the third image into a gray scale image, respectively. The method of claim 2,
An image stabilization apparatus further comprising a camera that acquires the first image and the second image. A smartphone comprising the image stabilization device of any one of claims 2 to 4.
Extracting feature points of the first image;
Removing feature points corresponding to the dynamic object based on the foreground image among the feature points of the first image and selecting feature points corresponding to the background;
By tracking selected feature points of the first image in a second image, corresponding points corresponding to the selected feature points of the first image are obtained in the second image, and corresponding points of the second image are Calculating motion; And
Generating a stabilized image obtained by correcting the motion of the second image; including,
The foreground image is generated from a difference image between the first image and a third image that is a previous image of the second image.

Images