KR20050103031A - Image alignment method by using statistical correlation - Google Patents

Abstract

The present invention relates to image fusion, and more particularly, to a method of aligning each image in image fusion. There is provided a method of aligning images using statistical correlation, which does not find feature points from images obtained by each image sensor, and aligns images from different image sensors through a statistical method through correlation. A first step of receiving an image from different image sensors; Extracting an outline from the input image; And a third step of aligning the image to a value at which the value of the correlation is maximum by statistically applying a correlation between the extracted contour lines of each image. This allows for accurate alignment while reducing overall computation.

Description

Image Alignment Method by Using Statistical Correlation

The present invention relates to image fusion, and more particularly, to a method of aligning each image in image fusion.

Recently, image fusion from multiple image sensors combines image information from each single image sensor, and allows processing of data according to the purpose from a database of image information of each image sensor combined to make it more accurate. As a technology for obtaining image information having various uses, research has been conducted in various fields. In particular, the technical importance of image fusion technology from multiple image sensors is increasing day by day to build such improved accuracy and more sophisticated specific element information.

Here, the image fusion from the multiple image sensor means that the optimal information is extracted from the image acquired in the unique wavelength band of each single image sensor according to the purpose of use, and a lot of information is added to one image. It is to fuse. Such technologies related to image fusion from multiple image sensors include computer vision, automatic object detection and identification, navigation, intelligent robotics, remote sensing, and medical devices.

Such image fusion can be roughly classified into three stages as a whole. Here, in step 3, a first step is a process of processing an input image of each image sensor, and a second step is alignment (alignment, registration) of each pixel of the input image from each image sensor. ), And the third step is a step of fusing the aligned images.

In particular, in the process of image fusion, when the image synthesis is performed in a state where the alignment of each image is not correct, the distortion of the image becomes more severe than the improvement of the image. Because of this phenomenon, research on how to align images (alignment, registration) has emerged as an important topic.

Hereinafter, a conventional technique for aligning an image will be described.

In the images obtained from image sensors having different characteristics (EO, IR, Radar, etc.), the correlation of brightness values corresponding to each pixel is mostly very complicated.

That is, as shown in Table 1 below, since the photon flux level of each wavelength is different, the information included in the image of each sensor has various variables. .

VIS NIR MWIR LWIR Wavelength range (um) 0.4-0.78 0.78-1.0 3-5 8-12 Sunny daylight 1.5 x 10 ¹⁷ 1 x 10 ¹⁷ 2 x 10 ¹³ 2 x 10 ¹⁷ Moonlit night 1.5 x 10 ¹¹ - 2 x 10 ¹³ 2 x 10 ¹⁷ Night under the stars 1.5 x 10 ⁹ 9 x 10 ⁹ 2 x 10 ¹³ 2 x 10 ¹⁷

As shown in Table 1, the feature present in the visible light band sensor does not exist in the infrared region sensor. In addition, what is present in the infrared region sensor does not exist in the visible light band sensor or appears as weak signal data. It is difficult to know the correlation between two images from different sensors. In practice, matching of feature points for each image may be performed after a number of steps in the preprocessing, and then the image may be registered.

1 is a configuration diagram of a general image input device for image fusion.

As shown in FIG. 1, a general image input device for image fusion includes two image input sensors 12 and 13 and two image sensors 12 and 13 having different characteristics from an optical system 14 receiving an image. And an image fusion processor 11 for fusing the image using the image input from the image.

The two image sensors 12 and 13 illustrated in FIG. 1 illustrate the charge-coupled device (CCD) type image sensor 12 and the IR (InfraRed) type image sensor 13. However, not only CCD and IR methods but also combinations of various image sensors such as Radar and electro-optic (EO) are possible.

In this way, image alignment is important in order to fuse images obtained from sensors having different characteristics (EO, IR, Radar, etc.). In the conventional method, an image alignment method using a feature point of each image is mainly used for image alignment.

3 is an exemplary diagram of image alignment using feature points of an image.

3A and 3E are exemplary views of images obtained by using an IR sensor and a CCD sensor, respectively. Each of the images obtained as described above is obtained with images having the feature points as shown in FIGS. 3 (d) and 3 (h) through FIGS.

When the feature points thus obtained are matched, the combination is performed as shown in FIG. 3 (k). Accordingly, the two images are combined to form FIGS. 3 (i) and 3 (j).

As described above, in the case of image alignment using a feature point, the preprocessing process is very complicated and difficult, and since only the feature points are compared, two images are aligned with each other, as shown in FIGS. 3 (i) and 3 (j). This happens.

The present invention has been proposed to solve the above problems, and does not find a feature point from images obtained by each image sensor, and aligns images from different image sensors through a statistical method through correlation. It is an object of the present invention to provide a method of arranging images using statistical correlation.

According to an aspect of the present invention, there is provided a method of aligning images using statistical correlations, the method comprising: a first step of receiving images from different image sensors; Extracting an outline from the input image; And a third step of aligning the image to a value at which the value of the correlation is maximum by statistically applying a correlation between the extracted contour lines for each image.

In particular, the third step comprises: setting a motion vector and a matrix for aligning the respective images; A fifth step of obtaining a normalized correlation between each of the images; A sixth step of taking a gradient of the matrix obtained from the arbitrary motion vector and a Hessian of the matrix obtained from the arbitrary moved vector; A seventh step of obtaining a difference between the motion vector obtained through the sixth step operation and the arbitrary motion vector; An eighth step of updating the arbitrary motion vector using the difference of the seventh step; And a ninth step of repeating the fifth to eighth steps until the correlation value is maximum, and sorting according to the corresponding motion vector when the correlation value is maximum.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the same components in the drawings are represented by the same reference numerals and symbols as much as possible even though they are shown in different drawings. In addition, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Since the image transmitted through the plurality of image sensors has a large difference in the photon flux level according to each wavelength of each image sensor, in order to fuse the images transmitted from the plurality of image sensors as described above, Similarly, image alignment is necessary.

In the present invention, in order to obtain an accurate result with a small amount of calculation in the image alignment, an image alignment method using a statistical technique using correlation (Correlation) is provided.

For the image alignment method of the statistical method using the correlation for image fusion from the multiple image sensor according to the present invention, for stable matching between the input image (correlated) at all resolution levels (correlated) You must have enough information. Of course, this condition is satisfied between images obtained from the same image sensor, but not in the case of an image from multiple image sensors.

In general, the image from the multiple image sensor has a relatively good relationship in the image component of the high frequency space. Therefore, since the edge information, which is a high frequency component, has information such as the appearance and outline of an object of an image, similar information may be included in an image from multiple image sensors. However, the low frequency image components are mostly dependent on the surrounding environment such as lighting and sensor characteristics, and thus are excluded from the alignment of the multiple image sensors.

Therefore, in order to obtain contour information of the image, harmonic components are extracted through a Laplacian filter. As described above, the Laplacian operation is performed to extract the contour information of the image. Since the Laplacian varies its information according to each direction, the contour of various directions including up, down, left, and right diagonal lines should be taken as shown in FIG. do.

However, in the present invention, in the method of extracting the contour, a method using a threshold is applied, not a conventional method through each directional Laplacian operation. That is, since the contour obtained through the conventional Laplacian operation has different information according to each direction, the contour for each direction is taken through the Laplacian operation in the up, down, left, and right directions.

2 is an exemplary diagram illustrating a direction of a Laplacian operation for obtaining a conventional outline.

That is, as shown in FIG. 2, the conventional Laplacian operation for obtaining the contour is performed by performing the Laplacian operation in the up, down, left, and right directions, respectively, and combining the obtained contours for each direction on an average, so that the entire Laplacian operation is obtained. You get an outline.

However, when the Laplacian operation is performed in each direction as described above, the system is heavily loaded. Accordingly, an embodiment of the present invention proposes a method of obtaining a contour similar to that of the conventional method by performing a Laplacian operation in only one direction.

In the embodiment of the present invention, a method of obtaining a contour is performed by performing a conventional Laplacian operation in one direction, by lowering the gray level for obtaining the contour to a threshold value compared to the conventional method.

In general, when the gray level is viewed in 100 steps, the gray level of the contour detected through the conventional Laplacian operation is about 45 degrees. However, in the embodiment of the present invention, by lowering the gray level to about the threshold value (for example, 18 to 24), the desired outline can be obtained in one operation. As such, if the contour is detected only in one direction (for example, ② direction in Fig. 2) by lowering the gray level, the amount of computation is much larger, but one to Laplacian operations can be replaced by 4 to 8 Laplacian operations. This can substantially reduce computation.

4 is a flowchart illustrating an embodiment of a method for aligning an image using statistical correlation according to the present invention.

Before describing the image alignment method according to the present invention as shown in FIG. 4, the statistical correlation will be described as follows. Normalized correlation between multiple images is consistent with the difference between the average value of the image pixel values and the contrast. That is, if two images have a linearized correlation, the normalized correlation between the two images is “1”. On the other hand, the two images may be said to have a spatial displacement difference unless they have a linearized correlation. Therefore, the normalized correlation value can be used to predict the relative displacement difference between images by using a statistical prediction method, and can be used to align the images.

In the image alignment method using the statistical correlation according to the present invention, the two normalized images (in the embodiment of the present invention, exemplify images through two image sensors), a movement in which the normalized correlation values at f and g are maximized. vector To find.

For that first the random move vector ₀ and X matrix (M ( (401).

In detail, each of the two images has a relative displacement (u, v) of the relative images f _i , g _i Is equal to <Equation 1>.

Move vector as a parameter The motion field represented by = [u (x, y: ), v (x, y; )]. Is the biggest goal of image alignment Is max It is M to be vector ), M ( ) Is as follows.

M ( ) Becomes the maximum In operation 402, a newton method, which is a statistical technique, is obtained to find a normalized correlation between two images. In this case, the normalized correlation between the acquired two images is shown in Equation 3.

here, Is defined as

And, Is random Matrix M obtained from ₀ ( ) Is a gradient, Is random Matrix M obtained from ₀ ( Is Hessian.

here, Wow Obtained (403) Compute a vector of _zero (404) ₀ vectors are updated (405) to determine if the correlation is maximum (406) and the processes of 402 to 406 are repeated until the correlation is maximum.

Looking at the above process in more detail through the equation as follows.

In <Equation 3> Can be expressed as in Equation 4 by applying the approximate Newton method.

Here, in order to apply the Chain-Rule, each term of Equation 4 is expressed as follows.

And, by applying Equation 5 to Equation 4, the final The formula is as follows.

Where the term representing the correlation To Is redefined as the maximum correlation Can be obtained.

5 is an exemplary view illustrating image alignment by an image alignment method using statistical correlation according to the present invention.

The input image is an IR image shown in FIG. 5 (a) and an EO image shown in FIG. 5 (b). As shown in FIG. As described above, the embodiment of the present invention aligns the images by acquiring the outlines in order to fuse the distorted images due to the difference in the light sensing region of the image sensor. In particular, as shown in FIGS. 5C and 5D, different gray level thresholds are applied to each image sensor to obtain an outline.

After performing the image alignment method using the statistical correlation according to the present invention using the obtained contour, alignment can be made into a matched image as shown in FIG. 5 (f).

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the spirit of the present invention for those skilled in the art to which the present invention pertains, and the above-described embodiments and accompanying It is not limited by the drawings.

The present invention as described above, by providing an image alignment method using the statistical correlation between the images in the image alignment for each image input from the image sensor having different characteristics, so as to enable accurate alignment while reducing the overall calculation amount It is effective.

In addition, the present invention has the effect of obtaining a sufficient outline without performing the Laplacian operation in each direction by using the gray level threshold to extract the outline from the image.

1 is a block diagram of a general image input device for image fusion;

2 is an exemplary diagram showing a direction of a Laplacian operation for obtaining a conventional outline.

3 is an illustration of image alignment using feature points of an image.

4 is a flowchart illustrating an embodiment of a method for aligning an image using statistical correlation according to the present invention;

5 is a diagram illustrating an embodiment of image alignment by an image alignment method using statistical correlation according to the present invention;

Claims (4)

In the image alignment method using the statistical correlation, A first step of receiving images from different image sensors; Extracting an outline from the input image; And And statistically applying the correlation between the extracted contour lines of each image, and sorting the image to a value at which the correlation value is maximized. The method of claim 1, And the second step is performed through a Laplacian operation for extracting a high frequency component from the input image. The method of claim 2, And the Laplacian operation is performed according to the gray level threshold of each image sensor. The method according to any one of claims 1 to 3, The third step, A fourth step of setting an arbitrary motion vector and matrix for aligning the respective images; A fifth step of obtaining a normalized correlation between each of the images; A sixth step of taking a gradient of the matrix obtained from the arbitrary motion vector and a Hessian of the matrix obtained from the arbitrary moved vector; A seventh step of obtaining a difference between the motion vector obtained through the sixth step operation and the arbitrary motion vector; An eighth step of updating the arbitrary motion vector using the difference of the seventh step; And Repeating the fifth to eighth steps until the correlation value becomes the maximum, and aligning the image using the statistical correlation including the ninth step of sorting according to the corresponding motion vector when the correlation value becomes the maximum. Way.