KR20050104663A - Image fusion method for multiple image sonsor - Google Patents

Abstract

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to image fusion, and in particular, to a method for performing accurate image fusion using a small amount of computation. In order to provide a result of accurate image fusion with a small amount of computation, a feature level fusion method and a pixel level fusion method It is an object of the present invention to provide an improved image fusion method using a mixed image, comprising: a first step of receiving each image from different image sensors; A second step of extracting a feature image through a pyramid image from each of the input images; A third step of creating a combined image by selecting and combining a predominant image for each region of the extracted feature images; And a fourth step of combining and averaging each image area except for a feature image extracted from each image in the second step in pixel units and fusing the combined image of the third step.

Description

Image Fusion Method for Multiple Image Sonsor

TECHNICAL FIELD The present invention relates to image fusion, and more particularly to a method for performing accurate image fusion using a small amount of computation.

In recent years, image fusion from multiple image sensors combines image information from each single image sensor and allows processing of purpose-specific data from a database of image information of each combined image sensor, Research has been conducted in various fields as a technology for acquiring image information having accurate and various applications. In particular, the technical importance of image fusion technology from multiple image sensors is increasing day by day to build such improved accuracy and more sophisticated specific element information.

Here, the image fusion from multiple image sensors means that the optimal information is extracted according to the purpose of using the characteristic information of each wavelength from the image acquired in the unique wavelength band of each single image sensor. It is said to fuse. Technical fields of image fusion from such image sensors are computer vision, automatic object detection and identification, navigation, intelligent robotics, remote sensing, and medical devices.

Such image fusion can be roughly classified into three stages as a whole. Here, in step 3, the first step is to perform processing on the input image of each image sensor, and the second step is to align each pixel of the input image from each image sensor (alignment, registration). ), And the third step is a step of fusing the aligned images.

In particular, in the process of image fusion, if the image is synthesized in a state where the alignment of each image is not correct, the distortion of the image becomes more severe than the improvement of the image. Because of this phenomenon, research on how to align images (alignment, registration) has emerged as an important topic.

Hereinafter, a description will be given of a conventional technique for aligning an image.

In images obtained from image sensors with different characteristics (EO, IR, Radar, etc.), the correlation of brightness values corresponding to each pixel is most complicated.

That is, as shown in Table 1 below, since the photon flux level of each wavelength is different, the information included in the image of each sensor has various variables. .

VIS NIR MWIR LWIR Wavelength range (um) 0.4-0.78 0.78-1.0 3-5 8-12 Sunny daylight 1.5 x 10 ¹⁷ 1 x 10 ¹⁷ 2 x 10 ¹³ 2 x 10 ¹⁷ Moonlit night 1.5 x 10 ¹¹ - 2 x 10 ¹³ 2 x 10 ¹⁷ Night under the stars 1.5 x 10 ⁹ 9 x 10 ⁹ 2 x 10 ¹³ 2 x 10 ¹⁷

As shown in Table 1, the feature present in the visible light band sensor does not exist in the infrared region sensor. In addition, what is present in the infrared region sensor does not exist in the visible light band sensor or appears as weak signal data. This makes it difficult to know how two images from different sensors relate to the image's feature points. In practice, matching of feature points for each image may be performed after a number of steps in the preprocessing, and then the image may be registered.

1A to 1C illustrate exemplary embodiments of a conventional image fusion method for fusing an aligned image.

Image fusion is generally classified into three types: pixel unit fusion, feature unit fusion, and decision unit fusion.

1A is an exemplary diagram illustrating an embodiment of a pixel unit fusion method.

First, an image of an object 101 is received through a plurality of image sensors 102-1 through 102-N. The input image is fused for each pixel through the pixel level fusion unit 103. The feature extractor 104 extracts a feature from the fused image, and the identification unit 105 generates a result 106 of image fusion through the identification process of the image from which the feature is extracted.

1B is an exemplary diagram illustrating an embodiment of a method for fusion of feature units.

First, an image of an object 101 is received through a plurality of image sensors 102-1 through 102-N. The plurality of feature extractors 107-1 to 107 -N extracts features from the input images from the respective image sensors 102-1 to 102 -N. Then, fusion is performed for each feature through the feature level fusion unit 108 according to the features extracted from the feature extraction units 107-1 to 107 -N. The identification unit 109 generates a result 110 of image fusion through the identification process from the fused image.

1C is an exemplary diagram illustrating an embodiment of a crystalline unit fusion method.

First, an image of an object 101 is received through a plurality of image sensors 102-1 through 102-N. The plurality of feature extractors 107-1 to 107 -N extracts features from the input images from the respective image sensors 102-1 to 102 -N. Then, the identification unit 111-1 to 111-N performs an identification process according to the features extracted from each feature extraction unit 107-1 to 107-N. In the state where the identification of the image input from each of the image sensors 102-1 to 102 -N is made through the identification process, the decision level fusion unit 112 determines the result 113 of the image fusion through the determination process. Create

However, the image fusion by each of these methods causes problems such as excessive computation (pixel level fusion) or loss in the missing part (feature level fusion, crystal level fusion). do.

The present invention has been proposed to solve the above problems, and provides an improved image fusion method using a feature level fusion method and a pixel level fusion method in order to provide a result of accurate image fusion with a small amount of calculation. There is a purpose.

Another object of the present invention is to provide an image fusion method using a pyramid image having unsharp masking so that a feature image including low frequency information can be obtained and thus a clear feature image can be extracted. have.

According to an aspect of the present invention, there is provided an image fusion method for an image input from a plurality of image sensors, the method comprising: a first step of receiving each image from different image sensors; A second step of extracting a feature image through a pyramid image from each of the input images; A third step of creating a combined image by selecting and combining a predominant image for each region of the extracted feature images; And a fourth step of combining and averaging each image area except for a feature image extracted from each image in the second step in pixel units and fusing the combined image of the third step.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the same components in the drawings are represented by the same reference numerals and symbols as much as possible even though they are shown in different drawings. In addition, in describing the present invention, when it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

It is common for most images to simply represent an image only by the size (brightness) of the pixel value. However, the general characteristics of the image have many correlations between the surrounding pixels. If this is used, encoding using pixel correlation can be used as information with much utilization, rather than simply encoding with only pixel values.

Therefore, in the exemplary embodiment of the present invention, a pyramid image having global and local characteristics of the image will be described by using a multi-level image as a simple process on the peripheral value of one pixel. Using the pyramid image, it can be used as an effective information for image fusion, such as the edge (edge) component with a relatively small operation, it can be seen that it is very advantageous for real-time processing hardware implementation.

2 is an exemplary view of a Gaussian pyramid image used in the present invention.

When generating a multi-stage image as shown, a sub sampling image is obtained and used while generating a low pass image of the original image.

That is, the reduced primary image g1 is obtained while reducing the sample density and resolution in the first image g0. In a similar manner, the secondary image g2 is an image obtained by the bottom sampling of the primary image g1. At this time, the degree of high frequency filtering is changed by adjusting the weight function of the low pass filter. The weight function used here has a shape similar to a Gaussian distribution, which is called a Gaussian pyramid image.

3 is an exemplary view of a Laplacian Pyramid image used in the present invention.

Here, the value of the Laplacian pyramid is obtained by subtracting the Gaussian image g1 from the original image g0 as a Laplacian image. This is a relatively simple method, but it is a method of extracting the edge of a method that is easy to configure hardware and very convenient for real-time implementation.

0 <i <N in transplant The generation formula of the Laplacian pyramid at the level is shown in Equation 1.

L _i = g _i -EXPAND (g _{i + 1} )

= g _i -g _{i + 1,1}

Where L _i is a Laplacian image and g _i is a Gaussian image.

2 (a) to 2 (d) and 3 (a) to 3 (d), the first image of FIG. 2 (a) is the original image g ₀ . After zooming in to the same size and subtracting, the first Laplacian image is shown in FIG. 3 (a). Such a method can be applied between FIGS. 2 (b) and 3 (d).

4 is a flowchart illustrating an embodiment of an image fusion method according to the present invention.

The image fusion method according to the present invention can obtain the accuracy of the fusion image while reducing the amount of calculation by taking a dominant image for each feature image for the feature image, and taking image fusion in the pixel unit for the region excluding the feature image.

As shown in FIG. 4, first, an image is obtained from each of a plurality of image sensors. In an embodiment of the present invention, it can be seen that an image is received from two image sensors.

The feature images A and B are extracted from the received first image 41 and the second image 42 through feature extraction processes 43 and 44 using pyramid images. Then, the same weight is applied to each of the extracted feature images A and B (45).

In operation 46, the dominant feature image is selected from each feature image A and B to which the weight is applied.

In operation 47, a combined image combining the dominant feature images selected for each position is generated.

In operation 43 to 47, image combining is performed on the feature image (C).

On the other hand, in the portions other than the feature image from each of the received images (A, B), it is not possible to obtain a fusion image through the above 43 to 47 process. That is, since it is impossible to determine whether there is a dominance like the feature image, it is impossible to know which area to select.

Therefore, in steps 43 and 44, the average image of the images A 'and B' except for the feature images A and B is taken in the remaining regions except for the feature images A and B, and the image convergence image of the feature images generated through the above 47 processes The final fusion image D is generated by combining with (C) (48).

In the above image fusion process, the Laplacian pyramid image is important. The Laplacian pyramid is obtained as the difference from the image which passed the low frequency filter while the high frequency component is bottom-sampled from the original image. However, this inevitably results in the loss of important components of the image. In addition, in extracting the components of the contour, there is a limitation due to the loss of important components as described above.

Therefore, in order to achieve the image fusion, it is necessary to divide each high frequency region and the low frequency region image including as much information of the original image as possible, and to select the merits of each image and synthesize them into a new image, thus obtaining a general Laplacian pyramid image. There is a need for improvement.

An embodiment of the present invention proposes an improved image fusion method that applies non-sharpening mask processing in a Laplacian pyramid and adds an adaptive weight for obtaining a high frequency region faithful to the original image.

Here, the non-sharpening mask process represents the original image by multiplying the magnification factor represented by " A " to create the definition of the high frequency enhancement or high frequency enhancement filter. This will be described in Equation 2.

High boost = A (Original)-Lowpass

           = (A-1) (Original) + (Original-Lowpass)

           = (A-1) (Original) + Highpass

<Equation 2> is a general formula of general non-sharpening mask processing. Here, a value of A = 1 produces a standard high pass result. And, when A> 1, a part of the original image is added to the high pass result, which partially recovers the low frequency components lost in the high pass filter processing operation. As a result, the high frequency augmented image can include more information of the original image.

And the edge improvement of the original image depends on the value of A. As in the first line of Equation 2, a method in which a low frequency image is processed with A weights from an original image is called unsharp masking, and a general application is performed as in Equation 3.

Here, L (j, k) is a high frequency region image, G (j, k) is an oral image, and G _L (j, k) is a low pass region image. And, c is a weight constant, and generally ranges from "3/5" to "5/6".

In addition, if Equation 3 is expressed in another way, it is expressed as Equation 4.

Since the main purpose of image fusion is to synthesize each of the advantages of the original image to the maximum, it is very important to maintain the information of the original image to the maximum.

Therefore, in order to be as faithful as possible to the original image in Equation 4, the original image G (j, k) and the high frequency image L (j, k) are used as the terms having the adaptive weight value of the pyramid contour image and the original image. Obtain the product image of As a result, the compensation of the image for the loss of information of the image due to down sampling occurs to some extent.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the spirit of the present invention for those skilled in the art to which the present invention pertains, and the above-described embodiments and accompanying It is not limited by the drawings.

As described above, the present invention combines feature level fusion, which does not have a large amount of computation in image fusion, and accurate pixel level fusion, which has a large amount of computation, but performs image fusion through feature extraction in a region where a feature can be extracted. In regions where features cannot be extracted, pixel level fusion is performed to enable accurate image fusion without greatly increasing the amount of computation.

In addition, in the feature extraction, by using a pyramid image subjected to unsharp masking, a feature image including low frequency information can be obtained, whereby a clear feature image can be extracted. .

1A-1C illustrate one embodiment description of a conventional image fusion method for fusing an aligned image.

2 is an illustration of a Gaussian Pyramid image used in the present invention.

3 is an illustration of a Laplacian Pyramid image used in the present invention.

4 is a flowchart illustrating an embodiment of an image fusion method according to the present invention;

Claims (4)

In the image fusion method for an image input from a plurality of image sensors, A first step of receiving each image from different image sensors; A second step of extracting a feature image through a pyramid image from each of the input images; A third step of creating a combined image by selecting and combining a predominant image for each region of the extracted feature images; And A plurality of image sensors including a fourth step of combining and averaging each image area except for a feature image extracted from each image in the second step by a pixel unit and fusing the combined image of the third step; Image fusion method. The method of claim 2, And the pyramid image is a Laplacian pyramid image for extracting high frequency components from the input image. The method of claim 3, wherein Unsharp masking the Laplacian pyramid image to extract a feature having a large amount of information. The method of claim 1, The third step, A fifth step of applying the same weight to each of the extracted feature images; A sixth step of selecting a dominant image in each region from the weighted feature image; And And a seventh step of combining the predominant images for each of the selected regions to form a combined image.