KR102170260B1 - Apparatus and method for fusing synthetic aperture radar image and multispectral image, method for detecting change using it - Google Patents

Abstract

Disclosed are an apparatus and a method for fusing a synthetic aperture radar image and a multispectral image, and a method for detecting a change using the same. The method for fusing the synthetic aperture radar image and the multispectral image, according to one embodiment of the present application, comprises the steps of: processing a multispectral image; processing a synthetic aperture radar image; determining a training pixel from the multispectral image and the synthetic aperture radar image; learning a correlation between the multispectral image and the synthetic aperture radar image based on color information of the multispectral image corresponding to the training pixel and feature information of the synthetic aperture radar image; and colorizing the synthetic aperture radar image by fusing the multispectral image and the synthetic aperture radar image, based on the learned correlation.

Description

Synthetic aperture radar image and multispectral image fusion device, method, and change detection method using the same {APPARATUS AND METHOD FOR FUSING SYNTHETIC APERTURE RADAR IMAGE AND MULTISPECTRAL IMAGE, METHOD FOR DETECTING CHANGE USING IT}

The present application relates to an apparatus and method for fusion of a composite aperture radar image and a multispectral image, and a change detection method using the same.

In order to perform change detection using a satellite image captured in the same area, an image captured under the same conditions must be used, but in the case of an optical satellite image, it is difficult to acquire an image at a required time due to the shooting period, atmospheric conditions, and weather conditions. There is this. In particular, when the image information of Dajigi is used, it contains sensor characteristics and radially and seasonally different characteristics depending on the timing of acquisition. If the correction of such characteristics is not performed, an error of determining the object as a changed area may be made even though the object is not actually changed.

In addition, change detection is a process of analyzing images using satellite images captured at different times of the same area and monitoring changes in the area that occur naturally or generated by human activities. In order to perform such change detection with high accuracy, satellite images, which are the basis of change detection, must be close to each other in time and be able to reflect rich features. However, when only one type of image is considered (for example, when only a composite aperture radar image is used or only optical data is used), this requirement is difficult to meet. Recently, artificial satellites equipped with observation sensors that provide various resolutions and performance levels have been made and operated, and various sensors are being prepared to overcome the limitations of images subject to change detection.

Specifically, optical data (optical image) can reflect relatively rich information on reflection characteristics and radiation characteristics, and thus can be easily interpreted and used as main data for change detection. However, optical data is relatively largely affected by atmospheric conditions such as the presence of clouds during photographing, so it is difficult to overcome the temporal constraint for use in detecting changes. Conversely, radar sensors (eg, synthetic aperture radars) are less affected by weather conditions and are not affected by weather conditions through the provision of their own light source of long wavelengths that pass through the atmosphere. Furthermore, radar sensors are highly sensitive to backscattering of terrain and objects, and enable consistent imaging in terms of amplitude and phase. However, it has a limitation that it is relatively difficult to interpret when compared to an optical image due to speckle noise, oblique range imaging, layer over, influence by shadow, or limited band.

In this regard, in order to utilize various remote sensing data for change detection, it may be considered to complementarily use an image and optical image data by a Synthetic Aperture Radar (SAR) or fuse two different types of data. have. However, the conventional image fusion method has not narrowed the difference in imaging mechanism between two different types of data, and has not overcome the temporal limitation of optical image data used in change detection.

In particular, the conventional image fusion method has a relatively large difference in gray value from the existing image due to a significant difference between the imaging mechanism of the composite aperture radar image and the optical image, and cannot be ignored in the fused image. There is a disadvantage of causing color distortion.

The technology behind the present application is disclosed in Korean Patent Application Publication No. 10-1693705.

The present application is to solve the above-described problems of the prior art, by searching the correlation between the composite aperture radar image and the optical image (especially, multispectral image) and learning it, based on the fusion image of the composite aperture radar image and the optical image. An object of the present invention is to provide an apparatus for fusion of a composite aperture radar image and a multispectral image to colorize a composite aperture radar image.

The present application is to solve the problems of the prior art described above, and an object thereof is to provide a method and system for detecting changes through fusion of a composite aperture radar image and a multispectral image.

However, the technical problem to be achieved by the embodiment of the present application is not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the fusion method of a composite aperture radar image and a multispectral image according to an embodiment of the present application includes processing a multispectral image, processing a composite aperture radar image, the Determining a training pixel from the multispectral image and the composite aperture radar image, correlation between the multispectral image and the composite aperture radar image based on color information of the multispectral image corresponding to the training pixel and characteristic information of the composite aperture radar image And colorizing the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learning relationship and the learned correlation.

In addition, the learning of the correlation may include inputting characteristic information of a pixel corresponding to the position of the training pixel in the composite aperture radar image and color information of a pixel corresponding to the position of the training pixel in the multispectral image. It may be to learn a correlation between a multispectral image and a composite aperture radar image using a random forest model.

Further, the processing of the multispectral image may include extracting pixel values of an RGB band from the multispectral image and classifying the multispectral image into a plurality of classes.

In addition, the random forest model may be generated for each of the plurality of classes.

In addition, the learning of the correlation may be performed using each of the random forest models generated for each of a plurality of classes.

In addition, the step of classifying the multispectral image into a plurality of classes may include classifying the multispectral image into a plurality of classes by performing a K average clustering algorithm as many times as a preset repetition number.

In addition, the processing of the composite aperture radar image may include extracting pixel information from the composite aperture radar image, and generating a gray level co-occurrence matrix (GLCM) in each pixel of the composite aperture radar image. And extracting the characteristic information of the composite aperture radar image from the GLCM.

In addition, the determining of the training pixel may include differentiating pixel values of the composite aperture radar image and the multispectral image, and determining an unchanged pixel as the training pixel based on the difference result of the pixel values. have.

In addition, the step of colorizing the composite aperture radar image may include predicting color information in pixels other than the training pixels of the composite aperture radar image, based on the learned correlation, and based on the predicted color information. It may be to colorize the composite aperture radar image.

In addition, the multispectral image and the composite aperture radar image may be acquired at different times.

On the other hand, the change detection method based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application includes the steps of colorizing the composite aperture radar image using a fusion method of a composite aperture radar image and a multispectral image. And detecting a change in an area to be photographed based on the colored composite aperture radar image and the multispectral image.

In addition, the multispectral image and the composite aperture radar image may be acquired at different times.

On the other hand, according to an embodiment of the present application, the fusion device of the composite aperture radar image and the multispectral image, a multispectral image processing unit that extracts color information from the multispectral image and classifies the multispectral image into a plurality of classes, and A composite aperture radar image processing unit that extracts pixel information and characteristic information from an aperture radar image, a training pixel setting unit that determines a training pixel from the multispectral image and the composite aperture radar image, and the multispectral image corresponding to the training pixel. A learning unit that learns the correlation between the multispectral image and the composite aperture radar image based on the color information and the characteristic information of the composite aperture radar image, and the multispectral image and the composite aperture radar image based on the learned correlation. It may include a colorization unit for colorizing the composite aperture radar image by fusion.

In addition, according to an embodiment of the present disclosure, the apparatus for fusion of a composite aperture radar image and a multispectral image includes a change detection unit that detects a change in a region to be photographed based on the colored composite aperture radar image and the multispectral image. can do.

In addition, the learning unit may generate a random forest model for inputting characteristic information of a pixel corresponding to the position of the training pixel in the composite aperture radar image and color information of a pixel corresponding to the position of the training pixel in the multispectral image. Using this, it is possible to learn the correlation between the multispectral image and the composite aperture radar image.

In addition, the colorization unit predicts color information in pixels other than the training pixels of the composite aperture radar image based on the learned correlation, and colorizes the composite aperture radar image based on the predicted color information. It can be.

The above-described problem solving means are merely exemplary and should not be construed as limiting the present application. In addition to the above-described exemplary embodiments, additional embodiments may exist in the drawings and detailed description of the invention.

According to the above-described problem solving means of the present application, by acquiring and learning the correlation between the composite aperture radar image and the optical image (especially, multispectral image), a composite aperture radar based on the fusion image of the composite aperture radar image and the optical image. There is an effect of providing an apparatus for fusion of a composite aperture radar image and a multispectral image to colorize an image.

According to the above-described problem solving means of the present application, it is possible to provide a method and system for detecting changes through fusion of a composite aperture radar image and a multispectral image.

According to the above-described problem solving means of the present application, unlike the conventional image fusion technique, image fusion can be performed even based on images with a time difference at the captured point of view, and the fused image can be utilized for change detection. There is.

However, the effect obtainable in the present application is not limited to the effects as described above, and other effects may exist.

1 is a schematic configuration diagram of a change detection system based on a fusion image of a composite aperture radar image and a multispectral image, including a fusion device for a composite aperture radar image and a multispectral image according to an embodiment of the present application.
2 is a diagram showing a composite aperture radar image and a multispectral image according to an embodiment of the present application.
3 is a view showing a comparison of a fusion result of a synthetic aperture radar image and a multispectral image based on a random forest model and a fusion result by a conventional image fusion technique or a model based on another regression model according to an embodiment of the present application.
4 is a diagram illustrating a result of fusion of a composite aperture radar image and a multispectral image according to the number of classes according to an exemplary embodiment of the present application.
5 is a diagram illustrating a change detection result based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application.
6 is a chart in which a change detection result based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application is evaluated through a statistical index.
7 is a schematic configuration diagram of an apparatus for fusion of a composite aperture radar image and a multispectral image according to an embodiment of the present application.
8 is an operation flow diagram of a method for fusion of a composite aperture radar image and a multispectral image according to an embodiment of the present application.
9 is an operation flow diagram of a method of processing a composite aperture radar image according to an embodiment of the present application.
10 is a flowchart illustrating an operation of a method for detecting a change based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present disclosure.

Hereinafter, exemplary embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present application. However, the present application may be implemented in various different forms and is not limited to the embodiments described herein. In addition, in the drawings, parts not related to the description are omitted in order to clearly describe the present application, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the present specification, when a part is said to be "connected" with another part, it is not only the case that it is "directly connected", but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including the case.

Throughout this specification, when a member is positioned "on", "upper", "upper", "under", "lower", and "lower" of another member, this means that a member is located on another member. It includes not only the case where they are in contact but also the case where another member exists between the two members.

Throughout the specification of the present application, when a certain part "includes" a certain component, it means that other components may be further included rather than excluding other components unless specifically stated to the contrary.

1 is a schematic configuration diagram of a change detection system based on a fusion image of a composite aperture radar image and a multispectral image, including a fusion device for a composite aperture radar image and a multispectral image according to an embodiment of the present application.

Referring to FIG. 1, a change detection system 10 based on a fusion image of a composite aperture radar image and a multispectral image includes a fusion device 100 for a composite aperture radar image and a multispectral image (hereinafter referred to as ``image fusion device 100 )'.), a composite aperture radar 200, a multispectral imaging apparatus 300, and a network 20.

The synthetic aperture radar 200 (Synthetic Aperture Radar, SAR) may be a radar that observes the ground and the ocean from the air. Specifically, the composite aperture radar 200 refers to a radar system that generates a ground topographic map by sequentially firing radio signals on the ground and the ocean, and then synthesizing the time difference in which the radio signals are reflected from the observation area and return in a first-come-first-served basis. I can. In addition, the composite aperture radar image is an image created through the radar system. The composite aperture radar image may be produced through the radar system not only in normal time, but also in day and night, and in bad weather.

According to an embodiment of the present application, the composite aperture radar 200 may be a radar provided in the multipurpose satellite KOMPSAT-5. For reference, KOMPSAT-5 has three imaging modes: standard mode (ST mode, 3m, 30km swath), high resolution mode (HR mode, 1m, 5km swath), and wide water mode (WS mode, 20m, 100km swath). The composite aperture radar image according to an embodiment of the present application is taken in a standard mode (ST mode) that provides parameters of a spatial resolution of 3m, a descending orbit, and HH polarization of KOMPSAY-5. It may have been.

The multispectral imaging apparatus 300 may be, for example, a Landsat-8 Operational Land Imager (OLI). The multispectral image according to an embodiment of the present application has a spatial resolution of 30 m, and 7 bands (Band 1: 0.433-0.453 μm, Band 2: 0.450-0.515 μm, Band 3: 0.525-0.600 μm, Band 4: 0.630- 0.680 μm, band 5: 0.845-0.885 μm, band 6: 1.560-1.660 μm, band 7: 2.100-2.300 μm) may be photographed by the multispectral imaging apparatus 300 supporting.

The image fusion device 100 may communicate with the composite aperture radar 200 and the multispectral imaging device 300 through the network 20. The network 20 refers to a connection structure in which information exchange is possible between respective nodes such as terminals and servers, and examples of such a network 20 include a 3rd Generation Partnership Project (3GPP) network, and a Long Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, and a Digital Multimedia Broadcasting (DMB) network, but are not limited thereto.

Hereinafter, the process of fusing the composite aperture radar image and the multispectral image by the image fusion device 100 will be described in detail.

First, the image fusion device 100 may acquire a composite aperture radar image captured by the composite aperture radar 200 through the network 20.

In addition, the image fusion device 100 may acquire a multispectral image captured by the multispectral image capturing apparatus 300 through the network 20.

2 is a diagram showing a composite aperture radar image and a multispectral image according to an embodiment of the present application.

2A is a composite aperture radar image according to an embodiment of the present application, and (b) may be a multispectral image according to an exemplary embodiment of the present application.

Further, according to the exemplary embodiment of the present disclosure, the multispectral image and the composite aperture radar image may be acquired at different times. For example, the composite aperture radar image (a) of FIG. 2 may be acquired on August 21, 2014. In addition, the multispectral image (b) of FIG. 2 may be acquired on September 24, 2016. In other words, according to an embodiment of the present application, the multispectral image may be an image acquired (photographed) at a first point of view, and the composite aperture radar image is an image acquired (photographed) at a second point different from the first point of view. Can mean It goes without saying that the temporal relationship between the first viewpoint and the second viewpoint may vary according to embodiments.

Specifically, the composite aperture radar image photographed at the second point of view is, in the case of performing detection of a change in the area to be photographed based on the multispectral image photographed at the first viewpoint and the multispectral image photographed at the second viewpoint, the To replace the multispectral image captured at the second time point when the multispectral image captured at the second time point is not suitable for use in detecting changes due to atmospheric conditions such as the presence of clouds or other unforeseen causes. Can be utilized.

In other words, it is a fusion image of a multispectral image captured at different times and a composite aperture radar image by the image fusion method described later, and even if there is a problem with the multispectral image at a specific viewpoint, the multispectral image at that specific viewpoint is replaced. can do.

The image fusion device 100 may process a multispectral image. According to an embodiment of the present application, the multispectral image may be photographed at the first point of view. First, the image fusion apparatus 100 may perform pre-processing based on a dark-object substraction algorithm on a multispectral image in order to correct an effect of the atmospheric condition. In addition, according to the exemplary embodiment of the present disclosure, the image fusion device 100 may resample the resolution of the multispectral image to 15m so that the multispectral image has the same resolution as the composite aperture radar image.

Also, the image fusion device 100 may extract pixel values of an RGB band from a multispectral image. In other words, the image fusion device 100 may extract only visible spectrums of red (R), green (G), and blue (B) bands from the multispectral image. The pixel value of the RGB band may be understood to mean color information of a multispectral image throughout the present specification. In addition, the multispectral image may be photographed at the first point of view, and therefore, the color information may reflect a characteristic (color) of a region to be photographed at the first point of view.

Also, the image fusion device 100 may classify a multispectral image into a plurality of classes. In order to reduce the complexity of the entire image fusion algorithm and to ensure high prediction accuracy, the image fusion device 100 does not fusion the multispectral image and the composite aperture radar image as a whole, but multispectral multiple classes into a plurality of classes. An image fusion algorithm (for example, an algorithm for learning characteristics of two images) to be described later may be applied independently for each of a plurality of classes by dividing an image.

Specifically, the image fusion apparatus 100 may classify a multispectral image into a plurality of classes by performing a K-average clustering algorithm as many times as a preset repetition number. The K-means clustering algorithm is a centroid-based technique that establishes a model for data classification based on the cluster center, and is intended to minimize the sum of the distances between each pixel (point) for the center point vector. Models can be modified iteratively until results are achieved. That is, parameters applied to the K-means clustering algorithm may be the number of classes and the number of iterations.

According to an embodiment of the present application, the number of optimal classes may be determined in consideration of learning performance including learning time of an algorithm for learning characteristics of two images and information related to land cover distribution characteristics, The minimum number of classes determined based on this may be six.

In addition, according to an embodiment of the present disclosure, when the iteration of the K-means clustering algorithm exceeds 20, there is no significant improvement in learning performance, so the maximum number of iterations may be determined as 20.

The image fusion device 100 may process a composite aperture radar image. According to the exemplary embodiment of the present application, the composite aperture radar image may be photographed at the second viewpoint. First, the image fusion device 100 may perform image preprocessing such as multi-looking, speckle filtering, and terrain correction. For example, multi-looking may be performed based on three range looks and six azimuth looks in order to reduce speckle noise. In addition, speckle filtering may be performed through a 3x3 kernel gamma map filter in order to reduce speckle noise while maintaining the edge of the object. In addition, terrain correction may be performed in connection with a shuttle radar topography mission (SRTM), and a bilinear interpolation method may be applied together.

In addition, the image fusion device 100 may extract pixel information from the composite aperture radar image. The pixel information may mean intensity information of the composite aperture radar image, and each pixel of the composite aperture radar image may be mapped with each pixel of the multispectral image through the pixel information. In the conventional image fusion technique, only pixel information from a composite aperture radar image is considered, but when only pixel information is considered, it is difficult to expect a prediction result with high accuracy. Accordingly, the image fusion apparatus 100 according to the exemplary embodiment of the present disclosure may utilize the characteristic information (especially, including material characteristics, roughness characteristics, etc.) obtained through GLCM together with pixel information for image fusion. In addition, the composite aperture radar image may be photographed at the second viewpoint, and thus the pixel information and the characteristic information may reflect characteristics at the second viewpoint of the photographing target area. Hereinafter, a process of obtaining the characteristic information based on the GLCM will be described in detail by the image fusion device 100.

The image fusion device 100 may generate a gray level co-occurrence matrix (GLCM) in each pixel of the composite aperture radar image. The GLCM may refer to a matrix representing spatial characteristics of a pixel through statistical values between the pixel and adjacent pixels for one pixel.

Also, the image fusion device 100 may extract characteristic information of the composite aperture radar image from the GLCM. Specifically, the image fusion device 100 is a GLCM technician such as contrast, dissimilarity, homogeneity, correlation, angular second moment (ASM), and energy from GLCM. At least one can be obtained.

Specifically, contrast may represent the depth and softness of the texture structure of the composite aperture radar image. The dissimilarity is a GLCM descriptor similar to contrast, but the contrast increases exponentially (e.g., 0,1,4,9, etc.) While it is a value, there is a difference that the dissimilarity increases linearly (eg 0,1,2,3, etc.). Homogeneity measures the smoothness of the texture of the composite aperture radar image, and when the spectrum changes significantly in neighboring pixels, the value is measured to be small. Conversely, when the spectrum changes to small, the value can be measured large. The correlation may reflect the similarity in the horizontal or vertical direction of the texture of the composite aperture radar image. ASM can measure the regularity and uniformity of the image distribution, and energy can be calculated as the square root of the ASM.

In addition, in order to calculate each GLCM technician from the GLCM, a window of a specific size must be set in the composite aperture radar image. According to an embodiment of the present application, in the case of a 5x5 size window, detailed texture (characteristic information) Can reflect well.

In addition, according to an exemplary embodiment of the present application, in order to clearly measure all characteristic information, the image fusion device 100 may perform image co-register and terrain correction prior to extraction of characteristic information through GLCM. have.

The image fusion device 100 may determine a training pixel from the multispectral image and the composite aperture radar image.

In this regard, the image fusion device 100 may differentiate between pixel values of the composite aperture radar image and the multispectral image.

Also, the image fusion apparatus 100 may determine an unchanged pixel as a training pixel based on a result of difference between pixel values. Specifically, the image fusion device 100 determines a region in which spectral reflectance has changed only below a predetermined level (that is, a region that has not changed substantially) based on the difference result of pixel values as the unchanged pixel, This can be determined as a training pixel.

In addition, according to an embodiment of the present application, before determining the training pixel, the image fusion device 100 performs a luminosity method for matching the dimensions of the composite aperture radar image and the multispectral image. Can be applied. The photometric method may be performed by Equation 1 below.

[Equation 1]

The image fusion apparatus 100 may learn a correlation between the multispectral image and the composite aperture radar image based on color information of the multispectral image corresponding to the training pixel and characteristic information of the composite aperture radar image. In addition, the multispectral image may be photographed at the first viewpoint, and the composite aperture radar image may be photographed at the second viewpoint. Therefore, learning the correlation between the multispectral image and the composite aperture radar image is to grasp the color information of the first point of view from the multispectral image, grasp the characteristic information of the second point of view from the composite aperture radar image, It can mean to derive a correlation of. Further, when the correlation between the multispectral image and the composite aperture radar image is learned, color information of the second view may be predicted through color information of the first view and characteristic information of the second view.

According to an exemplary embodiment of the present application, the image fusion device 100 inputs characteristic information of a pixel corresponding to a position of a training pixel in a composite aperture radar image and color information of a pixel corresponding to a position of a training pixel in a multispectral image. The correlation between the multispectral image and the composite aperture radar image can be learned using a random forest (RF) model.

The random forest (RF) technique may refer to an ensemble learning method (learning algorithm) in which a number of independent trees are created in connection with a classifier or a regressor, and results are obtained through statistical analysis. The random forest model is suitable for modeling complex relationships identified in a large number of data and for modeling nonlinear relationships between predictors and response variables. In the case of the classifier, the modeling result is obtained using a number of tables determined from the results of individual trees, and in the case of the regressor, the modeling result is obtained using the average of the tree. Specifically, the regression tree is used as a basic learner in random forest-based regression analysis, and each regression tree is rearranged in a separate bootstrap sample derived from the existing data set (bootstrap aggregation, bagging). In other words, since results are obtained through an independent tree and then the results are aggregated, stability is high, the effect of noise on the data is relatively small, and excessive errors can be prevented. In addition, the performance of the random forest model is evaluated using some tree data (Out-of-Bag Data; OOB data) excluded from the training target.

In sum, random forest can perform modeling with high accuracy when the relationship between the two data is rather complex or nonlinear, and has advantages in terms of high prediction performance and stability, making it suitable for application in the field of remote sensing (change detection). It is a learning model.

According to an embodiment of the present application, a random forest model may be generated for each of a plurality of classes classified in a process of processing a multispectral image. In addition, the image fusion apparatus 100 may perform correlation learning using each of the random forest models generated for each of a plurality of classes. As the image fusion device 100 independently performs learning using a random forest model for each of a plurality of classes, the complexity of the entire algorithm (learning) may be reduced, and richer information may be retrieved from a multispectral image. Specifically, each of the random forest models generated for each of the plurality of classes uses the characteristic information (especially, material, roughness characteristics) of the composite aperture radar image in the training pixel and the pixel value (color information) in the RGB band of the multispectral image. Carry out learning.

In addition, according to an embodiment of the present application, the number of trees for random forest modeling may be determined in consideration of the performance including the training time of the random forest model, and the number of trees is exemplarily 32. Can be determined.

The image fusion device 100 may colorize the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learned correlation.

In more detail, the image fusion device 100 may predict color information in pixels other than the training pixels of the composite aperture radar image (eg, pixels excluding the training pixels from all pixels) based on the learned correlation. .

In addition, the image fusion device 100 may colorize the composite aperture radar image based on the predicted color information. In addition, when the multispectral image is captured at the first point of view, and the composite aperture radar image is taken at the second point of view, the predicted color information is related to the correlation between the multispectral image and the synthesized aperture radar image. Based on the color information of the first viewpoint and the characteristic information of the second viewpoint, it may mean color information of the second viewpoint.

Specifically, according to an embodiment of the present application, color information (eg, RGB value) of a position corresponding to a training pixel that is an unchanged pixel among all pixels of a composite aperture radar image is corresponding to a training pixel of a multispectral image. It can be determined by the pixel value of the RGB band of the location, and the color information (RGB value) of a region other than the training pixel among all pixels of the composite aperture radar image will be determined as predicted color information (RGB value) based on the learned correlation. I can.

3A is a view showing a comparison between a fusion result of a synthetic aperture radar image and a multispectral image based on a random forest model and a fusion result by a conventional image fusion technique or a model based on another regression model according to an embodiment of the present application.

3A, (a) is a reference multispectral image, (b) a fusion image of a composite aperture radar image and a multispectral image by the image fusion device 100 according to an embodiment of the present application, and (c) is an improvement. Convergence image by the intensity-hue-saturation (IHS) technique, (d) is a fusion image by principal component analysis (PCA) technique, (e) is a fusion image by Gram-Schmidt (GS) technique, (f) Denotes a fusion image using the Ehlers method, (g) a fusion image using a stochastic gradient boosting (SGB) regression model, and (h) a fusion image using an adaptive boosting (AdaBoost) regression model, respectively. The quality of the fusion image of the composite aperture radar image and the multispectral image can be visually confirmed through the fusion image comparison shown in FIG. 3.

3A, the fusion image of the composite aperture radar image and the multispectral image by the image fusion device 100 according to an embodiment of the present invention is more than when a single composite aperture radar image or a single multispectral image is used. It can be confirmed that it contains information. In particular, the fusion image according to the present application may reflect not only characteristic information of the composite aperture radar image but also color information of the multispectral image.

Specifically, the fusion image using the improved intensity-hue-saturation (IHS) technique in (c), the fusion image using the principal component analysis (PCA) technique in (d), and the Gram-Schmidt(GS) technique in (e) In the case of the fusion image by fusion, it can be seen that the color distortion is large, and the spectrum preservation is not performed properly. However, the fusion image is generally blurred and detailed characteristics may not be properly reflected.

In addition, in the case of the fusion image based on the stochastic gradient boosting (SGB) regression model in (g), it can be seen that the color of the background area is well expressed, but the color of the ground or the ground is not properly expressed, and the adaptive boosting of (h) (AdaBoost) In the case of a fusion image based on a regression model, the overall color may be slightly distorted with a reddish hue, and green noise may appear in the part mainly reflecting the characteristics of the composite aperture radar image.

On the other hand, the fusion image of the composite aperture radar image and the multispectral image by the image fusion device 100 according to the embodiment of the present application of (b) has relatively little color distortion, and the spectral characteristics of the multispectral image are well maintained, It can be seen that characteristic information (especially, material characteristics or roughness characteristics) of the composite aperture radar image is well reflected.

Referring to FIG. 3A, visual inspection of a fused image is relatively easy and intuitive, but in order to accurately evaluate the actual effect of the image fusion device 100, quantitative analysis based on statistical values must be performed in parallel.

Regarding the statistical technique for quantitatively evaluating the fused image, the UIQI (universal image quality index) is a fusion image of a composite aperture radar image and a multispectral image generated by the image fusion device 100 It can be calculated using a combination of luminance distortion, contrast distortion, and correlation loss between multispectral images.

Specifically, UIQI is measured through Equations 2-1 to 2-4 below.

[Equation 2-1]

[Equation 2-2]

[Equation 2-3]

[Equation 2-4]

The CC (correlation coefficient) may represent a correlation between a fusion image of a composite aperture radar image and a multispectral image and a reference multispectral image, and may be determined by Equation 3 below.

[Equation 3]

In Equations 2-1 to 3, μ _x and μ _y are the average values of the fusion image and the reference multispectral image of the composite aperture radar image and the multispectral image, and σ _x and σ _y are the composite aperture radar image and the multispectral image. The standard deviation value, σ _xy , of the fusion image and the reference multispectral image may mean the covariance of the two images, respectively.

The above-described UIQI and CC have a range of [-1,1], and the closer to 1 may indicate a better quality.

In addition, entropy and cumulative probability of blur detection (CPBD) for evaluating the spatial quality of the fusion image of the composite aperture radar image and the multispectral image and the reference multispectral image are considered. I can.

Entropy may display average information included in an image and reflect detailed information of the fusion image. In general, the greater the entropy of the fusion image, the richer the information contained in the image and the higher the quality of the fusion can be evaluated. Entropy can be calculated by Equation 4 below.

[Equation 4]

Here, P _i may mean the i- th probability in the image.

Further, the cloud detection cumulative probability (CPBD) may be calculated based on the sensitivity of the observer's blur recognition at different contrasts. Specifically, after estimating the probability of detecting blur at each edge of the image, the entire image may be pooled over to obtain a final quality evaluation score. Given a contrast C value, the cloud detection cumulative probability (CPBD) can be calculated by the following equations 5-1 and 5-2.

[Equation 5-1]

[Equation 5-2]

Where w(e _i ) is the width of the measured edge e _i , w _JNB (e _i ) is the visible bleeding width determined depending on the local contrast C value, and β is a parameter obtained by the least square sum, P (P _BLUR) is, it is possible to sense the probability distribution function value for a given P _BLUR, CPBD has a range of [0, 1], can be closer to 1 means that a better image quality.

3B is a fusion result of a synthetic aperture radar image and a multispectral image based on a random forest model and a fusion result based on a conventional image fusion technique or another regression model in terms of UIQI and CC values according to an embodiment of the present application. This is a comparison chart.

Referring to FIG. 3B, the UIQI value for the fusion image of the composite aperture radar image and the multispectral image by the image fusion device 100 according to an embodiment of the present application is based on the improved intensity-hue-saturation (IHS) technique. Fused image by principal component analysis (PCA), fused image by Gram-Schmidt (GS) method, and fused image by Ehlers method in both RGB bands (0.7794 in R band, It can be seen that 0.7388 in the G band and 0.7622 in the B band).

In addition, referring to FIG. 3B, the CC value for the fusion image of the composite aperture radar image and the multispectral image by the image fusion device 100 according to an embodiment of the present application is improved intensity-hue-saturation (IHS). The value closest to 1 in both RGB bands than the fusion image by the method, the fusion image by the principal component analysis (PCA) method, the fusion image by the Gram-Schmidt (GS) method, and the fusion image by the Ehlers method (in the R band). 0.8178, 0.7794 in the G band, and 0.7622 in the B band).

That is, when the image fusion device 100 according to the exemplary embodiment of the present application is used, it can be confirmed that the quality of the fusion image is improved quantitatively than in the conventional image fusion technique.

4 is a diagram illustrating a result of fusion of a composite aperture radar image and a multispectral image according to the number of classes according to an exemplary embodiment of the present application.

As described above, the number of optimal classes may be determined in consideration of information related to the learning performance including the learning time of the random forest model and the land cover distribution characteristics, and according to an embodiment of the present application, Accordingly, the minimum number of classes may be 6 and the maximum number of classes may be 10. FIG. 4 may be a view showing the learning time required by the random forest model by evaluating UIQI, CC, Entropy, and CPBD of a fusion image of a composite aperture radar image and a multispectral image according to the number of classes from 6 to 10.

Referring to FIG. 4, it can be seen that the learning time increases as the number of classes increases, but the performance (UIQI, CC, Entropy, CPBD values) according to the increase in the number of classes slightly increases or decreases. . This can be interpreted as having a possibility of causing overloading in the process of fusing images in the number of a large number of classes, and thus the optimal number of classes may be determined to be six in a preferred embodiment of the present application.

In addition, the image fusion apparatus 100 may detect a change in a region to be photographed based on a colored composite aperture radar image and a multispectral image. According to an embodiment of the present application, the multispectral image may be an image acquired (photographed) at a first point of view, and the colored composite aperture radar image is a synthesized aperture acquired (photographed) at a second point different from the first point of view. It may mean an image obtained by colorizing a radar image based on a multispectral image acquired at the first point of view. In this case, the temporal relationship between the first viewpoint and the second viewpoint may vary according to embodiments.

That is, when the multispectral image captured at the second point of view is not properly photographed to be used for change detection due to atmospheric conditions such as the presence of clouds or other unforeseen causes, the composite aperture radar image captured at the second point of view A colorized image based on the multispectral image acquired at the first point of view may replace the multispectral image captured at the second point of view to perform change detection of the region to be photographed.

In other words, according to the conventional method of detecting changes using the image fusion technique, the difference in the imaging mechanism of the composite aperture radar image and the multispectral image cannot be overcome, so that the composite aperture radar image and the multispectral image photographed close to each other in time cannot be overcome. Since only the fusion of the image was possible, when the multispectral image captured at the second viewpoint was not properly photographed to be used for change detection due to atmospheric conditions or other unforeseen causes, the composite aperture radar photographed at the second viewpoint Even if an image was secured, it was not possible to perform fusion with the multispectral image captured at the first viewpoint that was not close to the photographing viewpoint. Image fusion between images is possible, and further, there is an effect applicable to the change detection process.

According to an exemplary embodiment of the present disclosure, the image fusion apparatus 100 may perform histogram matching, which is generally used as a radiometric normalization technique, prior to detecting a change in a region to be photographed.

According to an exemplary embodiment of the present disclosure, the image fusion apparatus 100 may perform at least one of pixel-based change detection or object-based change detection based on a colored composite aperture radar image and a multispectral image.

According to an embodiment of the present application, the pixel-based change detection is a change detection method of detecting a pixel that has changed over time by comparing a composite aperture radar image colored by the image fusion device 100 with a multispectral image. It can mean the way. In this regard, an image difference technique may be used. Specifically, the image difference technique matches two images (a colored composite aperture radar image and a multispectral image) to a common coordinate system, subtracting the pixel values of the two images, and A changed pixel may be detected by setting a threshold pixel difference for distinguishing unchanged pixels. According to an exemplary embodiment of the present disclosure, an Otsu's threshold generally used based on a brightness distribution of an input image may be considered in order to set the threshold pixel difference.

However, the pixel-based change detection may have limitations such as derivation of fragmentation or incomplete change detection results.

In addition, according to an exemplary embodiment of the present disclosure, the object-based change detection may refer to a method of configuring a changed pixel obtained as a result of the pixel-based change detection into a changed object through morphological manipulation. To this end, the image fusion device 100 may sequentially perform a morphological closing process, a process of filling an empty space, and a morphological opening process. In this case, a rectangular structural element for performing the above-described morphological closing processing or morphological open processing may be set in two images, and according to an embodiment of the present application, the rectangular structural element is set to a size of 3x3 or 5x5. Can be.

5 is a diagram illustrating a change detection result based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application.

Referring to FIG. 5, (a) is an image representing an actually changed area in a photographing target area, (b) is a change detection result between two multispectral images, and (c) a composite aperture radar image and multiplex according to an embodiment of the present application The result of detecting the change between the fusion image of the spectral image and the multispectral image, (d) may represent the result of detecting the change between the two composite aperture radar images, respectively.

In addition, in FIG. 5A, the white area is a changed area or the area detected as changed, the black area is the area detected as a constant area or constant area, and the red square is a clear comparison of the change detection results of (a) to (d). In order to do so, it may be to indicate a region that is significantly different in (a) to (d).

5, the result of detecting the change between the fusion image of the composite aperture radar image and the multispectral image and the multispectral image according to an embodiment of the present application of (c) is that a multispectral image at any one point of view is not used for change detection. Although not, the result is similar to the result of detecting the change between the two multispectral images in (b), whereas the result of detecting the change between the two composite aperture radar images in (d) only shows the result of detecting changes in the form of multiple points. It can be seen that the accuracy is inferior to the change detection results of) to (c).

6 is a chart in which a change detection result based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application is evaluated through a statistical index.

Specifically, FIG. 6 is a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application based on the truth data identified through the image representing the actually changed area in the photographing target area of FIG. 5A. It is a chart that quantitatively compares the change detection result between the and multispectral images (first row), the change detection result between two multispectral images (second row), and the change detection result between two composite aperture radar images (third row).

Referring to FIG. 6, the change detection result may be evaluated using a recall, a precision, or an F1 score (F1). The recall, accuracy, and F1 score can be calculated by the following Equations 6-1 to 6-3.

[Equation 6-1]

[Equation 6-2]

The F1 score can be calculated as a harmonic average of recall and accuracy.

[Equation 6-3]

Referring to FIG. 6, the result of detection of a change between a fusion image of a composite aperture radar image and a multispectral image and a multispectral image according to an embodiment of the present application (first row) is an accuracy of 80.76%, a recall rate of 73.51%, and an F1 score of 76.96%. Can represent. The change detection result (2nd row) between the two multispectral images may indicate an accuracy rate of 90.805%, a recall rate of 69.425%, and an F1 score of 78.68%. The change detection result (third row) between the two composite aperture radar images can show an accuracy rate of 54.11%, a recall rate of 56.52%, and an F1 score of 59.29%.

That is, the change detection result based on the fusion image of the composite aperture radar image and the multispectral image according to an embodiment of the present application shows a dramatic improvement in performance (prediction accuracy) compared to the case using only the composite aperture radar image, and only the multispectral image It can be seen that the accuracy of change detection is similar to that of the case of using it.

7 is a schematic configuration diagram of an apparatus for fusion of a composite aperture radar image and a multispectral image according to an embodiment of the present application.

Referring to FIG. 7, the fusion device 100 (ie, the image fusion device 100) of a composite aperture radar image and a multispectral image according to an embodiment of the present disclosure includes a multispectral image processing unit 110, a composite aperture radar An image processing unit 120, a training pixel setting unit 130, a learning unit 140, a colorization unit 150, and a change detection unit 160 may be included.

The multispectral image processing unit 110 may extract color information from the multispectral image and classify the multispectral image into a plurality of classes.

The composite aperture radar image processing unit 120 may extract pixel information and characteristic information from the composite aperture radar image.

The training pixel determiner 130 may determine a training pixel from the multispectral image and the composite aperture radar image.

The learning unit 140 may learn a correlation between the multispectral image and the composite aperture radar image based on color information of the multispectral image corresponding to the training pixel and characteristic information of the composite aperture radar image.

In addition, the learning unit 140 inputs the characteristic information of a pixel corresponding to the position of the training pixel in the composite aperture radar image and the color information of the pixel corresponding to the position of the training pixel in the multispectral image. ) It is possible to learn the correlation between the multispectral image and the composite aperture radar image using the model.

The colorization unit 150 may color the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learned correlation.

In addition, the colorization unit 150 may predict color information in pixels other than the training pixels of the composite aperture radar image (eg, pixels excluding the training pixels from all pixels) based on the learned correlation.

Also, the colorization unit 150 may colorize the composite aperture radar image based on the predicted color information.

The change detector 160 may detect a change in an area to be photographed based on a colorized composite aperture radar image and a multispectral image.

8 is an operation flow diagram of a method for fusion of a composite aperture radar image and a multispectral image according to an embodiment of the present application.

The fusion method of the composite aperture radar image and the multispectral image according to the exemplary embodiment of the present application illustrated in FIG. 8 may be performed by the image fusion device 100 described above. Accordingly, even if the contents are omitted below, the contents described with respect to the image fusion apparatus 100 may be equally applied to FIG. 8.

Referring to FIG. 8, in step S810, the multispectral image processing unit 110 may process the multispectral image.

Specifically, in step S810, the multispectral image processing unit 110 may extract pixel values in the RGB band from the multispectral image.

In addition, in step S810, the multispectral image processing unit 110 may classify the multispectral image into a plurality of classes. In this case, the multispectral image processing unit 110 may classify the multispectral image into a plurality of classes by performing the K average clustering algorithm as many times as a preset repetition number.

Next, in step S820, the composite aperture radar image processing unit 120 may process the composite aperture radar image.

Next, in step S830, the training pixel setting unit 130 may determine a training pixel from the multispectral image and the composite aperture radar image.

Specifically, in step S830, the training pixel setting unit 130 may differentiate between pixel values of the composite aperture radar image and the multispectral image. Also, the training pixel setting unit 130 may determine an unchanged pixel as a training pixel based on a result of the difference between pixel values.

Next, in step S840, the learning unit 140 may learn the correlation between the multispectral image and the composite aperture radar image based on the color information of the multispectral image corresponding to the training pixel and the characteristic information of the composite aperture radar image. have.

According to an embodiment of the present application, in step S840, the learning unit 140 includes characteristic information of a pixel corresponding to a position of a training pixel in a composite aperture radar image, and color information of a pixel corresponding to a position of a training pixel of a multispectral image. The correlation between the multispectral image and the composite aperture radar image can be learned by using the random forest model having as an input.

Next, in step S850, the colorization unit 150 may color the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learned correlation.

Specifically, in step S850, the colorization unit 150 may predict color information in pixels other than the training pixels of the composite aperture radar image based on the learned correlation. Also, the colorization unit 150 may colorize the composite aperture radar image based on the predicted color information.

In the above description, steps S810 to S850 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present disclosure. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

9 is an operation flow diagram of a method of processing a composite aperture radar image according to an embodiment of the present application.

The method of processing the composite aperture radar image according to the exemplary embodiment of the present disclosure illustrated in FIG. 9 may be performed by the image fusion device 100 described above. Accordingly, even if the contents are omitted below, the contents described with respect to the image fusion apparatus 100 may be equally applied to FIG. 9.

Referring to FIG. 9, in step S910, the composite aperture radar image processing unit 120 may extract pixel information from the composite aperture radar image.

Next, in step S920, the composite aperture radar image processing unit 120 may generate a gray level co-occurrence matrix (GLCM) in each pixel of the composite aperture radar image.

Next, in step S930, the composite aperture radar image processing unit 120 may extract characteristic information of the composite aperture radar image from the GLCM.

In the above description, steps S910 to S930 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present disclosure. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

10 is an operation flow diagram of a method for detecting a change based on a fusion image of a composite aperture radar image and a multispectral image according to an embodiment of the present application.

The change detection method based on the fusion image of the composite aperture radar image and the multispectral image according to the exemplary embodiment of the present disclosure illustrated in FIG. 10 may be performed by the image fusion device 100 described above. Therefore, even if the contents are omitted below, the contents described with respect to the image fusion apparatus 100 may be equally applied to FIG. 10.

Referring to FIG. 10, in step S1010, the image fusion apparatus 100 may colorize the composite aperture radar image using the above-described fusion method of the composite aperture radar image and the multispectral image.

Next, in step S1020, the change detection unit 160 may perform detection of a change in the area to be photographed based on the colored composite aperture radar image and the multispectral image.

In the above description, steps S1010 to S1020 may be further divided into additional steps or may be combined into fewer steps, according to an embodiment of the present disclosure. In addition, some steps may be omitted as necessary, and the order between steps may be changed.

The fusion method of the composite aperture radar image and the multispectral image or the change detection method based on the fusion image of the composite aperture radar image and the multispectral image according to an embodiment of the present application are implemented in the form of program commands that can be executed through various computer means. And recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded in the medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler but also high-level language codes that can be executed by a computer using an interpreter or the like. The above-described hardware device may be configured to operate as one or more software modules to perform the operation of the present invention, and vice versa.

In addition, the above-described method for fusion of the composite aperture radar image and multispectral image or the change detection method based on the fusion image of the composite aperture radar image and multispectral image is in the form of a computer program or application executed by a computer stored in a recording medium. Can also be implemented.

The foregoing description of the present application is for illustrative purposes only, and those of ordinary skill in the art to which the present application pertains will be able to understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present application is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present application.

10: Change detection system based on fusion image of composite aperture radar image and multispectral image
100: fusion device for composite aperture radar image and multispectral image
110: multispectral image processing unit
120: composite aperture radar image processing unit
130: training pixel setting unit
140: Learning Department
150: colorization unit
160: change detection unit
200: composite aperture radar
300: multispectral imaging device
20: network

Claims (15)

In the fusion method of a composite aperture radar image and a multispectral image,
Processing a multispectral image;
Processing the composite aperture radar image;
Determining a training pixel from the multispectral image and the composite aperture radar image;
Learning a correlation between a multispectral image and a composite aperture radar image based on color information of a multispectral image corresponding to the training pixel and characteristic information of a composite aperture radar image; And
Colorizing the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learned correlation,
Including,
Determining the training pixel,
Dividing pixel values of the composite aperture radar image and the multispectral image; And
Determining an unchanged pixel as the training pixel based on the difference result of pixel values,
Including,
The step of colorizing the composite aperture radar image,
Predicting color information in pixels other than the training pixel of the composite aperture radar image based on the learned correlation, and colorizing the composite aperture radar image based on the predicted color information Fusion method of radar image and multispectral image. The method of claim 1,
Learning the correlation,
A multispectral image using a random forest model that inputs characteristic information of a pixel corresponding to the position of the training pixel in the composite aperture radar image and color information of a pixel corresponding to the position of the training pixel in the multispectral image. A method of fusion of a composite aperture radar image and a multispectral image to learn a correlation between a composite aperture radar image and a composite aperture radar image. The method of claim 2,
The step of processing the multispectral image,
Extracting pixel values of an RGB band from the multispectral image; And
Classifying the multispectral image into a plurality of classes,
The method of fusion of the composite aperture radar image and the multispectral image comprising a. The method of claim 3,
The random forest model is generated for each of the plurality of classes,
Learning the correlation,
A method of fusion of a composite aperture radar image and a multispectral image that is performed using each of the random forest models generated for each of a plurality of classes. The method of claim 3,
Classifying the multispectral image into a plurality of classes,
A method for fusion of a composite aperture radar image and a multispectral image, which classifies the multispectral image into a plurality of classes by performing a K average clustering algorithm as many times as a preset number of repetitions. The method of claim 2,
Processing the composite aperture radar image,
Extracting pixel information from the composite aperture radar image;
Generating a gray level co-occurrence matrix (GLCM) in each pixel of the composite aperture radar image; And
Extracting the characteristic information of the composite aperture radar image from the GLCM,
The method of fusion of the composite aperture radar image and the multispectral image comprising a. delete delete The method of claim 1,
The method of fusion of the composite aperture radar image and the multispectral image, wherein the multispectral image and the composite aperture radar image are acquired at different times. In the change detection method based on a fusion image of a composite aperture radar image and a multispectral image,
Colorizing the composite aperture radar image using the fusion method of the composite aperture radar image and the multispectral image according to claim 1; And
Performing change detection of an area to be photographed based on the colored composite aperture radar image and the multispectral image,
A change detection method based on the fusion image of the composite aperture radar image and the multispectral image comprising a. The method of claim 10,
The method of detecting a change based on a fusion image of the composite aperture radar image and the multispectral image, wherein the multispectral image and the composite aperture radar image are acquired at different times. In the fusion device of a composite aperture radar image and a multispectral image,
A multispectral image processing unit for extracting color information from a multispectral image and classifying the multispectral image into a plurality of classes;
A composite aperture radar image processing unit that extracts pixel information and characteristic information from the composite aperture radar image;
A training pixel setting unit determining a training pixel from the multispectral image and the composite aperture radar image;
A learning unit for learning a correlation between the multispectral image and the composite aperture radar image based on the color information of the multispectral image corresponding to the training pixel and the characteristic information of the composite aperture radar image; And
A colorization unit for colorizing the composite aperture radar image by fusing the multispectral image and the composite aperture radar image based on the learned correlation,
Including,
The training pixel setting unit,
Differentiate pixel values of the composite aperture radar image and the multispectral image, and determine an unchanged pixel as the training pixel based on the difference result of the pixel values,
The colorization unit,
Predicting color information in pixels other than the training pixel of the composite aperture radar image based on the learned correlation, and colorizing the composite aperture radar image based on the predicted color information A fusion device for radar image and multispectral image. The method of claim 12,
A change detection unit that detects a change in an area to be photographed based on the colored composite aperture radar image and the multispectral image,
The fusion device of the composite aperture radar image and the multispectral image further comprising a. The method of claim 12,
The learning unit,
A multispectral image using a random forest model that inputs characteristic information of a pixel corresponding to the position of the training pixel in the composite aperture radar image and color information of a pixel corresponding to the position of the training pixel in the multispectral image. A fusion device of a composite aperture radar image and a multispectral image to learn the correlation between the composite aperture radar image and the composite aperture radar image. delete

Images