KR102375353B1 - Apparatus and method for restoring aerosol region based on non-subsampled contourlet transfrom - Google Patents

Abstract

Disclosed are an apparatus and method for restoring an aerosol region based on a non-subsampled contourlet transform. The method for restoring an aerosol region based on a non-subsampled contourlet transform according to an embodiment of the present invention includes the steps of: preparing a target image and a reference image for a predetermined region; dividing the target image and the reference image into low-frequency components and high-frequency components by performing a non-sampling contourlet transform on the target image and the reference image; reconstructing a low-frequency component of the aerosol region in the target image based on the low-frequency components of the reference image through an artificial intelligence-based reconstruction model; reconstructing a high-frequency component of the aerosol region based on the high-frequency components of the reference image; and generating a reconstructed image by applying an inverse non-subsampled contourlet transform to the reconstructed low-frequency and high-frequency components.

Description

Apparatus and method for restoring an aerosol region based on non-subsampling contour transformation

The present application relates to an aerosol region restoration apparatus and method based on non-subsampling contour transformation.

Aerosol refers to effects generated from water vapor, mist, dust, smoke and other small particles in the atmosphere and is a factor that affects optical remote sensing satellite images. These aerosols reduce the quality of images and further adversely affect land cover classification and change detection, which are major fields of remote sensing. Therefore, removing the effect of aerosols present in the optical satellite image is a pre-processing process that must be performed in order to provide a high-quality optical satellite image.

On the other hand, conventionally proposed methods to remove the effect of aerosols in optical satellite images are largely divided into spatial-based methods, spectral-based methods, and temporal-based methods. can be In this regard, first, the spatial-based technique is relatively easy and efficient as it is performed through impinging, interpolation, etc., but it cannot be used to restore a large loss region or a region with a complex texture. Limitations exist. In addition, the spectrum-based technique has a limitation in that it is effective only in a specific band of a specific sensor by using other spectral data to restore it. In addition, the temporal-based technique can derive a restoration result by considering various situations in a restoration method using time series data, but there is a limitation in that it is data-dependent and requires a large amount of data.

The background technology of the present application is disclosed in Korean Patent Publication No. 10-1829287.

The present application is to solve the problems of the prior art described above, and restores an aerosol region based on a non-subsampled contour transformation that restores an aerosol region in a target image based on learning within a non-subsampled contour transform (NSCT) region. It is intended to provide an apparatus and method.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the non-subsampling contour transformation-based aerosol region restoration apparatus method according to an embodiment of the present application includes the steps of preparing a target image and a reference image for a predetermined region, the target image and performing non-subsampling contour transformation on the reference image to divide each of the target image and the reference image into a low-frequency component and a high-frequency component. Based on the low-frequency component of the reference image through an artificial intelligence-based reconstruction model, the Restoring the low-frequency component of the aerosol region in the target image, restoring the high-frequency component of the aerosol region based on the high-frequency component of the reference image, and restoring by applying inverse non-subsampling contour transformation to the restored low-frequency and high-frequency components It may include generating an image.

In addition, the step of restoring the low-frequency component includes deriving an unchanged pixel based on a predetermined reference point in the target image and the reference image, setting the aerosol region based on the quality index for the target image step, selecting a learning pixel including the unchanged pixel and the aerosol region, generating the artificial intelligence-based restoration model based on the learning pixel and a preset characteristic variable, and adding the learned restoration model The method may include inputting a low-frequency component of the target image and a characteristic variable for the target image to obtain a restored low-frequency component for the aerosol region.

In addition, the generating may include learning the reconstruction model based on a low-frequency component of the reference image corresponding to the learning pixel.

In addition, the characteristic variable may include a plurality of spectral indices.

Also, the restoring of the low-frequency component may include applying a median filter-based post-processing to the restored low-frequency component.

In addition, the deriving the unchanged pixel may include detecting the unchanged pixel based on near-infrared band information of the target image and the reference image.

Also, the restoration model may be an XGBoost-based machine learning model.

In addition, in the restoring of the high-frequency component, the high-frequency component of the target image is allocated to pixels included in the target image that are not affected by the aerosol, and the high-frequency component of the reference image is assigned to the pixels in the aerosol region. It can be performed in a way of allocating.

On the other hand, the apparatus for restoring an aerosol region based on non-subsampling contour transformation according to an embodiment of the present application includes an image acquisition unit that prepares a target image and a reference image for a predetermined region, and a non-subsampling contour with respect to the target image and the reference image. A transform unit that performs transformation to divide each of the target image and the reference image into a low-frequency component and a high-frequency component, based on the low-frequency component of the reference image through an artificial intelligence-based reconstruction model, the low-frequency component of the aerosol region in the target image A low-frequency restoration unit that restores a, a high-frequency restoration unit that restores a high-frequency component of the aerosol region based on a high-frequency component of the reference image, and a reconstructed image by applying non-subsampling contour inverse transformation to the restored low-frequency component and high-frequency component It may include an inverse transform unit.

In addition, the low-frequency restoration unit derives an unchanged pixel based on a predetermined reference point in the target image and the reference image, sets the aerosol region based on the quality index for the target image, and the unchanged pixel and a learning pixel setting unit that selects a learning pixel including the aerosol region, a model generation that generates the artificial intelligence-based restoration model based on the learning pixel and preset characteristic variables, and the target image in the learned restoration model and a model application unit configured to obtain a restored low-frequency component for the aerosol region by inputting a low-frequency component of and a characteristic variable for the target image.

Also, the model generator may learn the reconstruction model based on a low-frequency component of the reference image corresponding to the learning pixel.

In addition, the high-frequency restoration unit may allocate a high-frequency component of the target image to pixels included in the target image that are not affected by aerosol, and a high-frequency component of the reference image to pixels in the aerosol region. .

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, a non-subsampled contour transformation-based aerosol region restoration apparatus and method for restoring an aerosol region in a target image based on learning in a non-subsampled contour transform (NSCT) region can provide

According to the above-described problem solving means of the present application, restoration in consideration of background information and object information can be performed by performing a restoration process independently for each component after separating a low-frequency component and a high-frequency component.

According to the above-described problem solving means of the present application, by using the spectral index as a parameter, information such as atmospheric effects reflected in the target image and the reference image and the vegetation status of the indicator are fused, and non-linear relationship establishment based on regression models such as XGBoost Restoration may be performed.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 is a schematic configuration diagram of an image providing system including an aerosol region restoration device based on non-subsampling contour transformation according to an embodiment of the present application.
2 is a conceptual diagram for explaining the entire process of the aerosol region restoration technique based on the non-subsampling contour transformation of the present application.
3 is a diagram for explaining a quality index of a target image.
4A is a diagram illustrating a target image and a reference image in the first experimental example in connection with an aerosol region restoration technique based on non-subsampling contour transformation according to an embodiment of the present application.
4B is a diagram illustrating a target image and a reference image in the second experimental example linked to the aerosol region restoration technique based on non-subsampling contour transformation according to an embodiment of the present application.
5 is a view showing restored images in Experimental Example 1 and Experiment 2 in connection with an aerosol region restoration technique based on non-subsampling contour transformation according to an embodiment of the present application.
6 is a schematic configuration diagram of an aerosol region restoration apparatus based on non-subsampling contour transformation according to an embodiment of the present application.
7 is a detailed configuration diagram of a low-frequency restoration unit of an aerosol region restoration apparatus based on non-subsampling contour transformation according to an embodiment of the present application.
8 is a flowchart of an aerosol region restoration method based on non-subsampling contour transformation according to an embodiment of the present application.
9 is a detailed operation flowchart of a low-frequency component restoration process according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily carry out. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is "connected" with another part, it is not only "directly connected" but also "electrically connected" or "indirectly connected" with another element interposed therebetween. "Including cases where

Throughout this specification, when a member is positioned “on”, “on”, “on”, “on”, “under”, “under”, or “under” another member, this means that a member is positioned on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

The present application relates to an aerosol region restoration apparatus and method based on non-subsampling contour transformation.

1 is a schematic configuration diagram of an image providing system including an aerosol region restoration device based on non-subsampling contour transformation according to an embodiment of the present application.

Referring to FIG. 1 , the image providing system 10 according to an embodiment of the present application is an aerosol region restoration apparatus 100 based on non-subsampling contour transformation according to an embodiment of the present application (hereinafter, 'reconstruction apparatus 100') ), a photographing device 200 , and an image storage server 300 .

The restoration apparatus 100 , the photographing apparatus 200 , and the image storage server 300 may communicate with each other through the network 20 . The network 20 refers to a connection structure in which information exchange is possible between each node, such as terminals and servers, and an example of such a network 20 includes a 3rd Generation Partnership Project (3GPP) network, a long-term LTE (LTE) network. Term Evolution) network, 5G network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area) Network), a wifi network, a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but are not limited thereto.

In addition, although not shown in the drawings, the restored image generated by the restoration apparatus 100 is a user terminal (not shown) connected to at least one of the restoration apparatus 100 and the image storage server 300 through the network 20 . can be displayed (played) through The user terminal is, for example, a smartphone, a smart pad, a tablet PC, and the like and a PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System) ), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may be a wireless communication device of

In the description of the embodiments of the present application, at least one of a target image and a reference image may be an optical satellite image. For example, the target image and the reference image may include a synthetic aperture radar image, a multispectral image, and the like. Also, in this regard, the imaging device 200 in the present application may be a multi-spectral imaging device. As another example, the photographing apparatus 200 may be a synthetic aperture radar (SAR), which is a radar that captures images of observation of the ground and the sea from the air.

More specifically, the imaging apparatus 200 may be a device mounted on a Landsat-8 satellite including an Operational Land Imager (OLI) and a Thermal Infrared Sensor (TIRS), but is not limited thereto. As another example, the imaging device 200 may be a radar provided in KOMPSAT-5, which is a multi-purpose satellite. In addition, various types of optical satellite images such as Landsat 4-7 and Sentinel 2 may be applied to the photographing apparatus 200 and the target image and the reference image obtained by the photographing apparatus 200 according to the embodiment of the present application.

Hereinafter, for convenience of description, it is assumed that the target image and the reference image are the Landsat-8 OLI/TIRS-based optical satellite images to describe the embodiment of the present application.

In addition, in the description of the embodiment of the present application, the image storage server 300 stores the restored image in which the restoration of the aerosol region is performed as described below based on the target image, the reference image, and the target image and the reference image. It can be a storage or a server. In addition, the image storage server 300 is a target image to the user terminal 400 so that at least one of a target image, a reference image, and a restored image can be displayed (played) through the user terminal 400 according to the embodiment of the present application; It may be provided to transmit (provide) at least one of a reference image and a restored image.

2 is a conceptual diagram for explaining the entire process of the aerosol region restoration technique based on the non-subsampling contour transformation of the present application.

Referring to FIG. 2 , the restoration apparatus 100 may prepare a target image and a reference image for a predetermined area. For example, the restoration apparatus 100 may store a target image and a reference image photographed by the photographing apparatus 200 or previously photographed and stored in the image storage server 300 , at least of the photographing apparatus 200 and the image storage server 300 . can be received from one. For example, the restoration device 100 receives a target image to which the restoration process for the aerosol region disclosed herein is applied from the photographing device 200 , and from the image storage server 300 , with respect to the target space reflected in the target image A target image and a reference image may be prepared by receiving a previously secured reference image.

In other words, in the description of the embodiments of the present application, the target image is an optical satellite image, and the effect of an aerosol is reflected in at least a partial region in the image, so that restoration of the region is required, and the reference image is the image to be restored. As an optical satellite image obtained by photographing a space (target space) corresponding to the target image, it may refer to an image in which the effect of an aerosol is not reflected. In addition, here, that the influence of the aerosol is not reflected in the reference image means that the region in the reference image corresponding to the region to be restored in the target image is not affected by the aerosol, and always the entire region of the reference image. It is not necessary to be free from the effects of this aerosol.

In addition, according to the embodiment of the present application, the target image and the reference image may be the same type of optical satellite image, but the present invention is not limited thereto. According to the embodiment of the present application, the target image and the reference image may be different types of optical satellite images. can For example, the reference image may be an image captured in the same space within a preset time range from a time point at which the target image is captured.

In addition, the restoration apparatus 100 may perform non-subsampling contour transformation on the prepared target image and reference image to divide each of the target image and the reference image into a low-frequency component and a high-frequency component ('NSCT application' in FIG. 2 ). . For reference, since the low-frequency component of a predetermined image may correspond to an area having a small difference (eg, color difference) from the surrounding area, and the high-frequency component may correspond to an area having a large difference from the surrounding area, a specific object (object) may be selected. As a reference, the low-frequency component may indicate an inner region of the object, and the high-frequency component may indicate a boundary region with respect to the outside of the object.

Specifically, when using the non-subsampled contour transform (NSCT), the band can be divided according to the frequency size through the non-subsampled pyramid and directional filter, and since it has a shift-invariant characteristic, frequency aliasing In contrast, in the case of wavelet transform, it does not separate two-dimensional structures such as curves or contours well, There is a disadvantage in that the entire area cannot be considered by reducing the image size in the horizontal or vertical direction.

Also, according to an exemplary embodiment of the present disclosure, the reconstruction apparatus 100 may perform non-subsampling contour transformation based on two decomposition layers corresponding to each of a low frequency component and a high frequency component. For example, the restoration apparatus 100 may apply the [4] decomposition layer to the low-frequency component and apply the [4, 8] decomposition layer to the high-frequency component.

Also, the restoration apparatus 100 may reconstruct the aerosol region for the low-frequency component of the target image based on the low-frequency component of the target image and the low-frequency component of the reference image. Also, the restoration apparatus 100 may reconstruct the aerosol region of the high-frequency component of the target image based on the high-frequency component of the target image and the high-frequency component of the reference image. Hereinafter, the aerosol region restoration process for each of the high frequency component and the low frequency component divided by the non-subsampling contour transformation will be separately described.

First, as a restoration process of the low-frequency component of the target image, the restoration apparatus 100 may restore the low-frequency component of the aerosol region in the target image based on the low-frequency component of the reference image through an artificial intelligence-based restoration model.

Specifically, according to an embodiment of the present disclosure, the restoration apparatus 100 may derive an unchanged pixel based on a predetermined reference point in the target image and the reference image. According to an embodiment of the present disclosure, the restoration apparatus 100 may detect an unchanged pixel based on near-infrared band information of the target image and the reference image.

According to an embodiment of the present disclosure, the restoration apparatus 100 may detect an unchanged pixel based on a scattergram between the near-infrared bands of the target image and the reference image. Illustratively, the restoration apparatus 100 selects as a reference point a local maximum value corresponding to each center of two different types of clusters (eg, a cluster corresponding to water and a cluster corresponding to an index) as a reference point, and does not change A no-change line may be obtained, and a half vertical width and HVW value may be obtained based on the obtained shape of the unchanged line to select a pixel existing therein as the unchanged pixel.

In addition, according to an embodiment of the present application, the restoration apparatus 100 may set the aerosol region based on the quality index of the target image.

3 is a diagram for explaining a quality index of a target image.

Referring to FIG. 3 , when a target image is a Landsat-8 OLI/TIRS-based optical satellite image, a quality factor may mean a Quality Assessment Band Attribute provided by the Landsat-8 image. In addition, referring to FIG. 3 , the quality index classifies the characteristics of pixels into attributes based on the characteristics of each pixel or a local area including each pixel in the image (eg, Terrain Occlusion, Clear, Cloud, Snow/Ice, etc.), and a value (Pixel Value of FIG. 3 ) corresponding to the classified type may be assigned.

In addition, according to an embodiment of the present application, the restoration apparatus 100 determines a pixel corresponding to a preset classification type as a pixel not affected by the aerosol based on the quality index for each pixel included in the target image, and the corresponding A pixel in an area other than the pixel may be determined as a pixel in the aerosol area. For example, referring to FIG. 3 , the restoration apparatus 100 converts pixels to which pixel values of 2720, 2724, 2728, and 2732 corresponding to the 'Clear' attribute are assigned as pixels not affected by the aerosol. may be determined, and pixels in an area other than the determined pixel may be determined as pixels included in the aerosol area, but the classification type considered for specifying the aerosol area is not limited to the above-described example.

In addition, according to an embodiment of the present application, the restoration apparatus 100 considers the boundary of the aerosol area and has a predetermined size (eg, 11 X 11 size, etc.) may be set to perform post-processing to more accurately specify the aerosol area.

Also, according to an embodiment of the present application, the restoration apparatus 100 may select a training pixel including a set unchanged pixel and an aerosol region. In the learning process of the artificial intelligence-based reconstruction model to be described later, the training pixel does not learn all the areas (pixels) of the target image and the reference image, but rather of the target image and the reference image requiring learning to restore the aerosol region. As a specific area for learning non-linear correlation, the artificial intelligence-based reconstruction model disclosed herein may perform learning on areas (pixels) including previously set unchanged pixels and aerosol areas.

Also, according to an embodiment of the present application, the restoration apparatus 100 may generate (construct) an artificial intelligence-based restoration model based on the learning pixel and preset characteristic variables. Specifically, the restoration apparatus 100 may learn the restoration model based on the low-frequency component of the reference image corresponding to the learning pixel.

In addition, the aforementioned characteristic variable may include a plurality of spectral indices. Spectral indices or optical indices are optical indices that reflect the various spectral characteristics of satellite images. It is an index for analyzing various characteristics and situations such as vegetation, water resources, snowfall, soil, and fire reflected in the satellite image. can

For example, the restoration apparatus 100 derives a spectral index based on band characteristics such as a visible ray region, an infrared region, and an ultraviolet region of the target image, and a learning parameter applied to the artificial intelligence-based restoration model can be set to

More specifically, the restoration apparatus 100 is a normal snow index (Normalized Difference Snow Index, NDSI), a regular vegetation index (Normalized Difference Vegetation Index, NDVI), meanVis, whiteness (Whiteness), HOT and at least one of B5 / B6 The included spectral index may be set as a learning parameter of the restoration model, and the specific calculation of the spectral index is as shown in Equations 1-1 to 1-6 below.

[Equation 1-1]

[Equation 1-2]

[Equation 1-3]

[Equation 1-4]

[Equation 1-5]

[Equation 1-6]

On the other hand, the above formula for calculating the spectral index is applied on the basis that the target image is a Landsat-8 OLI/TIRS-based satellite image type, and when the target image is a type other than Landsat-8, based satellite image, It goes without saying that the specific formula for calculating the spectral index may vary.

In addition, the artificial intelligence-based restoration model according to an embodiment of the present application may be an eXtreme Gradient Boosting (XGBoost)-based machine learning model. Here, the XGBoost-based machine learning model may be an ensemble-type machine learning model based on a decision tree. However, the present invention is not limited thereto, and various artificial intelligence-based learning algorithms (models) that have been previously known or developed in the future may be applied.

On the other hand, the XGBoost-based restoration model is built in a learning method that creates a complex classifier by stacking a plurality of classifiers. ), the branch of the decision tree can be removed, which has the advantage that overfitting occurs relatively less.

For reference, the hyper parameters for training the restoration model according to an embodiment of the present application are gamma = 0, lambda = 1, alpha = 0, n_estimators = 100 (early stopping = 10), max depth = 6, minimum child weight = 5, subsample = 0.8, colsample by tree = 1.0, may be exemplarily set to learning rate = 0.1.

In addition, according to an embodiment of the present application, the restoration apparatus 100 obtains the restored low-frequency component for the aerosol region by inputting the low-frequency component of the target image and the characteristic variable for the target image to the learned artificial intelligence-based restoration model. can

On the other hand, when a target image requiring restoration of the aerosol region is input as described above, the restoration apparatus 100 is artificial intelligence-based restoration for restoration of a low-frequency component based on a reference image for restoration of the aerosol region of the corresponding target image. It may operate to generate a model, but is not limited thereto, and according to an embodiment of the present disclosure, the restoration apparatus 100 generates a new target image based on training data including a plurality of target images and a reference image corresponding thereto. When an input is received, an artificial intelligence-based restoration model that is learned to perform the restoration process of the low-frequency component is built before a new target image is input and is retained, and when a new target image is input (approved), a pre-generated restoration model It may operate to perform restoration of the aerosol region with respect to the low frequency component of the new target image based on .

Also, according to an embodiment of the present application, the restoration apparatus 100 may apply a median filter-based post-processing to the restoration low-frequency component restored by the artificial intelligence-based restoration model. For example, the median filter may be applied in a size of 3×3, but is not limited thereto.

Also, as a restoration process of the high-frequency component of the target image, the restoration apparatus 100 may reconstruct the high-frequency component of the aerosol region based on the high-frequency component of the reference image. Specifically, the restoration apparatus 100 allocates a high-frequency component of the target image to pixels included in the target image that are not affected by the aerosol, and pixels included in the target image within the aerosol-affected aerosol region. A high-frequency component restoration process may be performed by selectively allocating pixel components in a manner of allocating high-frequency components of the reference image.

Also, the restoration apparatus 100 may generate a restored image by applying a non-subsampling contour inverse transform ('Inverse NSCT' in FIG. 2 ) to the reconstructed low frequency component and the reconstructed high frequency component. Also, the restoration apparatus 100 may transmit the generated restored image to at least one of the image storage server 300 and the user terminal 400 .

Figure 4a is a view showing a target image and a reference image in the first experimental example linked to the aerosol region restoration technique based on non-subsampling contour transformation according to an embodiment of the present application, It is a view showing the target image and the reference image in the second experimental example linked with the sampling contour transformation-based aerosol region restoration technique, and FIG. 5 is linked with the non-subsampling contour transformation-based aerosol region restoration technique according to an embodiment of the present application It is a diagram showing the restored images in the first and second experimental examples.

4A, 4B and 5 , according to the restoration apparatus 100 disclosed herein, the user sets the desired region and performs restoration of the target image to perform restoration of the target image in the aerosol region (‘A’ in FIGS. 4A and 4B). ' area) can be reconstructed to obtain a reconstructed image. Therefore, in the case of the restoration device 100 of the present application, even when a part of the optical image-based time series data is affected by aerosol due to an unforeseen cause, it can be utilized for building time series data by restoring the image to exclude the influence of aerosol. there is.

6 is a schematic configuration diagram of an aerosol region restoration apparatus based on non-subsampling contour transformation according to an embodiment of the present application.

Referring to FIG. 6 , the restoration apparatus 100 may include an image acquisition unit 110 , a transformation unit 120 , a low-frequency restoration unit 130 , a high-frequency restoration unit 140 , and an inverse transformation unit 150 . .

The image acquisition unit 110 may prepare a target image and a reference image for a predetermined region.

The transform unit 120 may perform non-subsampling contour transformation on the target image and the reference image to divide each of the target image and the reference image into a low-frequency component and a high-frequency component.

The low-frequency restoration unit 130 may reconstruct the low-frequency component of the aerosol region in the target image based on the low-frequency component of the reference image through an artificial intelligence-based restoration model.

7 is a detailed configuration diagram of a low-frequency restoration unit of an aerosol region restoration apparatus based on non-subsampling contour transformation according to an embodiment of the present application.

Referring to FIG. 7 , the low-frequency restoration unit 130 may include a learning pixel setting unit 131 , a model generation unit 132 , a model application unit 133 , and a post-processing unit 134 .

The learning pixel setting unit 131 may derive unchanged pixels based on predetermined reference points in the target image and the reference image. Specifically, the learning pixel setting unit 131 may detect an unchanged pixel based on near-infrared band information of the target image and the reference image. Also, the learning pixel setting unit 131 may set the aerosol region based on the quality index of the target image. Also, the learning pixel setting unit 131 may select a learning pixel including an unchanged pixel and an aerosol region.

The model generator 132 may generate an artificial intelligence-based reconstruction model based on the learning pixel and preset characteristic variables. Specifically, the model generator 132 may learn the reconstruction model based on the low-frequency component of the reference image corresponding to the learning pixel.

The model application unit 133 may obtain a restored low-frequency component for the aerosol region by inputting a low-frequency component of the target image and a characteristic variable for the target image to the learned reconstruction model.

The post-processing unit 134 may apply a median filter-based post-processing to the restored low-frequency component.

The high frequency restoration unit 140 may reconstruct the high frequency component of the aerosol region of the target image based on the high frequency component of the reference image. Specifically, the high frequency restoration unit 140 allocates the high frequency component of the target image to pixels included in the target image that are not affected by the aerosol, and allocates the high frequency component of the reference image to pixels in the aerosol region. Restoration of components can be carried out.

The inverse transform unit 150 may generate a reconstructed image by applying a non-subsampling contour inverse transform to the reconstructed low and high frequency components.

Hereinafter, an operation flow of the present application will be briefly reviewed based on the details described above.

8 is a flowchart of an aerosol region restoration method based on non-subsampling contour transformation according to an embodiment of the present application.

The aerosol region restoration method based on the non-subsampling contour transformation shown in FIG. 8 may be performed by the restoration apparatus 100 described above. Therefore, even if omitted below, the description of the restoration apparatus 100 may be equally applied to the description of the aerosol region restoration method based on the non-subsampling contour transformation.

Referring to FIG. 8 , in step S11 , the image acquisition unit 110 may prepare a target image and a reference image for a predetermined region.

Next, in step S12 , the transform unit 120 may perform non-subsampling contour transformation on the target image and the reference image to divide each of the target image and the reference image into a low-frequency component and a high-frequency component.

Next, in step S13, the low-frequency restoration unit 130 may reconstruct the low-frequency component of the aerosol region in the target image based on the low-frequency component of the reference image through the artificial intelligence-based restoration model.

Next, in step S14 , the high frequency restoration unit 140 may reconstruct the high frequency component of the aerosol region of the target image based on the high frequency component of the reference image. Specifically, in step S14, the high-frequency restoration unit 140 allocates the high-frequency component of the target image to pixels included in the target image that are not affected by the aerosol, and allocates the high-frequency component of the reference image to the pixels in the aerosol region. In this way, the high-frequency component can be restored.

Next, in step S15 , the inverse transform unit 150 may generate a reconstructed image by applying a non-subsampling contour inverse transform to the reconstructed low and high frequency components.

In the above description, steps S11 to S15 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

9 is a detailed operation flowchart of a low-frequency component restoration process according to an embodiment of the present application.

The low-frequency component restoration process of the target image shown in FIG. 9 may be performed by the low-frequency restoration unit 130 described above. Accordingly, even if omitted below, the description of the low-frequency restoration unit 130 may be equally applied to the description of FIG. 9 .

Referring to FIG. 9 , in step S131 , the learning pixel setting unit 131 may derive unchanged pixels based on predetermined reference points in a target image and a reference image. Specifically, in operation S131 , the learning pixel setting unit 131 may detect an unchanged pixel based on near-infrared band information of the target image and the reference image.

Next, in step S132 , the learning pixel setting unit 131 may set the aerosol region based on the quality index of the target image.

Next, in step S133 , the learning pixel setting unit 131 may select an unchanged pixel and a learning pixel including the aerosol region.

Next, in step S134 , the model generator 132 may generate an artificial intelligence-based reconstruction model based on the learning pixel and preset characteristic variables. Specifically, in operation S134 , the model generator 132 may learn the reconstruction model based on the low-frequency component of the reference image corresponding to the training pixel.

Next, in step S135, the model application unit 133 may obtain a restored low-frequency component for the aerosol region by inputting the low-frequency component of the target image and the characteristic variable of the target image to the learned reconstruction model.

Next, in step S136 , the post-processing unit 134 may apply a median filter-based post-processing to the restored low-frequency component.

In the above description, steps S131 to S136 may be further divided into additional steps or combined into fewer steps according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between steps may be changed.

The non-subsampling contour transformation-based aerosol region restoration method according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to carry out the operations of the present invention, and vice versa.

In addition, the above-described non-subsampling contour transformation-based aerosol region restoration method may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a dispersed form, and likewise components described as distributed may be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalents should be construed as being included in the scope of the present application.

10: video providing system
100: aerosol region restoration device based on non-subsampling contour transformation
110: image acquisition unit
120: conversion unit
130: low frequency restoration unit
131: learning pixel setting unit
132: model generation unit
133: model application part
134: post-processing unit
140: high-frequency restoration unit
150: inverse transform unit
200: shooting device
300: video storage server
20: network

Claims (12)

In an aerosol region restoration method based on non-subsampling contour transformation,
preparing a target image and a reference image for a predetermined region;
performing non-subsampling contour transformation on the target image and the reference image to divide each of the target image and the reference image into a low-frequency component and a high-frequency component;
reconstructing a low-frequency component of the aerosol region in the target image based on the low-frequency component of the reference image through an artificial intelligence-based reconstruction model;
reconstructing the high-frequency component of the aerosol region based on the high-frequency component of the reference image; and
generating a reconstructed image by applying a non-subsampling contour inverse transform to the reconstructed low-frequency component and the high-frequency component;
including,
The step of restoring the low-frequency component,
deriving an unchanged pixel based on a predetermined reference point in the target image and the reference image;
setting the aerosol area based on the quality index for the target image;
selecting a learning pixel including the unchanged pixel and the aerosol region;
generating the artificial intelligence-based reconstruction model based on the learning pixels and preset characteristic variables; and
It characterized in that it comprises the step of inputting the low-frequency component of the target image and the characteristic variable of the target image to the learned reconstruction model to obtain the restored low-frequency component for the aerosol region,
How to restore. delete According to claim 1,
The generating step is
The reconstruction method of learning the reconstruction model based on the low-frequency component of the reference image corresponding to the training pixel. 4. The method of claim 3,
The characteristic variable, the restoration method comprising a plurality of spectral indices. According to claim 1,
The step of restoring the low-frequency component,
applying a median filter-based post-processing to the restored low-frequency component;
Which further comprises a, restoration method. According to claim 1,
The step of deriving the unchanged pixel comprises:
The method of claim 1, wherein the unchanged pixel is detected based on near-infrared band information of the target image and the reference image. 5. The method of claim 4,
The restoration model, characterized in that the XGBoost-based machine learning model, restoration method. According to claim 1,
The step of restoring the high-frequency component,
Among the pixels included in the target image, pixels that are not affected by the aerosol are assigned a high-frequency component of the target image, and pixels in the aerosol region are allotted a high-frequency component of the reference image. . In an aerosol region restoration device based on non-subsampling contour transformation,
an image acquisition unit for preparing a target image and a reference image for a predetermined region;
a transform unit configured to divide the target image and the reference image into low-frequency components and high-frequency components by performing non-subsampling contour transformation on the target image and the reference image;
a low-frequency restoration unit for reconstructing a low-frequency component of the aerosol region in the target image based on the low-frequency component of the reference image through an artificial intelligence-based restoration model;
a high-frequency restoration unit for reconstructing a high-frequency component of the aerosol region based on the high-frequency component of the reference image; and
An inverse transform unit for generating a reconstructed image by applying a non-subsampling contour inverse transform to the reconstructed low-frequency component and the high-frequency component;
including,
The low-frequency restoration unit,
Deriving an unchanged pixel based on a predetermined reference point in the target image and the reference image, setting the aerosol region based on a quality index for the target image, and including the unchanged pixel and the aerosol region a learning pixel setting unit for selecting a learning pixel;
a model generator for generating the artificial intelligence-based reconstruction model based on the learning pixel and preset characteristic variables; and
A model application unit for obtaining a restored low-frequency component for the aerosol region by inputting a low-frequency component of the target image and a characteristic variable for the target image to the learned reconstruction model;
That comprising a, restoration device. delete 10. The method of claim 9,
The characteristic variable includes a plurality of spectral indices,
The model generation unit,
The restoration apparatus, which trains the restoration model based on the low-frequency component of the reference image corresponding to the learning pixel. 10. The method of claim 9,
The high-frequency restoration unit,
Among the pixels included in the target image, pixels that are not affected by the aerosol are assigned a high-frequency component of the target image, and pixels in the aerosol region are assigned a high-frequency component of the reference image.

Images