CN107194872A - Remote sensed image super-resolution reconstruction method based on perception of content deep learning network - Google Patents

Abstract

The invention discloses a remote sensing image super-resolution reconstruction method based on a content-aware deep learning network. The invention proposes a comprehensive measurement index and a calculation method for the complexity of the image content. Based on this, the sample images are classified according to the content complexity. , build and train three deep GAN network models of high, medium and low complexity, and then select the corresponding network for reconstruction according to the content complexity of the input image to be super-scored. In order to improve the learning performance of the GAN network, the present invention provides an optimized loss function definition at the same time. The method of the invention overcomes the ubiquitous contradiction of over-fitting and under-fitting in super-resolution reconstruction based on machine learning, and effectively improves the super-resolution reconstruction accuracy of remote sensing images.

Description

Super-resolution reconstruction method of remote sensing images based on content-aware deep learning network

technical field

The invention belongs to the technical field of image processing, and relates to an image super-resolution reconstruction method, in particular to a remote sensing image super-resolution reconstruction method based on a content-aware deep learning network.

technical background

Remote sensing images with high spatial resolution can describe the ground objects more finely and provide rich detail information. Therefore, people often hope to obtain images with high spatial resolution. With the rapid development of space detection theory and technology, remote sensing images with meter-level or even sub-meter-level spatial resolution (such as IKNOS and QuickBird) have been gradually applied, but their temporal resolution is generally relatively low. On the contrary, some sensors with lower spatial resolution (such as MODIS) have high temporal resolution, and they can acquire a large-scale remote sensing image in a short time. If high-spatial-resolution images can be reconstructed from these low-spatial-resolution images, remote sensing images with both high spatial and high temporal resolutions can be obtained. Therefore, it is very necessary to reconstruct lower resolution remote sensing images to obtain higher resolution images.

In recent years, deep learning has been widely used to solve various problems in computer vision and image processing. In 2014, C.Dong et al. from the Chinese University of Hong Kong took the lead in introducing deep CNN learning into the super-resolution reconstruction of images, which achieved better results than the previous mainstream sparse expression methods; in 2015, J. .Kim et al. further proposed an improved method based on RNN, and the performance was further improved; in 2016, Y.Romano et al. of Google developed a fast and accurate learning method; shortly thereafter, C.Ledig of Twitter et al. used GAN network (generative confrontation network) for image super-resolution and achieved the best reconstruction effect so far. Moreover, the bottom layer of GAN is a deep belief network, which no longer strictly relies on supervised learning, and can be trained even without one-to-one high- and low-resolution image sample pairs.

After the deep learning model and network architecture are determined, the performance of the deep learning-based super-resolution method is largely determined by the quality of the network model training. The training of the deep learning network model is not the more thorough and effective, but sufficient and appropriate sample learning (just as the number of layers of the deep network model is not the more the better). For complex images, more sample training is needed so that more image features can be learned, but such a network is prone to overfitting to images with simple content, resulting in blurred super-resolution results; on the contrary, reduce the training intensity, It can avoid the over-fitting phenomenon of images with simple content, but it will cause the under-fitting problem of images with complex content, which reduces the naturalness and fidelity of reconstructed images. How to ensure that the trained network can take into account the high-quality reconstruction of complex and simple images at the same time is an unavoidable problem for deep learning-based methods in practical super-resolution applications.

Contents of the invention

In order to solve the above technical problems, the present invention proposes a remote sensing image super-resolution reconstruction method based on content-aware deep learning network.

The technical solution adopted in the present invention is: a method for super-resolution reconstruction of remote sensing images based on a content-aware deep learning network, characterized in that it includes the following steps:

Step 1: Collect high and low resolution remote sensing image samples and perform block processing;

Step 2: Calculate the complexity of each image block, divide it into three categories according to the complexity: high, medium, and low, and form high, medium, and low complexity training sample sets respectively;

Step 3: Use the obtained sample sets to train three GAN networks with high, medium and low complexity respectively;

Step 4: Calculate the complexity of the input image, and select the corresponding GAN network reconstruction according to the complexity.

Compared with existing image super-resolution methods, the present invention has the following advantages and positive effects:

(1) By using the simple idea of image classification, the present invention successfully overcomes the ubiquitous over-fitting and under-fitting contradictions in super-resolution reconstruction based on machine learning, and effectively improves the super-resolution reconstruction accuracy of remote sensing images ;

(2) The deep learning network model based on the method of the present invention is a GAN network, which does not rely on strict one-to-one alignment of high and low resolution sample blocks during training, thus improving the application universality, especially suitable for high and low resolution in the field of remote sensing A multi-source asynchronous imaging environment of images.

Description of drawings

Fig. 1 is a flowchart of an embodiment of the present invention.

detailed description

In order to facilitate those of ordinary skill in the art to understand and implement the present invention, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described here are only used to illustrate and explain the present invention, and are not intended to limit this invention.

Please see Fig. 1, a kind of remote sensing image super-resolution reconstruction method based on content-aware deep learning network provided by the present invention, comprises the following steps:

Step 1: Collect high and low resolution remote sensing image samples, evenly divide the high resolution image into 128x128 image blocks, and evenly divide the low resolution image into 64x64 image blocks;

Step 2: Calculate the complexity of each image block, divide it into three categories according to the complexity: high, medium, and low, and form high, medium, and low complexity training sample sets respectively;

The calculation principle and method of image complexity are as follows:

The complexity of image content includes texture complexity and structural complexity. Information entropy and gray level consistency can better describe texture complexity, while structural complexity is suitable to be described by the edge ratio of objects in the image. The image content complexity metric C is composed of information entropy H, gray level consistency U and edge ratio R, weighted by the following formula:

C=w _h ×H+w _u ×U+w _e ×E;

Here w _h , w _u , we _e are their respective weights, and the weights are determined by experiments.

The respective calculation methods of information entropy, texture consistency and edge ratio are given below.

(1) Information entropy

Information entropy reflects the number of image gray levels and the appearance of each gray level pixel. The higher the entropy value, the more complex the image texture. The calculation formula of image information entropy H is:

N is the number of gray levels, _ni is the number of occurrences of each gray level, and K is the number of gray levels.

(2) Grayscale consistency

Gray consistency can reflect the uniformity of the image, if its value is small, it corresponds to a simple image, otherwise it corresponds to a complex image. The gray scale consistency formula is:

Among them, M and N are the number of rows and columns of the image respectively, f(i, j) is the gray value at the pixel (i, j), is the gray mean value of the 3×3 neighborhood pixels centered on (i,j).

(3) Edge ratio

The number of targets in the map directly reflects the complexity of the image. If the number of targets is large, the image is generally more complex, and vice versa. Since the counting of targets involves complex graph segmentation, it is not easy to calculate. The number of target edges indirectly reflects the number and complexity of targets in the image, so it can be used to describe the complexity of the image. The proportion of the target edge in the image can be described by the edge ratio, and the calculation formula is:

Among them, M and N are the number of rows and columns of the image, respectively, and E is the number of edge pixels in the image. The edge of the target in the image is a place where the gray level changes significantly, which can be obtained by the difference algorithm. Generally, the edge pixels of the image are detected by edge detection operators (such as Canny operator, Sobel operator, etc.).

Among them, the number of high-resolution sample set image blocks is not less than 500,000, the number of medium-resolution image blocks is not less than 300,000, and the number of low-resolution image blocks is not less than 200,000.

Step 3: Use the obtained sample sets to train three GAN networks with high, medium and low complexity respectively;

The loss function of GAN network training is defined as follows:

The loss functions of GAN network training include content loss, generation-adversarial loss and total variation loss. The content loss describes the distortion of the image content, the generation-adversarial loss describes the statistical characteristics of the generated results and the distinction between data such as natural images, and the total variation loss describes the coherence of the image content. The overall loss function is weighted by three loss functions:

Here w _v , w _g , and w _t are their respective weights, and the weights are determined by experiments.

The calculation method of each loss function is given below.

(1) Loss of content

The traditional content loss function is represented by MSE (pixel mean square error), which examines the loss of image content pixel by pixel. MSE-based network training dilutes the high-frequency components of the image structure, resulting in excessive blurring of the image. To overcome this defect, the feature loss function of the image is introduced here. Since manually defining and extracting valuable image features is a complex task, and considering the ability of deep learning to automatically extract features, this method uses the hidden layer features obtained by VGG network training for measurement. Use φ _i,j to denote the feature map obtained by the jth convolutional layer in front of the ith pooling layer in the VGG network, and define the feature loss as the reconstructed image with the reference image The Euclidean distance of the VGG features, namely:

Here W _i,j , H _i,j represent the dimensions of the VGG feature map.

(2) Generation-Adversarial Loss

The generation-adversarial loss takes into account the production function of the GAN network, and encourages the network to produce a solution consistent with the manifold space of the natural image, so that the discriminator cannot distinguish the generated result from the natural image. The generation-adversarial loss is measured based on the discriminative probability of the discriminator for all training samples, and the formula is as follows:

here, Indicates that the discriminator D will reconstruct the result The probability of identifying a natural image; N represents the total number of training samples.

(3) Total variation loss

The purpose of increasing the total variation loss is to strengthen the local coherence of the learning results on the image content, and its calculation formula is:

Here W, H represent the width and height of the reconstructed image.

Step 4: Calculate the complexity of the input image, and select the corresponding GAN network reconstruction according to the complexity.

Specifically, it consists of the following sub-steps:

Step 4.1: Divide the input image evenly into 16 equal subgraphs, calculate the complexity of each subgraph, and judge whether it belongs to the type of high, medium and low complexity;

Step 4.2: Select the corresponding GAN network for super-resolution reconstruction according to the complexity type.

The present invention classifies sample images according to the complexity of the image content, constructs and trains deep network models with different complexity, and then selects the corresponding network for reconstruction according to the content complexity of the input image to be super-resolved. Remote sensing images record large-scale scenes. Because they are not affected by the fine information of ground objects, there are many spatial homogeneous areas with consistent content complexity and large areas, such as urban areas, dry fields, paddy fields, lakes, mountains and other large-scale objects. , so it is more suitable for pre-classification training and reconstruction.

The GAN deep learning network model is used here not only because the GAN network gives the best super-resolution performance at present, but also because the sources of high and low spatial resolution remote sensing images used as training samples are different and belong to multi-temporal images taken asynchronously. There may be one-to-one alignment in the pixel sense, which greatly limits the training of the CNN network, and the GAN network is an unsupervised learning network, so this problem does not exist.

It should be understood that the parts not described in detail in this specification belong to the prior art.

It should be understood that the above-mentioned descriptions for the preferred embodiments are relatively detailed, and should not therefore be considered as limiting the scope of the patent protection of the present invention. Within the scope of protection, replacements or modifications can also be made, all of which fall within the protection scope of the present invention, and the scope of protection of the present invention should be based on the appended claims.

Claims (13)

1. A remote sensing image super-resolution reconstruction method based on content-aware deep learning network, is characterized in that, comprises the following steps: Step 1: Collect high and low resolution remote sensing image samples and perform block processing; Step 2: Calculate the complexity of each image block, divide it into three categories according to the complexity: high, medium, and low, and form high, medium, and low complexity training sample sets respectively; Step 3: Use the obtained sample sets to train three GAN networks with high, medium and low complexity respectively; Step 4: Calculate the complexity of the input image, and select the corresponding GAN network reconstruction according to the complexity. 2. The remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 1, characterized in that: in step 1, the high-resolution image is evenly divided into 128x128 image blocks, low-resolution images Divide evenly into 64x64 image blocks. 3. the remote sensing image super-resolution reconstruction method based on content perception deep learning network according to claim 1, is characterized in that, the complexity of image block described in step 2, its computing method is: C=w _h ×H+w _u ×U+w _e ×E; Among them, C represents the complexity of the image block, H represents the image information entropy, U represents the image gray level consistency, R represents the image edge ratio, w _h , w _u , we _e are respective weights, and the weights are determined by experiments. 4. the remote sensing image super-resolution reconstruction method based on content perception deep learning network according to claim 3, is characterized in that, the computing formula of described image information entropy H is: <mrow> <mi>H</mi> <mo>=</mo> <mo>-</mo> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>K</mi> </munderover> <msub> <mi>n</mi> <mi>i</mi> </msub> <mo>/</mo> <mi>N</mi> <mo>.</mo> <mi>log</mi> <mrow> <mo>(</mo> <msub> <mi>n</mi> <mi>i</mi> </msub> <mo>/</mo> <mi>N</mi> <mo>)</mo> </mrow> <mo>;</mo> </mrow> Among them, N is the number of gray levels, n _i is the number of occurrences of each gray level, and K is the number of gray levels. 5. the remote sensing image super-resolution reconstruction method based on content perception deep learning network according to claim 3, is characterized in that, described image grayscale consistency U formula is: <mrow> <mi>U</mi> <mo>=</mo> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>M</mi> </munderover> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>j</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <msup> <mrow> <mo>(</mo> <mi>f</mi> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>-</mo> <mover> <mi>f</mi> <mo>&amp;OverBar;</mo> </mover> <mrow> <mo>(</mo> <mi>i</mi> <mo>,</mo> <mi>j</mi> <mo>)</mo> </mrow> <mo>)</mo> </mrow> <mn>2</mn> </msup> <mo>;</mo> </mrow> Among them, M and N are the number of rows and columns of the image respectively, f(i, j) is the gray value at the pixel (i, j), is the gray mean value of the 3×3 neighborhood pixels centered on (i,j). 6. the remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 3, is characterized in that, described image edge ratio R computing formula is: <mrow> <mi>R</mi> <mo>=</mo> <mfrac> <mi>E</mi> <mrow> <mi>M</mi> <mo>&amp;times;</mo> <mi>N</mi> </mrow> </mfrac> <mo>;</mo> </mrow> Among them, M and N are the number of rows and columns of the image respectively; E is the number of edge pixels in the image, which is calculated by the difference algorithm. 7. The remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to any one of claims 1-6, characterized in that: the high, medium and low complexity training sample sets described in step 2, wherein The number of image blocks in the high-complexity training sample set is not less than 500,000, the number of image blocks in the medium-complexity training sample set is not less than 300,000, and the number of image blocks in the low-complexity training sample set is not less than 200,000. 8. the remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 1, is characterized in that, the loss function of GAN network training in step 3 is defined as: <mrow> <mi>C</mi> <mo>=</mo> <msub> <mi>w</mi> <mi>v</mi> </msub> <mo>&amp;times;</mo> <msubsup> <mi>l</mi> <mrow> <mi>V</mi> <mi>G</mi> <mi>G</mi> </mrow> <mrow> <mi>S</mi> <mi>R</mi> </mrow> </msubsup> <mo>+</mo> <msub> <mi>w</mi> <mi>g</mi> </msub> <mo>&amp;times;</mo> <msubsup> <mi>l</mi> <mrow> <mi>G</mi> <mi>A</mi> <mi>N</mi> </mrow> <mrow> <mi>S</mi> <mi>R</mi> </mrow> </msubsup> <mo>+</mo> <msub> <mi>w</mi> <mi>t</mi> </msub> <mo>&amp;times;</mo> <msubsup> <mi>l</mi> <mrow> <mi>T</mi> <mi>V</mi> </mrow> <mrow> <mi>S</mi> <mi>R</mi> </mrow> </msubsup> <mo>;</mo> </mrow> 1 Among them, C represents the loss function of network training, Denotes the content loss function, Denotes the generative-adversarial loss function, Represents the total variation loss function; w _v , w _g , w _t are their respective weights, and the weights are determined by experiments. 9. the remote sensing image super-resolution reconstruction method based on content perception deep learning network according to claim 8, is characterized in that, described content loss function for: Among them, φ _i,j represents the feature map obtained by the jth convolutional layer in front of the i-th pooling layer in the VGG network, W _i,j , H _i,j represent the dimension of the VGG feature map; represents the reference image, Represents the reconstructed image. 10. the remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 8, is characterized in that, described generation-adversarial loss function for: <mrow> <msubsup> <mi>l</mi> <mrow> <mi>G</mi> <mi>A</mi> <mi>N</mi> </mrow> <mrow> <mi>S</mi> <mi>R</mi> </mrow> </msubsup> <mo>=</mo> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>n</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>N</mi> </munderover> <mo>-</mo> <mi>log</mi> <mi> </mi> <mi>D</mi> <mrow> <mo>(</mo> <mi>G</mi> <mo>(</mo> <msubsup> <mi>I</mi> <mi>n</mi> <mrow> <mi>L</mi> <mi>R</mi> </mrow> </msubsup> <mo>)</mo> <mo>)</mo> </mrow> </mrow> in, Indicates the reconstructed image, D(G(I ^LR )) indicates that the discriminator D will reconstruct the result The probability of identifying a natural image; N represents the total number of training samples. 11. the remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 8, is characterized in that, the total variation loss function for: <mrow> <msubsup> <mi>l</mi> <mrow> <mi>T</mi> <mi>V</mi> </mrow> <mrow> <mi>S</mi> <mi>R</mi> </mrow> </msubsup> <mo>=</mo> <mfrac> <mn>1</mn> <mrow> <mi>W</mi> <mi>H</mi> </mrow> </mfrac> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>x</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>W</mi> </munderover> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>y</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>H</mi> </munderover> <mo>|</mo> <mo>|</mo> <mo>&amp;dtri;</mo> <mi>G</mi> <msub> <mrow> <mo>(</mo> <msup> <mi>I</mi> <mrow> <mi>L</mi> <mi>R</mi> </mrow> </msup> <mo>)</mo> </mrow> <mrow> <mi>x</mi> <mo>,</mo> <mi>y</mi> </mrow> </msub> <mo>|</mo> <mo>|</mo> <mo>;</mo> </mrow> Among them, G(I ^LR ) represents the reconstructed image, and W and H represent the width and height of the reconstructed image. 12. the remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 1, is characterized in that, the specific realization of step 4 comprises the following sub-sub-steps: Step 4.1: Divide the input image evenly, calculate the complexity of each subimage, and judge whether it belongs to the type of high, medium, or low complexity; Step 4.2: Select the corresponding GAN network for super-resolution reconstruction according to the complexity type. 13. The remote sensing image super-resolution reconstruction method based on content-aware deep learning network according to claim 12, characterized in that: in step 4.1, the input image is evenly divided into 16 equal sub-images.