CN114092803A

CN114092803A - Cloud detection method and device based on remote sensing image, electronic device and medium

Info

Publication number: CN114092803A
Application number: CN202111284919.4A
Authority: CN
Inventors: 周严; 黄炎
Original assignee: Wuhan Zmvision Technology Co ltd
Current assignee: Wuhan Zmvision Technology Co ltd
Priority date: 2021-11-01
Filing date: 2021-11-01
Publication date: 2022-02-25

Abstract

The invention provides a cloud detection method, a device, electronic equipment and a medium based on a remote sensing image, wherein the method comprises the following steps: preprocessing is carried out on the basis of the remote sensing image to obtain a target image; and inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image. The cloud detection method, the cloud detection device, the electronic equipment and the medium based on the remote sensing image can increase the reception fields of the convolutions layer by layer under the condition of not losing information, so that the output of each convolution contains semantic information in a large range, and the fineness of image semantic segmentation can be improved. And further improve the accuracy of cloud detection.

Description

Cloud detection method and device based on remote sensing image, electronic device and medium

Technical Field

The invention relates to the technical field of image processing, in particular to a cloud detection method and device based on a remote sensing image, electronic equipment and a medium.

Background

The cloud detection task requires that given categories such as cloud and non-cloud are allocated to pixels of the remote sensing image, and spatial position information of the cloud and the non-cloud is accurately predicted at the same time, so that understanding of content pixel levels of the remote sensing image is achieved. Cloud detection has important significance in multiple fields such as weather forecast, energy exploration, fine agriculture and natural disaster system monitoring.

In the traditional cloud detection method, the classic Fmak algorithm only considers the characteristic of the pixel value of the remote sensing image, does not consider the spatial information characteristic, and is sensitive to image noise. In recent years, with the increase of computer power and the continuous development of deep learning, some conventional segmentation methods have been unable to compare the effect with that of the segmentation method based on deep learning. The FCN full convolution neural network can accept image input of any size, and up-samples the last convolution layer by deconvolution to restore the last convolution layer to the same size of the input image, so that the FCN network can classify each pixel of the image and simultaneously reserve spatial information of the input image. The FCN network is characterized in that the feature fusion is used for recovering image semantic information, and the network carries out point-by-point addition on a feature map of the last layer and a shallow feature map of the network, so that local prediction can be carried out while image global prediction is carried out, and richer semantic information is provided. Although the FCN adopts feature map accumulation to supplement image details, the final result map is still not fine enough, the deconvolution result is fuzzy and smooth, and is insensitive to the details in the image and has larger difference in cloud layer identification precision for different types of remote sensing images.

Disclosure of Invention

The invention provides a cloud detection method, a cloud detection device, electronic equipment and a medium based on a remote sensing image, which are used for overcoming the defect of poor cloud layer identification accuracy in the prior art and improving the cloud detection accuracy.

The invention provides a cloud detection method based on a remote sensing image, which comprises the following steps:

preprocessing is carried out on the basis of the remote sensing image to obtain a target image;

inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image;

wherein the cloud detection model comprises:

the multi-scale down-sampling layer is used for performing down-sampling operation and convolution operation of different scales on the basis of the target image to obtain first characteristic images of different scales;

the first fusion layer is used for carrying out feature fusion based on the first feature images with different scales to obtain a first fusion image;

the multi-scale up-sampling layer is used for performing up-sampling operation and convolution operation of different scales on the basis of the first fusion image to obtain second characteristic images of different scales;

the second fusion layer is used for carrying out feature fusion based on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image;

the output layer is used for acquiring a cloud detection result corresponding to the target image based on the second fusion image;

wherein the multi-scale down-sampling layer comprises a plurality of cascaded levels of down-sampling sub-layers, the multi-scale up-sampling layer comprises a plurality of cascaded levels of up-sampling sub-layers, and the number of down-sampling sub-layers is the same as the number of up-sampling sub-layers.

According to the cloud detection method of the remote sensing image, provided by the invention, the preprocessing is carried out based on the remote sensing image to obtain the target image, and the method comprises the following steps:

based on a preset formula, obtaining the brightness temperature value and the reflectivity of each pixel point of the remote sensing image;

and acquiring a target image based on the brightness temperature value and the reflectivity of each pixel point.

According to the cloud detection method of the remote sensing image provided by the invention, the feature fusion is carried out based on the first feature images with different scales to obtain a first fusion image, and the method comprises the following steps:

performing corresponding pooling operation based on the first characteristic images of all scales to obtain all first images with the same two-dimensional scale;

and performing fusion operation and convolution dimensionality reduction operation on the basis of the first images to obtain the first fusion images.

According to the cloud detection method of the remote sensing image, the up-sampling operation and convolution operation of different scales are carried out based on the first fusion image, and second characteristic images of different scales are obtained, and the method comprises the following steps:

based on the first fusion image, performing level-by-level up-sampling operation to obtain second images with different two-dimensional scales;

and performing corresponding convolution operation based on each second image to obtain each second characteristic image with different scales.

According to the cloud detection method of the remote sensing image, the feature fusion is performed on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image, and the method comprises the following steps:

performing splicing operation and up-sampling operation based on the first characteristic image and the second characteristic image with the same scale to obtain each third image;

and performing feature fusion based on each third image to obtain the second fusion image.

According to the cloud detection method of the remote sensing image, the up-sampling operation comprises up-sampling operation based on a bilinear interpolation method.

The invention also provides a cloud detection device of the remote sensing image, which comprises:

the preprocessing module is used for preprocessing based on the remote sensing image to obtain a target image;

the detection module is used for inputting the target image into a cloud detection model and outputting a cloud detection result corresponding to the target image;

wherein the cloud detection model comprises:

The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the program to realize the steps of the cloud detection method of the remote sensing image.

The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method of cloud detection of remote sensing images as any one of the above.

The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the method for cloud detection of remote sensing images as described in any one of the above.

The invention provides a cloud detection method, a device, electronic equipment and a medium based on remote sensing images with multi-channel numbers, wherein a target image is obtained based on the remote sensing images with multi-channel numbers, first feature images with multiple scales are extracted through a plurality of cascaded levels of downsampling sublayers, a first fusion image is obtained through feature fusion of the first feature images with multiple scales, the first fusion image is restored into a second feature image with multiple scales through a plurality of cascaded levels of upsampling sublayers, the first feature image with the same scale and the first feature image are spliced and fused to obtain a second fusion image, and a cloud detection result is restored through the second fusion image. Under the condition of not losing information, the receptive field of the convolution can be increased layer by layer, so that the output of each convolution contains semantic information in a larger range, the fineness of image semantic segmentation can be improved, and the accuracy of cloud detection is further improved.

Drawings

In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed for the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.

FIG. 1 is a schematic flow chart of a cloud detection method based on remote sensing images provided by the invention;

FIG. 2 is a schematic structural diagram of a cloud detection model based on remote sensing images provided by the invention;

FIG. 3 is a schematic structural diagram of a cloud detection device based on remote sensing images provided by the invention;

fig. 4 is a schematic structural diagram of an electronic device provided in the present invention.

Detailed Description

In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Fig. 1 is a schematic flow chart of a cloud detection method based on remote sensing images provided by the invention. As shown in fig. 1, the cloud detection method based on a remote sensing image provided by the embodiment of the present invention includes: step 101, preprocessing is carried out based on the remote sensing image, and a target image is obtained.

The execution subject of the cloud detection method based on the remote sensing image provided by the embodiment of the invention is the cloud detection device based on the remote sensing image. The detection object of the cloud detection device based on the remote sensing image is the remote sensing image.

The remote sensing image is obtained by remote sensing means to obtain electromagnetic spectrum information of the ground feature in different wave bands. The remote sensing image can be divided according to different wave bands to generate two-dimensional arrays of different channels to represent ground feature information of different wave bands, and pixel values in the two-dimensional arrays are called brightness values (or called gray values and DN values).

Specifically, in step 101, the cloud detection device based on the remote sensing image performs corresponding preprocessing on the remote sensing image according to the set requirement, and obtains a target image.

The target image is an image which is compared with the original remote sensing image and is close to the real image in terms of geometry and radiation as much as possible. The target image has at least more than three channels, and the resolution and the width, height and size of the gray scale images corresponding to different channels are different. The target image is used for being input into the cloud detection model so as to obtain a corresponding cloud detection result.

And the preprocessing refers to processing performed before the cloud detection model is applied to the input original remote sensing image. Preprocessing is used to eliminate distorted information in the remote sensing image, so that the generated target image contains useful real information, the detectability of related information is enhanced, and the data is simplified to the maximum extent. The embodiment of the present invention does not specifically limit the pretreatment.

Illustratively, the preprocessing can be radiation correction, and the remote sensor calibration, the atmospheric correction (or solar altitude correction) and the terrain correction are respectively carried out on three kinds of radiation distortion generated by the characteristics of the remote sensor, the illumination conditions of the ground objects (terrain influence and solar altitude influence) and the atmospheric action in the remote sensing image by the cloud detection device based on the remote sensing image so as to obtain the target image.

For example, the preprocessing can also be geometrically corrected, and the cloud detection device based on the remote sensing image needs to select control point pairs respectively on the remote sensing image to be corrected and a standard graph with coordinate information, perform pixel coordinate transformation, and resample the brightness value of a pixel to obtain a target image.

Preferably, the cloud detection device based on the remote sensing image performs corresponding preprocessing by using different channel data according to cloud layer characteristics of different scenes to generate a target image, so as to input the target image to the cloud detection model and detect whether cloud exists in the remote sensing image.

And 102, inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image.

Wherein, the cloud detection model includes:

and the multi-scale down-sampling layer is used for performing down-sampling operation and convolution operation of different scales based on the target image to acquire first characteristic images of different scales.

And the first fusion layer is used for carrying out feature fusion based on the first feature images with different scales to obtain a first fusion image.

And the multi-scale up-sampling layer is used for performing up-sampling operation and convolution operation of different scales on the basis of the first fusion image to obtain second characteristic images of different scales.

And the second fusion layer is used for carrying out feature fusion based on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image.

And the output layer is used for acquiring a cloud detection result corresponding to the target image based on the second fusion image.

The multi-scale down-sampling layer comprises a plurality of cascaded levels of down-sampling sub-layers, the multi-scale up-sampling layer comprises a plurality of cascaded levels of up-sampling sub-layers, and the number of the down-sampling sub-layers is the same as that of the up-sampling sub-layers.

The cloud detection model is obtained by training based on remote sensing image sample data and a predetermined remote sensing image type label.

The cloud detection model can be an artificial intelligence model, and the model type is not particularly limited in the embodiment of the invention.

For example, the cloud detection model may be a Deep Convolutional Neural Network (DCNN) model, and the DCNN is used as a data-driven algorithm, performs hierarchical feature extraction on data by means of a sampling layer and a Convolutional layer which are stacked layer by layer, and performs iterative optimization on model parameters by using a back propagation algorithm. The structure and parameters of the DCNN model include, but are not limited to, the number of layers of the model, and the weight parameters of each layer, etc.

The sample data includes remote sensing image data and a remote sensing image type tag corresponding to the sample data. The remote sensing image sample data is divided into a training set and a test set, and the sample proportion of the training set and the test set is not particularly limited in the embodiment of the invention.

Illustratively, sample data is 12000 pieces of acquired satellite load simulation data, the data is divided according to the ratio of 3:1, 9000 pieces of data are used as a training set for model training, and 3000 pieces of data are used as a test set for model verification. The embodiment of the invention does not specifically limit the training of the cloud detection model.

For example, the model may be trained directly using a training set, and the test set may be used to obtain the test results.

For example, according to different remote sensing data distribution scenes, ocean region remote sensing data with the latitude of 60 degrees S-60 degrees N in the test set can be divided into ocean scene test data, land region remote sensing data with the latitude of 60 degrees S-60 degrees N can be divided into land scene test data, and polar region remote sensing data with the latitudes of 60 degrees N-90 degrees N and 60 degrees S-90 degrees S can be divided into polar scene test data. Wherein, the daytime scene is that the sun zenith angle is less than 85 degrees.

The training set and the test set are divided into 4 types and applied to a cloud detection model to derive 4 types of cloud detection models, namely a full-landform all-day-time infrared model, a marine day model, a land day model and a polar all-day-time infrared model. In order to simplify the model, all models are binary models, the value of each pixel point is 0 or 1, 0 represents that the pixel point is a background, and 1 represents that the pixel point is a cloud. I.e. the true value mask is a binary result: "cloud" and "not cloud". When the cloud detection model is used in a specific scene, the model can be trained through the training set corresponding to the scene, so that the model is more targeted.

In the cloud detection model training process, the cloud detection device based on the remote sensing image adopts a Pytorch frame, and processes the image format of the input target image into a pitch height width channel by using a torch _ null function, wherein height is the height of the target image, width is the width of the target image, and channel is the number of channels of the target image. And the training samples with the number of batchsize are selected in one training and input into the model, and then the gradients of the training samples are calculated for back propagation, so that the utilization rate of the memory is improved.

Specifically, in step 102, after the cloud detection device based on the remote sensing image sets the cloud detection model according to the trained model parameters, the model performs cloud detection on any target image in step 101, so as to obtain a cloud detection result corresponding to the target image.

The embodiment of the present invention does not specifically limit the form of the behavior detection result.

For example, the cloud detection result may be a binary array having the same width and height as the target image, the value range of any one value in the array is [0, 1], where 0 is that the pixel is cloud-free and 1 is that the pixel is cloud-free, and it can be known which areas in the image are cloud through the position information of the pixel in the two-dimensional array.

For example, the cloud detection result may be a statistical value, and it may be indicated that the target image is "cloud" or "cloud-free" by a numerical value obtained by performing statistics on all pixel points,

the embodiment of the invention does not specifically limit the cloud detection model.

Illustratively, the cloud detection device based on the remote sensing image detects whether cloud exists in the remote sensing image by utilizing a cloud detection model, and the model at least comprises an input layer, a hidden layer and an output layer.

The input layer directly receives the target image generated in step 101 at the foremost part of the entire network.

The hidden layer at least comprises four layers, namely a multi-scale down-sampling layer, a first fusion layer, a multi-scale up-sampling layer and a second fusion layer. The hidden layer is used for performing down-sampling operation on the target image layer by layer to obtain feature maps of different levels of remote sensing semantic information. And splicing the characteristic graphs of each level on the channel dimension to obtain a multi-level characteristic fusion image. And performing up-sampling operation on the multi-level feature fusion image, and recovering the spatial information of the image layer by layer. And splicing the feature map of each level extracted in the down-sampling process and the feature map of each level restored in the up-sampling process, and re-extracting the features.

The output layer is the last layer, the output layer is used for performing full-connection dimensionality reduction by adopting convolution kernels with the size of 1x1 and the number of 1 to obtain a characteristic diagram with the width and the height identical to those of the original image, and the characteristic diagram is a gray level diagram with the channel dimensionality of 1, namely the final cloud detection result.

Fig. 2 is a schematic structural diagram of a cloud detection model based on a remote sensing image provided by the invention. As shown in fig. 2, the hidden layer of the cloud detection model may include:

the multi-scale down-sampling layer at least comprises 4 down-sampling sublayers in cascade connection, each down-sampling sublayer is followed by a pooling layer, and each pooling layer is connected to the first fusion layer. After the processing of the first fusion layer, the result is input to a multi-scale up-sampling layer, the multi-scale up-sampling layer at least comprises 4 cascaded up-sampling sublayers, and an anti-convolution layer is connected behind each up-sampling sublayer. The system is provided with four jump connection chains, a down-sampling sublayer 4 is connected with an up-sampling sublayer 1, a down-sampling sublayer 3 is connected with an up-sampling sublayer 2, the down-sampling sublayer 2 is connected with the up-sampling sublayer 3, the down-sampling sublayer 1 is connected with the up-sampling sublayer 4, the splicing result of each jump connection chain is sent to a second fusion layer, the second fusion layer is connected to an output layer (namely, a full connection layer), and an activation function is a ReLU function.

The multi-scale down-sampling layer has the function of extracting a high-dimensional characteristic diagram from a matrix corresponding to a remote sensing image through a large number of convolution kernels, so that a plurality of characteristic diagrams can be extracted, each characteristic diagram is local perception extracted from a picture, and interested parts in the picture can be extracted by integrating the characteristic diagrams. The convolution kernels of the four downsampling sublayers adopted by the embodiment of the invention are all 1x1, the number of the convolution kernels is respectively 64, 128, 256 and 512, downsampling is carried out by using pooling operation, and the size of a pooling window is 2. And (3) performing convolution in each layer and then pooling, so that the height and the width of the feature map of the image processed in each layer are shortened by half, and the number of channels is doubled. The method can effectively extract the feature information of the picture, reduce the dimension of the feature, compress the number of data and parameters, and reduce the training fitting.

The first fusion layer is used for pooling all feature maps output by the multi-scale down-sampling layer to reduce feature sizes of the feature maps, the size of a pooling window is 16, 8, 4 and 2 in sequence, so that the height and the width of all the feature maps are the same, and multi-level feature fusion is performed according to channel dimensions to obtain a fused feature map.

The multi-scale up-sampling layer is used for restoring the fused feature map into a picture. The fused feature map is subjected to up-sampling, the width and height of the image are amplified to be twice of the original size, then deconvolution operation is carried out, and spatial information (channel dimensionality) of the image is restored layer by layer.

The second fusion layer is used for performing jump splicing on the feature map of the up-sampling sublayer of each layer and the feature map of the down-sampling sublayer of the corresponding layer, and performing convolution by using a convolution kernel with the size of 3x3 to re-extract features. And then, sequentially reducing the widths and heights of all the feature maps to be the same as the width and height of the target image by using the upsampling multiples of 2, 4, 8 and 16, and generating a fused image through feature fusion.

In the multilayer neural network, the excitation function ReLU is used for carrying out nonlinear mapping on the feature map obtained by convolution in the output layer, and after multilayer convolution, the value of the feature vector map is not changed greatly during training, so that the gradient disappears, and further the convolutional network model cannot be trained. After each convolution, the ReLU excitation function is used, nonlinear mapping can be carried out on the linear calculation result of the convolution layer, so that the change of the characteristic diagram before and after training is not too small, and the model can be trained.

It is understood that the loss function used in the cloud detection model training process is not specifically described in the embodiments of the present invention.

Illustratively, the loss function is a binary _ cross entropy loss function, and the calculation formula is as follows:

wherein i represents the position of one pixel point in the cloud detection result, y'_iThe predicted pixel value, y, representing the ith point_iRepresenting the real pixel value of the ith point.

Illustratively, the Adam function is taken as an optimization function, first-order gradient optimization is performed on the loss function, and the size of the loss function value is reduced.

It is understood that after step 102, the cloud detection model may be evaluated using the image segmentation evaluation index mlou. The calculation formula is as follows:

FN is false negative, which means that the true value of the target image is 1, and the true value of the cloud detection result is 0; TP is true, the true value of the target image is 1, and the true value of the cloud detection result is 1; FP is false positive, indicating that the true value of the target image is 0 and the true value of the cloud detection result is 1.

Compared with the conventional CNN convolutional neural network, the mIoU score of the cloud detection model in the cloud detection method based on the remote sensing image is higher. Meanwhile, the network with the Resnet34 as the skeleton has very fast calculation speed and higher accuracy than the structure without the residual error network.

The method comprises the steps of obtaining a target image based on a multi-channel number remote sensing image, extracting first feature images of multiple scales through cascaded downsampling sublayers of multiple levels, carrying out feature fusion on the first feature images of the multiple scales to obtain a first fusion image, restoring the first fusion image into second feature images of the multiple scales through cascaded upsampling sublayers of the multiple levels, splicing and fusing the first feature images of the same scales and the first feature images to obtain a second fusion image, and restoring a cloud detection result through the second fusion image. Under the condition of not losing information, the receptive field of the convolution can be increased layer by layer, so that the output of each convolution contains semantic information in a larger range, the fineness of image semantic segmentation can be improved, and the accuracy of cloud detection is further improved.

On the basis of any one of the above embodiments, the preprocessing is performed based on the remote sensing image to obtain the target image, and the method includes: and acquiring the brightness temperature value and the reflectivity of each pixel point of the remote sensing image based on a preset formula.

Here, the reflectance refers to a percentage of the reflected energy to the incident energy. The reflectivity of different ground objects is different, which depends on the nature of the ground objects, the wavelength and the incident angle of the incident electromagnetic wave, and the size range of the reflectivity is always less than or equal to 1. The reflectivity is used for characterizing the property of the ground object in the visible light wave band.

The brightness temperature value refers to the physical temperature of a black body when the radiance of an object is equal to the radiance of the black body, which is called the "brightness temperature" of the object. The brightness temperature value is used for representing the ground object radiation brightness of the ground object in the infrared band

The preset formula is a calibration calculation formula for converting the gray value of each pixel point in the remote sensing data into a reflectivity or brightness temperature value.

Specifically, in step 101, the cloud detection device based on the remote sensing image acquires original image data of the remote sensing satellite with multi-channel and high resolution, and performs calibration calculation according to bands corresponding to different channels, so as to convert the gray value of each pixel point in the gray level maps of the different channels into a brightness value or a reflectivity.

If the wave band corresponding to a certain channel is a visible light wave band, the cloud detection device based on the remote sensing image obtains the reflectivity corresponding to each pixel point in the gray scale map of the channel according to a calibration calculation formula.

And if the wave band corresponding to a certain channel is an infrared wave band, the cloud detection device based on the remote sensing image obtains the brightness temperature value corresponding to each pixel point in the gray scale image of the channel according to a calibration calculation formula.

Specifically, in step 101, the cloud detection device based on the remote sensing image updates each pixel value in the grayscale images corresponding to all target channels to the brightness temperature value or the reflectivity corresponding to the target channel according to the preset channel selection rule, and merges all the updated images into the target image.

The digital image may be represented in the form of a matrix, the pixel data of a grayscale image being a matrix, the rows of the matrix corresponding to the height of the image (in pixels), the columns of the matrix corresponding to the width of the image (in pixels), the elements of the matrix corresponding to the pixels of the image, the values of the elements of the matrix being the grayscale values of the pixels.

The target image may be represented in the form of a three-dimensional array, where a first dimension and a second dimension of the array respectively correspond to a width and a height of the target image, a third dimension corresponds to a number of channels of the target image, and each element in the three-dimensional array corresponds to a pixel in the target image at a position corresponding to any one of the group of the width, the height, and the number of channels. The number of channels of the target image is less than or equal to the number of channels of the remote sensing image, and the number of channels of the target image is not particularly limited in the invention.

Illustratively, the cloud detection device based on the remote sensing image can convert the grayscale images of 25 channels of the remote sensing image into corresponding bright temperature images and reflectivity images according to the calibration calculation process of the remote sensing image, and select 3 visible light channels, 1 near infrared channel, 6 middle infrared channels and 6 far infrared channels, which are 10 channels in total. Because the different channels have different corresponding resolutions, the widths and the heights of the different channels are not uniform, so that data processing needs to be performed according to gray value data, a scaling calculation formula and a spatial resolution of the different channels, so that the widths and the heights of the channels in the target image are uniform. The corresponding relationship of the related data of different channels is shown in table 1.

TABLE 1 channel data corresponding relationship schematic table

If the channels 1 to 4 are in the same size, the cloud detection device based on the remote sensing image can judge that the channels belong to the visible light band or the near infrared band according to the corresponding wavelength range, and then the DN value of each pixel point of the picture of each channel is calibrated and calculated to be converted into the reflectivity. And because the spatial resolution of the channels 3 and 4 is 2 times that of the channels 1 and 2, the reflectivity images corresponding to the channels 3 and 4 need to be interpolated to expand the width and height of the reflectivity images to 2 times that of the original reflectivity images, so that the width and height of the reflectivity images of the channels 3 and 4 are the same as those of the reflectivity images of the channels 1 and 2.

Similarly, for the channels 20 to 25, the cloud detection device based on the remote sensing image can judge that the remote sensing image belongs to the middle infrared band or the far infrared band according to the corresponding wavelength range, and then the DN value of each pixel point of the picture of each channel is calibrated and calculated to be converted into the brightness temperature rate. And because the spatial resolution of the channels 20-25 is 4 times that of the channels 1 and 2, the brightness temperature value images corresponding to the channels 20-25 need to be interpolated to expand the width and height of the brightness temperature value images to 4 times that of the original brightness temperature value images, so that the width and height of the reflectivity images of the channels 20-25 are the same as those of the reflectivity images of the channels 1 and 2.

The embodiment of the invention determines the selected channel through the channel data corresponding relation based on the remote sensing image with the multi-channel number, and updates the original gray value of the corresponding channel into the corresponding brightness temperature value and reflectivity by utilizing radiometric calibration calculation so as to obtain the target image. The method can correct errors in the acquired remote sensing image, can improve the fineness of image semantic segmentation, and further improves the accuracy of cloud detection.

On the basis of any one of the above embodiments, performing feature fusion based on first feature images of different scales to obtain a first fusion image includes: and performing corresponding pooling operation based on the first characteristic images of all scales to obtain all the first images with the same two-dimensional scale.

The first feature image is a feature image extracted by convolving and pooling a target image. And the scales of the corresponding generated first characteristic images are different due to different degrees of convolution and pooling.

Illustratively, a target image with an image size of 4 × 512 × 10 is input to the cloud detection model, and the cascaded downsampling sublayers and the corresponding connected convolution layers are used, wherein the sizes of the convolution kernels are 64, 128, 256 and 512 respectively, and the downsampling is performed by using a pooling window with the size of 2, so that a first feature image a1 with the size of 4 × 256 × 64, a first feature image a2 with the size of 4 × 128, a first feature image A3 with the size of 4 × 64 × 256, and a first feature image a4, a1, a2, A3 and a4 with the size of 4 × 32 × 512 respectively contain information of the target image at different levels.

Specifically, in step 102, the cloud detection device based on the remote sensing image performs pooling operation on the first feature images with different scales by using the corresponding pooling windows according to the corresponding relationship between the scales and the pooling windows to generate first images corresponding to the number of channels.

The first image is an image generated by pooling the first feature image. The first images corresponding to the first feature images with different scales have the same width and height and different channel numbers, and the different first images and the corresponding first feature images have the same channel numbers. The embodiment of the present invention does not specifically limit the process of the reflooling.

Illustratively, the cloud detection device based on the remote sensing images performs pooling of the first feature images a1, a2, A3 and a4 to reduce the feature sizes thereof, and the pooled window sizes are 16, 8, 4, 2 in order, resulting in a first image B1 with a scale of 4 × 16 × 64, a first image B2 with a scale of 4 × 16 × 128, a first image B3 with a scale of 4 × 16 × 256, and a first image B4 with a scale of 4 × 16 × 512, respectively.

And performing fusion operation and convolution dimensionality reduction operation based on each first image to obtain a first fusion image.

Specifically, in step 102, the cloud detection device based on the remote sensing image performs feature fusion according to different channel dimensions of all the first images to obtain a first fusion image.

The first fused image is an image obtained by performing feature fusion on all the first images according to the number of channels. The first fused image contains a set of semantic information of the target image at different levels.

Preferably, after feature fusion is performed on all the first images by the cloud detection device based on the remote sensing image, convolution is performed to reduce the channel dimension of the first fused image. The embodiment of the present invention does not specifically limit this process.

Illustratively, the remote sensing image-based cloud detection device performs multi-level feature fusion on the first images B1, B2, B3 and B4 in a channel dimension to obtain a feature map with the size of 4 × 16 × 960, in order to prevent the feature dimension from being too high, an overfitting phenomenon is generated, the fused feature map is subjected to convolution dimensionality reduction operation again, the convolution kernel size is 256, and a first fused image B with the scale of 4 × 16 256 is obtained.

According to the embodiment of the invention, pooling is carried out based on the first characteristic images of all scales to obtain the first images with uniform width and height, and the first images with different channel numbers are fused and subjected to dimensionality reduction to obtain the first fused image. The image can simultaneously contain low-level detail features and high-level semantic features, the fineness of image semantic segmentation can be improved, and the accuracy of cloud detection is further improved.

On the basis of any one of the above embodiments, performing upsampling operation and convolution operation of different scales based on the first fusion image to obtain second feature images of different scales includes: and performing level-by-level up-sampling operation based on the first fusion image to obtain second images with different two-dimensional scales.

Specifically, in step 102, the remote sensing image-based cloud detection apparatus performs an upsampling operation on the first fused image by using the corresponding upsampling window according to the correspondence between the scale and the upsampling window, and restores the width and the height of the first fused image to second images which are the same as the width and the height of the first feature images a1, a2, A3 and a4, respectively.

The second image is an image generated by up-sampling and expanding the width and height of the first fused image B layer by layer. The second images of different width and height dimensions have the same number of channels and the same width and height as the corresponding first images. The upsampling process is not particularly limited in the embodiments of the present invention.

Illustratively, the cloud detection device based on the remote sensing images performs upsampling on the width and height of the first fused image B layer by layer to twice the original size, and a second image C1 with the scale of 4 × 32 × 256, a second image C2 with the scale of 4 × 64 × 256, a second image C3 with the scale of 4 × 128 and a second image C4 with the scale of 4 × 256.

The upsampling method in the embodiment of the present invention is not particularly limited, and the upsampling method includes, but is not limited to, bilinear interpolation, trilinear interpolation, nearest neighbor interpolation, bicubic interpolation, or the like.

It should be noted that the second feature image is a feature image restored after upsampling and deconvolution of the first fused image. And the sizes of the corresponding restored second characteristic images are different due to different degrees of up-sampling and deconvolution.

Specifically, the cloud detection device based on the remote sensing image performs deconvolution operation on second images with different width and height sizes by using corresponding convolution kernels according to the corresponding relationship between the width and height sizes and the convolution kernels to generate second characteristic images different from the number of channels.

The second feature image is a feature image restored by deconvoluting the second image. And deconvolution of different degrees ensures that the number of channels of the second characteristic image correspondingly generated according to the second image is different.

Illustratively, the cloud detection device based on the remote sensing images performs deconvolution operations on the second images C1, C2, C3 and C4 respectively according to convolution kernels with the sizes of 512, 256, 128 and 64, restores the number of channels of the images in a hierarchy by hierarchy, restores the second image C1 to obtain a second feature image D1 with the scale of 4 x 32 x512, restores the second image C2 to obtain a second feature image D2 with the scale of 4 x 64 x 256, restores the second image C3 to obtain a second feature image D3 with the scale of 4 x 128, and restores the second image C4 to obtain a second feature image D4 with the scale of 4 x 256 x 64.

The embodiment of the invention performs upsampling and convolution of different scales based on the first fusion image, and respectively restores the scales of the first fusion image to the second characteristic images with the scales corresponding to the first characteristic images. The rich features contained in the first fusion image can be restored to different degrees, the fineness of image semantic segmentation can be improved, and the accuracy of cloud detection is further improved.

On the basis of any one of the above embodiments, performing feature fusion based on the first feature image and the second feature image corresponding to the scale to obtain a second fusion image includes: and performing splicing operation and up-sampling operation based on the first characteristic image and the second characteristic image with the same scale to obtain each third image.

It should be noted that, in the multi-scale down-sampling layer of the cloud detection model in step 102, the down-sampling sublayers cascaded from top to bottom are respectively jump-linked with the up-sampling sublayers cascaded from bottom to top in the multi-scale up-sampling layer, so that the feature images with the same scale are integrated.

Specifically, the cloud detection device based on the remote sensing image splices and fuses the first characteristic image and the second characteristic image corresponding to each skip connection chain according to skip connection corresponding to each sublayer in a multi-scale down-sampling layer and a multi-scale up-sampling layer in the cloud detection model, and obtains a third image.

The third image is an image including the features extracted in the down-sampling process and the features restored in the up-sampling process.

Preferably, the cloud detection device based on the remote sensing image splices and fuses the first characteristic image and the second characteristic image corresponding to each skip connection chain according to skip connection corresponding to each sublayer in a multi-scale down-sampling layer and a multi-scale up-sampling layer in the cloud detection model, and then performs up-sampling operation, so that the width and height of each obtained third image are the same as the width and height of the target image. The embodiment of the present invention does not specifically limit this process.

Illustratively, the cloud detection device based on the remote sensing image splices the first feature images a4 at two ends of the jump link of the fourth layer with the second feature images D1 to obtain feature maps with a scale of 4 × 32 × 1024 corresponding to the fourth layer, and performs convolution by a convolution kernel of 3 × 3 to re-extract features to obtain a third image E4 with a scale of 4 × 32 × 512, and upsamples the scale of E4 to 4 × 515 × 512.

Similarly, the first feature image a1 at both ends of the first layer of jumper links is stitched with the second feature image D4 to obtain a third image E1 with a scale of 4 × 515 × 512 × 64.

The first feature images a2 at both ends of the second layer of jumpers were stitched to the second feature image D3 to obtain a third image E2 with a scale of 4 × 515 × 512 × 128.

The first feature images a3 at both ends of the jumper link of the third layer were stitched to the second feature image D2 to obtain a third image E3 with a scale of 4 × 515 × 512 × 256.

And performing feature fusion based on each third image to obtain a second fusion image.

Specifically, in step 102, the cloud detection device based on the remote sensing image performs feature fusion according to different channel dimensions of all the third images to obtain a second fusion image.

The second fused image is an image obtained by performing feature fusion on all the third images according to the number of channels. The second fused image contains a set of semantic information extracted and restored at different levels of the target image.

Illustratively, the cloud detection device based on the remote sensing images performs feature fusion on the third images D1, D2, D3 and D4 to obtain a second fused image D with the scale of 4 × 512 × 960.

It can be understood that the cloud detection device based on the remote sensing image inputs the second fused image D to the last layer of the cloud detection model for pixel level detection. The embodiment of the present invention is not particularly limited thereto.

Illustratively, the cloud detection device based on the remote sensing image randomly discards 0.5 times of feature dimensions of the second fused image D, and finally performs dimensionality reduction by using a convolution layer with the size of 1 × 1 and the size of a convolution kernel of 1 to obtain feature maps of 4 × 512 × 1, which are the same as the size of the original image, wherein each feature map is a grayscale map with the channel dimension of 1.

And comparing the value of any pixel point in each gray scale image with a preset threshold, and if the value of the pixel point is greater than or equal to the preset threshold, indicating that the detection result of the pixel point is 'cloud', and updating the value of the pixel point to 1.

If the value of the pixel point is smaller than the preset threshold, the detection result of the pixel point is that the pixel point is not cloud, and the value of the pixel point is updated to be 0.

Therefore, the cloud detection result output by the cloud detection model may be a cloud detection label map, and the image size is 512x512x1, that is, the value set of the pixel values is {0,1 }. 0 represents that the point is a non-cloud part, and 1 represents that the point is a cloud.

The embodiment of the invention fuses the first characteristic image and the first fused image of the second characteristic image based on the same level, and restores the width and height of the first characteristic image and the second characteristic image to the second fused image which is the same as the target image. The features extracted and restored in different degrees can be integrated, the fineness of image semantic segmentation can be improved, and the accuracy of cloud detection is further improved.

On the basis of any of the above embodiments, the upsampling operation includes an upsampling operation based on a bilinear interpolation method.

Specifically, in step 102, the cloud detection device based on the remote sensing image performs upsampling layer by using bilinear interpolation on a plurality of cascaded levels of upsampling sublayers in the multi-scale upsampling layer, and gradually reduces the width and height dimensions to be the same as those of the target image.

Illustratively, the known function f is at Q₁₁＝(x1，y1)，Q₁₂＝(x1，y2)，Q₂₁(x2, y1), and Q₂₂The pixel values of four points (x2, y2) require that the size of the pixel values at the point P (x, y) is:

and performing linear interpolation in the x direction of the coordinate axis to obtain:

and performing linear interpolation in the y direction of the coordinate axis to obtain:

the size of the pixel value at the point P ═ x, y is obtained.

According to the embodiment of the invention, the bilinear interpolation method is adopted for carrying out the up-sampling operation, so that the extracted feature size can be restored, and the accuracy of cloud detection and the fineness of image segmentation are further improved.

Fig. 3 is a schematic structural diagram of a cloud detection device based on a remote sensing image provided by the invention.

As shown in fig. 3, the apparatus includes: a preprocessing module 310 and a detection module 320, wherein:

and the preprocessing module 310 is configured to perform preprocessing based on the remote sensing image to obtain a target image.

The detection module 320 is configured to input the target image into the cloud detection model, and output a cloud detection result corresponding to the target image.

Wherein, the cloud detection model includes:

Specifically, the pre-processing module 310 and the detection module 320 are electrically connected in sequence.

The preprocessing module 310 updates each pixel value in the grayscale images corresponding to all target channels to the brightness temperature value or the reflectivity corresponding to the target channel according to a preset channel selection rule, and merges all the updated images into a target image.

After the detection module 320 sets the cloud detection model according to the trained model parameters, cloud detection is performed on any target image in the preprocessing module 310 through the model, so that a cloud detection result corresponding to the target image can be obtained.

The pre-processing module 310 further comprises a first acquisition unit and a target image acquisition unit, wherein:

the first obtaining unit is used for obtaining the brightness temperature value and the reflectivity of each pixel point of the remote sensing image based on a preset formula.

And the target image acquisition unit is used for acquiring a target image based on the brightness temperature value and the reflectivity of each pixel point.

The detection module 320 further comprises a first pooling unit and a first fusion unit, wherein:

and the first pooling unit is used for performing corresponding pooling operation based on the first characteristic images of all scales to obtain all the first images with the same two-dimensional scale.

And the first fusion unit is used for performing fusion operation and convolution dimensionality reduction operation on the basis of each first image to acquire a first fusion image.

The detection module 320 further comprises an upsampling unit and a second obtaining unit, wherein:

and the up-sampling unit is used for carrying out up-sampling operation level by level based on the first fusion image and acquiring each second image with different two-dimensional scales.

And the second acquisition unit is used for performing corresponding convolution operation on the basis of each second image to acquire each second characteristic image with different scales.

The detection module 320 further includes a splicing unit and a second fusion unit, wherein:

the splicing unit is used for carrying out splicing operation and up-sampling operation on the basis of the first characteristic image and the second characteristic image with the same scale to obtain each third image;

and the second fusion unit is used for carrying out feature fusion based on each third image to obtain a second fusion image.

The upsampling operation of the detection module 320 includes an upsampling operation based on bilinear interpolation.

The cloud detection device based on the remote sensing image provided by the embodiment of the invention is used for executing the cloud detection method based on the remote sensing image, the implementation mode of the cloud detection device based on the remote sensing image provided by the invention is consistent with that of the cloud detection method based on the remote sensing image provided by the invention, the same beneficial effects can be achieved, and the description is omitted here.

Fig. 4 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 4: a processor (processor)410, a communication interface (communication interface)420, a memory (memory)430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 are communicated with each other via the communication bus 440. Processor 410 may invoke logic instructions in memory 430 to perform a method of remote sensing image-based cloud detection, the method comprising: preprocessing is carried out on the basis of the remote sensing image to obtain a target image; inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image; wherein, the cloud detection model includes: the multi-scale down-sampling layer is used for performing down-sampling operation and convolution operation of different scales based on the target image to obtain first characteristic images of different scales; the first fusion layer is used for carrying out feature fusion based on first feature images with different scales to obtain a first fusion image; the multi-scale up-sampling layer is used for performing up-sampling operation and convolution operation of different scales on the basis of the first fusion image to obtain second characteristic images of different scales; the second fusion layer is used for carrying out feature fusion based on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image; the output layer is used for acquiring a cloud detection result corresponding to the target image based on the second fusion image; the multi-scale down-sampling layer comprises a plurality of cascaded levels of down-sampling sub-layers, the multi-scale up-sampling layer comprises a plurality of cascaded levels of up-sampling sub-layers, and the number of the down-sampling sub-layers is the same as that of the up-sampling sub-layers.

In addition, the logic instructions in the memory 430 may be implemented in the form of software functional units and stored in a computer readable storage medium when the software functional units are sold or used as independent products. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

In another aspect, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, a computer is capable of executing the remote sensing image-based cloud detection method provided by the above methods, the method including: preprocessing is carried out on the basis of the remote sensing image to obtain a target image; inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image; wherein, the cloud detection model includes: the multi-scale down-sampling layer is used for performing down-sampling operation and convolution operation of different scales based on the target image to obtain first characteristic images of different scales; the first fusion layer is used for carrying out feature fusion based on first feature images with different scales to obtain a first fusion image; the multi-scale up-sampling layer is used for performing up-sampling operation and convolution operation of different scales on the basis of the first fusion image to obtain second characteristic images of different scales; the second fusion layer is used for carrying out feature fusion based on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image; the output layer is used for acquiring a cloud detection result corresponding to the target image based on the second fusion image; the multi-scale down-sampling layer comprises a plurality of cascaded levels of down-sampling sub-layers, the multi-scale up-sampling layer comprises a plurality of cascaded levels of up-sampling sub-layers, and the number of the down-sampling sub-layers is the same as that of the up-sampling sub-layers.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program, which when executed by a processor, implements a method for remote sensing image-based cloud detection provided by the above methods, the method including: preprocessing is carried out on the basis of the remote sensing image to obtain a target image; inputting the target image into a cloud detection model, and outputting a cloud detection result corresponding to the target image; wherein, the cloud detection model includes: the multi-scale down-sampling layer is used for performing down-sampling operation and convolution operation of different scales based on the target image to obtain first characteristic images of different scales; the first fusion layer is used for carrying out feature fusion based on first feature images with different scales to obtain a first fusion image; the multi-scale up-sampling layer is used for performing up-sampling operation and convolution operation of different scales on the basis of the first fusion image to obtain second characteristic images of different scales; the second fusion layer is used for carrying out feature fusion based on the first feature image and the second feature image corresponding to the scales to obtain a second fusion image; the output layer is used for acquiring a cloud detection result corresponding to the target image based on the second fusion image; the multi-scale down-sampling layer comprises a plurality of cascaded levels of down-sampling sub-layers, the multi-scale up-sampling layer comprises a plurality of cascaded levels of up-sampling sub-layers, and the number of the down-sampling sub-layers is the same as that of the up-sampling sub-layers.

The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.

Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.

Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A cloud detection method based on remote sensing images is characterized by comprising the following steps:

wherein the cloud detection model comprises:

2. The remote sensing image-based cloud detection method according to claim 1, wherein the preprocessing based on the remote sensing image to obtain the target image comprises:

3. The remote sensing image-based cloud detection method according to claim 1, wherein the performing feature fusion based on the first feature images of different scales to obtain a first fused image comprises:

4. The remote sensing image-based cloud detection method according to claim 1, wherein the performing of upsampling and convolution operations of different scales based on the first fused image to obtain second feature images of different scales comprises:

5. The remote sensing image-based cloud detection method according to claim 1, wherein the performing feature fusion on the first feature image and the second feature image corresponding to the scale to obtain a second fused image comprises:

6. The remote sensing image-based cloud detection method according to claim 4, wherein the upsampling operation comprises an upsampling operation based on a bilinear interpolation method.

7. A cloud detection device based on remote sensing images is characterized by comprising:

wherein the cloud detection model comprises:

8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the steps of the method for remote sensing image based cloud detection according to any of claims 1 to 6 when executing the program.

9. A non-transitory computer-readable storage medium, on which a computer program is stored, wherein the computer program, when being executed by a processor, implements the steps of the remote sensing image-based cloud detection method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, carries out the steps of the remote sensing image based cloud detection method according to any of claims 1 to 6.