WO2024087574A1

WO2024087574A1 - Panoptic segmentation-based optical remote-sensing image raft mariculture area classification method

Info

Publication number: WO2024087574A1
Application number: PCT/CN2023/092747
Authority: WO
Inventors: 汪承义; 郭艳君; 陈建胜; 杜云艳; 王雷; 汪祖家
Original assignee: 中国科学院空天信息创新研究院
Priority date: 2022-10-27
Filing date: 2023-05-08
Publication date: 2024-05-02
Also published as: CN115908894A

Abstract

Provided in the present application is a panoptic segmentation-based optical remote-sensing image raft mariculture area classification method, comprising: acquiring an image to be segmented, said image being an optical remote-sensing image of a mariculture area (S101); and inputting said image into a pre-trained panoptic segmentation model to predict a multi-classification segmentation result (S102), in which semantic segmentation is performed on said image by using a semantic segmentation branch network so as to obtain an initial semantic segmentation result, instance segmentation is performed on said image by using an instance segmentation branch network so as to obtain an initial instance segmentation result, and the initial semantic segmentation result and the initial instance segmentation result are fused by using a panoptic fusion module so as to obtain the multi-classification segmentation result. By effectively using a variety of rich information in remote-sensing images, the present application can implement high-precision multi-classification recognition tasks of raft mariculture areas, and improve the segmentation precision of panoptic segmentation models.

Description

Classification method of marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Chinese patent application No. 202211328346.5, filed on October 27, 2022, and entitled “Classification method for marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images”, which is incorporated herein by reference in its entirety.

Technical Field

The present application relates to the field of marine remote sensing and image processing technology, and in particular to a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images.

Background technique

Marine raft aquaculture is an important part of marine aquaculture. Compared with ponds and tidal flats near the coast, raft aquaculture has a wide range and is dispersed in the region. The traditional on-site measurement method is not only time-consuming and labor-intensive, but also difficult to obtain accurate results for large areas. The development of remote sensing technology has greatly made up for the shortcomings of traditional ground measurement, such as small coverage and low data acquisition efficiency. At the same time, the use of deep learning methods to realize intelligent information extraction of remote sensing images can quickly and accurately obtain the distribution and aquaculture type information of marine aquaculture areas, which is a reliable and advanced technical means for dynamic monitoring of marine raft aquaculture. In order to develop and utilize marine resources and support the implementation of supervision, it is necessary to finely classify marine aquaculture areas without reducing the segmentation accuracy, such as subdividing them into fish (cages), algae (longlines), shellfish (rafts) and others.

Synthetic aperture radar (SAR) has the characteristics of all-day and all-weather, and has been widely used in the field of remote sensing. However, SAR images have disadvantages such as low resolution, susceptibility to noise interference, severe geometric distortion, and fewer available features. Optical remote sensing has a wide range of applications. Optical remote sensing clearly describes the boundary information of ground objects and contains rich spectral information, which is conducive to the extraction of raft aquaculture boundary and aquaculture type information. However, some optical images are interfered by clouds, fog and light. These interference factors restrict the extraction of feature information of optical remote sensing images and increase the difficulty of target recognition and segmentation in optical remote sensing images.

The existing convolutional neural network-based marine aquaculture area extraction is mainly divided into semantic segmentation and instance segmentation, for example, improved SOLO, D-ResUnet, HCHNet and other segmentation algorithms. Segmentation and instance segmentation are both pixel-level classification. In the process of semantic segmentation model training, the predicted value of each pixel is mapped to a probability value of [0,1] through the Softmax function, and then the error between the predicted value and the true label value is judged by the cross entropy loss function. The model is continuously trained by the gradient descent method to minimize the error between the two. The more types of semantic segmentation targets, that is, the more data set labels, the more interference items are encountered when identifying and segmenting each target. When reflected in the mathematical model, the more dispersed the probability distribution of the predicted value of each sample is, the greater the variance of the probability distribution is. At this time, the difficulty of focusing the sample prediction probability distribution on a label value is increased, which reduces the convergence speed of the loss function and reduces the accuracy of segmentation and recognition. From the above analysis, it can be seen that there is a mutual constraint between multi-class recognition and high-precision segmentation in the semantic segmentation task. Therefore, simple target detection, recognition and segmentation cannot complete the refined classification of marine raft aquaculture areas.

Most of the data labels of existing convolutional neural network models used for raft aquaculture area extraction are only suitable for single-task model training of semantic or instance segmentation, and lack data labels for multi-classification panoramic segmentation of marine aquaculture areas to distinguish different aquaculture types.

In summary, there is an urgent need for a panoramic segmentation method of marine aquaculture areas that can achieve multi-classification.

Summary of the invention

The present application provides a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation to solve the above problems.

The present application provides a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images, including:

Acquire an image to be segmented, where the image to be segmented is an optical remote sensing image of a marine aquaculture area;

Inputting the image to be segmented into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result, wherein the multi-classification segmentation result includes raft aquaculture area, non-raft aquaculture area and multiple aquaculture area categories;

The pre-trained panoramic segmentation model includes a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module;

Using the semantic segmentation branch network to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result includes an initial raft aquaculture area and an initial non-raft aquaculture area;

The instance segmentation branch network is used to perform instance segmentation on the image to be segmented to obtain An initial instance segmentation result, wherein the initial instance segmentation result includes a plurality of initial breeding area categories;

The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-classification segmentation result.

According to a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided by the present application, the semantic segmentation branch network is an improved U ² -Net network, and the improved U ² -Net network includes at least 6 U-shaped secondary encoders and 5 U-shaped secondary decoders, the 6 U-shaped secondary encoders are sequentially 4 first secondary encoders and 2 second secondary encoders, and the 5 U-shaped secondary decoders are sequentially 4 first secondary decoders and 1 second secondary decoder; the first secondary encoder and the first secondary decoder are both composed of a first convolution block, an LSFE module, a plurality of down-sampling modules, a DPC module, a second convolution block, a first convolution block, and a plurality of up-sampling modules in sequence;

The LSFE module is used to extract the features of the breeding area within a large field of view, and includes a separable convolution and an output filter;

The DPC module is used to capture long-range context information, which includes separable convolution and output channels.

According to a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the present application, the instance segmentation branch network includes an improved SOTR network, and the improved SOTR network includes at least a Transformer module; wherein the Transformer module includes a separable convolution and an iABN synchronization layer; and the Transformer module is used to predict the category of each instance.

According to a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided by the present application, the instance segmentation branch network also includes a feature extraction module, and the feature extraction module includes a mobile reverse bottleneck unit and a bidirectional feature pyramid network;

Using the feature extraction module to extract features from the image to be segmented to obtain multi-scale features;

Based on the multi-scale features, instance segmentation is performed on the image to be segmented through the improved SOTR network to obtain an initial instance segmentation result.

According to a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the present application, the pre-trained panoramic segmentation model is trained in the following manner:

Obtain the training data set and its corresponding labels, and build a panoramic segmentation model; The labels include semantic labels of raft aquaculture areas and non-raft aquaculture areas and instance labels of various aquaculture area categories;

Inputting the training data set into the semantic segmentation branch network, predicting a training semantic segmentation result, calculating the loss between the training semantic segmentation result and the semantic label, and obtaining a first loss;

Inputting the training data set into the instance segmentation branch network, predicting a training instance segmentation result, calculating the loss between the training instance segmentation result and the instance label, and obtaining a second loss;

Using the panoramic fusion module to adaptively fuse the training semantic segmentation result and the training instance segmentation result to obtain a training multi-classification result;

A total loss is obtained according to the first loss and the second loss, and the panoramic segmentation model is trained based on the training multi-classification result and the total loss until the panoramic segmentation model converges to obtain a trained panoramic segmentation model.

According to a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided by the present application, after obtaining the training data set and its corresponding labels, the method further includes:

Respectively constructing the normalized vegetation index feature and the normalized water index feature of the training data set;

fusing the normalized vegetation index feature and the normalized water index feature with the training data set to obtain a shared synthetic data set;

Accordingly, inputting the training data set into the semantic segmentation branch network includes:

Inputting the shared synthetic dataset into the semantic segmentation branch network;

The step of inputting the training data set into the instance segmentation branch network comprises:

The shared synthetic dataset is input into the instance segmentation branch network.

According to a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the present application, the training data set includes a label data set and an adversarial sample set. The label data set is a data set with corresponding labels after annotation, and the adversarial sample set is obtained by performing adversarial training on the segmentation results of the training instances.

According to a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in this application, the label data set is obtained in the following manner:

Obtain an optical remote sensing image of a marine aquaculture area for training, and perform at least The row storage format is unified, and the cloud removal, normalization and cropping processes are performed.

According to a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in the present application, the multiple aquaculture area categories include at least fish, algae and shellfish.

The present application also provides a device for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images, comprising:

An image acquisition module is used to acquire an image to be segmented, wherein the image to be segmented is an optical remote sensing image of a marine aquaculture area;

An image segmentation module is used to input the image to be segmented into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result, wherein the multi-classification segmentation result includes raft aquaculture area, non-raft aquaculture area and multiple aquaculture area categories;

Using the instance segmentation branch network to perform instance segmentation on the image to be segmented to obtain an initial instance segmentation result, wherein the initial instance segmentation result includes a plurality of initial breeding area categories;

The method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in this application uses a semantic segmentation branch network and an instance segmentation branch network to parallelly segment the segmented image, and fuses the outputs of the two branch networks through a parameter-free panoramic fusion module, and finally obtains a multi-classification segmentation result to achieve multi-task classification. Moreover, the adaptive fusion method of the panoramic fusion module can more completely utilize the logical outputs of the semantic segmentation head and the instance segmentation head to improve the accuracy of multi-classification tasks.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, Other drawings can be obtained based on these drawings without any creative work.

FIG1 is a schematic diagram of a flow chart of a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in an embodiment of the present application;

FIG2 is a second flow chart of a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of an existing U ² -Net network;

FIG4 is a structural comparison diagram of the En_1 substructure in the existing U ² -Net network and the improved En_1 substructure of the present application;

FIG5 is a schematic diagram of the training process of the panoptic segmentation model provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a device for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

Based on the existing convolutional neural network models mentioned above, they can only realize single tasks of semantic or instance segmentation, and in the face of complex and changeable marine environments, it is necessary to make more full and effective use of the rich information in optical remote sensing images. Therefore, this application unifies the semantic segmentation and instance segmentation prediction subnetworks, and fuses the outputs to form an overall panoramic segmentation network model; constructs a panoramic segmentation label dataset for the multi-classification task of marine raft aquaculture areas, thereby realizing multi-classification tasks, and making the classification of marine raft aquaculture areas more refined, and improving the segmentation accuracy of the model. The following is a specific description of the optical remote sensing image marine raft aquaculture area classification method based on panoramic segmentation proposed in this application in conjunction with the accompanying drawings.

Figure 1 is one of the flow charts of the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in an embodiment of the present application; Figure 2 is the second flow chart of the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images provided in an embodiment of the present application.

As shown in Figures 1 and 2, the optical remote sensing image based on panoramic segmentation of the marine raft aquaculture area Classification methods include:

S101, obtaining an image to be segmented, wherein the image to be segmented is an optical remote sensing image of a marine aquaculture area.

S102, inputting the image to be segmented into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result.

The multi-classification segmentation results include raft aquaculture areas, non-raft aquaculture areas and multiple aquaculture area categories, and the multiple aquaculture area categories include fish, algae, shellfish and others.

The pre-trained panoramic segmentation model (High precision panoptic segmentation, HPPS) includes a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module.

The image to be segmented is semantically segmented using the semantic segmentation branch network to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result includes an initial raft aquaculture area and an initial non-raft aquaculture area.

The image to be segmented is subjected to instance segmentation using the instance segmentation branch network to obtain an initial instance segmentation result, wherein the initial instance segmentation result includes a plurality of initial breeding area categories.

Among them, the panoramic fusion module is a parameter-free panoramic fusion module, which selectively attenuates or amplifies the fused logical output score based on the pixel-based head prediction adaptability to adaptively fuse the initial semantic segmentation results and the initial instance segmentation results. Through this end-to-end approach, the entire panoramic segmentation network is jointly optimized to obtain the final multi-classification and high-precision panoramic segmentation output results of the marine raft aquaculture area, realizing the multi-classification task of the optical remote sensing image marine raft aquaculture area.

In addition, before the image to be segmented is input into the pre-trained panoramic segmentation model, the image to be segmented is standardized and pre-processed, and then slidingly cropped into an image of 2048*2048 size, and the cropped images are sequentially input into the trained HPPS model (i.e., the panoramic segmentation model). The output result of the HPPS model is the multi-classification result of the offshore aquaculture area, and all the images are spliced to obtain the overall panoramic segmentation result map of the image to be segmented.

The optical remote sensing image marine raft aquaculture area classification method based on panoramic segmentation provided in the embodiment of the present application uses a semantic segmentation branch network and an instance segmentation branch network to parallelly segment the image to be segmented, and fuses the outputs of the two branch networks through a parameter-free panoramic fusion module to finally obtain a multi-classification segmentation result and realize multi-task classification. This method can more completely utilize the logical outputs of the semantic segmentation head and the instance segmentation head to improve the accuracy of multi-classification tasks.

Furthermore, the semantic segmentation branch network is an improved U ² -Net network, the semantic segmentation branch network is an improved U ² -Net network, the improved U ² -Net network includes at least 6 U-shaped secondary encoders and 5 U-shaped secondary decoders, the 6 U-shaped secondary encoders are sequentially 4 first secondary encoders and 2 second secondary encoders, the 5 U-shaped secondary decoders are sequentially 4 first secondary decoders and 1 second secondary decoder; the first secondary encoder and the first secondary decoder are both composed of a first convolution block, an LSFE module, multiple down-sampling modules, a DPC module, a second convolution block, a first convolution block and multiple up-sampling modules in sequence.

The LSFE module is used to extract the features of the breeding area within a large field of view, and includes separable convolution and output filters, specifically including two 3×3 separable convolutions and 128 output filters.

The DPC module is used to capture remote context information, which includes separable convolution and output channels. Specifically, it includes a 3×3 separable convolution and 256 output channels, and is extended to five parallel branches. Then the outputs of all parallel branches are connected to generate a tensor with 1280 channels, which is finally input to a 1×1 convolution with 256 output channels. The output of the 1×1 convolution is the output of the DPC module.

In the prior art, the U ² -Net network is a saliency detection model, and its specific network structure diagram is shown in FIG3 . It is a two-level nested U-shaped structure. In this application, the overall U-shaped structure is referred to as the primary structure, and each small U-shaped structure contained in the primary structure is referred to as a secondary structure. This application does not make improvements on the primary structure, but makes specific improvements on the secondary structure.

The specific concept of the improvement is: since remote sensing images are observations of a large area of the ocean coast from a satellite platform, with a wide viewing angle and huge data, they reflect the distribution of marine aquaculture areas in a macro and comprehensive manner. Therefore, the improved U ² -Net network model proposed in this application not only realizes the macro detection of marine aquaculture areas, but also needs to identify, extract and classify each small piece of aquaculture raft. In order to achieve this goal, this application improves the secondary U-shaped structure, specifically using a large scale feature extractor (LSFE) module and a dense prediction cell (DPC) module in the secondary structure.

The following describes the improved U ² -Net network in detail based on the overall network structure.

First, the improved U ² -Net network provided in the present application is also shown in FIG3 in its structure at this level, which specifically includes 6 U-shaped secondary encoders (i.e., En_1 to En_6) and 5 U-shaped secondary decoders (i.e., De_1 to De_5), wherein, in structure, En_1 corresponds to De_1, En_2 corresponds to De_2, En_3 corresponds to De_3, En_4 corresponds to De_4, and En_5 corresponds to De_5.

The present application improves the first four substructures of the U ² -Net network, namely, the structures of En_1, De_1, En_2, De_2, En_3, De_3, En_4, and De_4, while En_5, En_6, and De_5 are not improved and still use the structures in the prior art, which will not be described in detail in the present application.

Taking En_1 as an example, the existing network structure of En_1 consists of 2 first convolution blocks (i.e. ①Conv+BN+RELU in Figure 4), 5 downsampling modules (i.e. ③Downsample×1/2Conv+BN+RELU in Figure 4), 1 second convolution block (i.e. ⑤Conv+BN+RELU dilation＝4 in Figure 4), 1 first convolution block and 5 upsampling modules (i.e. ⑧Upsample×2Conv+BN+RELU in Figure 4).

The improved En_1 network structure is composed of 1 first convolution block, 1 LSFE module (i.e., ② in Figure 4), 4 downsampling modules, 1 DPC module (i.e., ④ in Figure 4), 1 second convolution block, 1 first convolution block, and 5 upsampling modules. That is, the second first convolution block in the prior art is replaced by an LSFE module, the last downsampling module is replaced by a DPC module, and the rest are the same as before. Since the structural difference between En_2, En_3, and En_4 in the prior art and the aforementioned En_1 is the decrease in the number of downsampling modules and the number of upsampling modules. Therefore, in the same way, the improved En_2 network structure is composed of 1 first convolution block, 1 LSFE module, 3 downsampling modules, 1 DPC module, 1 second convolution block, 1 first convolution block, and 4 upsampling modules. The improved En_3 network structure is composed of 1 first convolution block, 1 LSFE module, 2 downsampling modules, 1 DPC module, 1 second convolution block, 1 first convolution block and 3 upsampling modules. The improved En_4 network structure is composed of 1 first convolution block, 1 LSFE module, 1 downsampling module, 1 DPC module, 1 second convolution block, 1 first convolution block and 2 upsampling modules.

In addition, the improved De_1, De_2, De_3, and De_4 correspond one-to-one to the aforementioned improved En_1, En_2, En_3, and En_4, and will not be described in detail here.

This application adopts binary classification semantic segmentation, and its corresponding data set labels only include label 0 (non-raft aquaculture area) and label 1 (raft aquaculture area). During the training process, the loss function involved in the semantic segmentation branch network adopts the binary cross entropy loss function.

The embodiment of the present application provides a method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images. The LSFE module is used to extract the features of aquaculture areas within a large field of view to achieve macroscopic detection of marine aquaculture areas. The DPC module is used to capture remote context information to identify, extract and classify each small aquaculture raft, thereby improving the segmentation accuracy of the semantic segmentation branch network.

Furthermore, the instance segmentation branch network includes an improved SOTR network, and the improved SOTR network includes at least a Transformer module; wherein the Transformer module includes a separable convolution and an iABN synchronization layer; and the Transformer module is used to predict each instance category.

In the prior art, SOTR uses Transformer to simplify the segmentation process, using two parallel subtasks: 1) predicting each instance category through Transformer; 2) dynamically generating segmentation masks using a multi-level upsampling module. Among them, the encoder-decoder Transformer model unifies the instance segmentation tasks through a series of learnable mask embeddings. This application extends the Transformer with separable convolution and iABN (inplace activated batch normalization) synchronization layers, which improves the segmentation accuracy and training convergence to a certain extent.

The method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the embodiment of the present application improves the segmentation accuracy and training convergence of the model through the expansion of separable convolution and iABN synchronization layers.

Furthermore, the instance segmentation branch network also includes a feature extraction module, which includes a moving reverse bottleneck unit and a bidirectional feature pyramid network (Feature pyramid network, FPN).

The feature extraction module is used to extract features from the image to be segmented to obtain multi-scale features.

The method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the embodiment of the present application realizes the extraction of multi-scale features by moving the reverse bottleneck unit and the bidirectional feature pyramid network, and can obtain more large-scale features, small-scale features, shallow information, and deep information to improve the accuracy of instance segmentation.

FIG5 is a schematic diagram of the training process of the panoramic segmentation model provided in an embodiment of the present application; as shown in FIG5 , the pre-trained panoramic segmentation model is trained in the following manner:

Get the training dataset and its corresponding labels and build a panoptic segmentation model.

The labels include semantic labels of raft aquaculture areas and non-raft aquaculture areas and instance labels of various aquaculture area categories.

It should be noted that after obtaining the training data set, the training data set needs to be annotated. The specific annotations are divided into "stuff" and "thing" classes. Among them, the "stuff" class is annotated with a semantic mask, 0 represents a non-raft aquaculture area, and 1 represents a raft aquaculture area; the "thing" class is annotated with an instance mask, including four instances of fish, algae, shellfish, and others. According to the above annotation categories, semantic labels of the background (i.e., non-raft aquaculture area) and foreground (i.e., raft aquaculture area) are created for the training set and the test set, and on this basis, instance labels of instance 1 (fish), instance 2 (algae), instance 3 (shellfish), and instance 4 (others) are created.

The training data set is input into the semantic segmentation branch network, a training semantic segmentation result is predicted, and a loss between the training semantic segmentation result and the semantic label is calculated to obtain a first loss.

The training data set is input into the instance segmentation branch network, the training instance segmentation result is predicted, and the loss between the training instance segmentation result and the instance label is calculated to obtain a second loss.

The panoramic fusion module is used to adaptively fuse the training semantic segmentation result and the training instance segmentation result to obtain a training multi-classification result.

The total loss is obtained by adaptively weighting the first loss and the second loss according to the logical output score of the attenuated or amplified fusion.

It should be noted that this application trains the panoramic segmentation model by sharing the synthetic data set, and after the panoramic segmentation model reaches convergence accuracy, the panoramic segmentation model is tested by the test set to obtain qualitative evaluation results.

If the test does not meet the accuracy requirements, the model is trained again by adjusting hyperparameters, supplementing training samples, etc. until it meets the qualitative evaluation requirements. The model is then tested with the test set and the panoramic segmentation accuracy PQ of the model is evaluated to finally obtain a trained panoramic segmentation model.

The embodiment of the present application provides a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation. The dataset labels mentioned above not only classify and label the raft aquaculture areas and background areas as a whole, but also further classify and label the various aquaculture categories in the raft aquaculture areas in a refined manner to achieve multi-classification tasks.

Furthermore, after obtaining the training data set and its corresponding labels, the method further includes:

The normalized vegetation index feature (Normalized Difference Vegetation Index, NDVI) and the normalized water index feature (Normalized Difference Water Index, NDWI) of the training data set are constructed respectively.

The normalized vegetation index feature and the normalized water body index feature are fused with the training data set to obtain a shared synthetic data set.

The shared synthetic dataset is input into the semantic segmentation branch network.

It should be noted that the normalized vegetation index feature and the normalized water index feature can also be fused with the image to be segmented after the image to be segmented is obtained, so as to segment the fused image using the panoramic segmentation model.

The method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation provided in the embodiment of the present application obtains a shared synthetic data set by fusing two custom features, NDVI and NDWI, into a training data set, thereby being able to more fully and effectively utilize the rich variety of information in optical remote sensing images to improve the segmentation accuracy of the panoramic segmentation model.

Furthermore, the training data set includes a labeled data set and an adversarial sample set, the labeled data set is a data set with corresponding labels after being annotated, and the adversarial sample set is obtained by performing adversarial training on the training instance segmentation results.

Specifically, the multi-classification data label set of the marine aquaculture area is used to perform adversarial training on the instance segmentation branch to improve the anti-interference ability of multi-target multi-classification. The adversarial samples generated during the adversarial training are added to the training dataset and together with the label dataset constitute the training dataset.

It should be noted that adversarial training can be achieved by adding a discriminator or generating new samples based on gradient feedback. The adversarial training method is a conventional method and this application does not limit it.

The embodiment of the present application provides a method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation. An adversarial sample set is obtained through adversarial training, and a panoramic segmentation model is trained based on the adversarial sample set and a label data set, thereby improving the anti-interference ability of the panoramic segmentation model.

Furthermore, the label data set is obtained by:

An optical remote sensing image of a training marine aquaculture area is obtained, and the optical remote sensing image is subjected to at least storage format unification, cloud removal, normalization, and cropping.

Specifically, we first obtain medium-resolution optical remote sensing images from Sentinel-2, GF-1 (PMS/WFV) and Landsat covering 30km of my country's coastline. Then, we unify the storage format, remove the cloud and fog, and normalize the optical remote sensing images, and then slide and crop them into standard images of 2048*2048 size, thus forming a standard label dataset.

The following is a description of the marine raft aquaculture area classification device based on panoramic segmentation of optical remote sensing images provided in the present application. The marine raft aquaculture area classification device based on panoramic segmentation of optical remote sensing images described below and the marine raft aquaculture area classification method based on panoramic segmentation of optical remote sensing images described above can be referenced to each other.

Figure 6 is a structural schematic diagram of the optical remote sensing image marine raft aquaculture area classification device based on panoramic segmentation provided in an embodiment of the present application; as shown in Figure 6, the optical remote sensing image marine raft aquaculture area classification device based on panoramic segmentation includes an image acquisition module 601 and an image segmentation module 602.

The image acquisition module 601 is used to acquire the image to be segmented, where the image to be segmented is an optical remote sensing image of a marine aquaculture area.

The image segmentation module 602 is used to input the image to be segmented into a pre-trained panoramic image. In the segmentation model, multi-classification segmentation results are predicted.

The optical remote sensing image marine raft aquaculture area classification device based on panoramic segmentation provided in the embodiment of the present application uses a semantic segmentation branch network and an instance segmentation branch network to parallelly segment the segmented image, and fuses the outputs of the two branch networks through a parameter-free panoramic fusion module, and finally obtains a multi-classification segmentation result to achieve multi-task classification. Moreover, the adaptive fusion method of the panoramic fusion module can more completely utilize the logical outputs of the semantic segmentation head and the instance segmentation head to improve the accuracy of multi-classification tasks.

The above-described device embodiments are merely illustrative, and the components described as separate components are The units may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

A method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images, comprising:

Acquire an image to be segmented, where the image to be segmented is an optical remote sensing image of a marine aquaculture area;

Inputting the image to be segmented into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result, wherein the multi-classification segmentation result includes a raft aquaculture area, a non-raft aquaculture area, and multiple aquaculture area categories;

The pre-trained panoramic segmentation model includes a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module;

Using the semantic segmentation branch network to perform semantic segmentation on the image to be segmented to obtain an initial semantic segmentation result, wherein the initial semantic segmentation result includes an initial raft aquaculture area and an initial non-raft aquaculture area;

Using the instance segmentation branch network to perform instance segmentation on the image to be segmented to obtain an initial instance segmentation result, wherein the initial instance segmentation result includes a plurality of initial breeding area categories;

The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-classification segmentation result.
According to the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images in claim 1, wherein the semantic segmentation branch network is an improved U2 -Net network, and the improved U2 -Net network includes at least 6 U-shaped secondary encoders and 5 U-shaped secondary decoders, the 6 U-shaped secondary encoders are sequentially 4 first secondary encoders and 2 second secondary encoders, and the 5 U-shaped secondary decoders are sequentially 4 first secondary decoders and 1 second secondary decoder; the first secondary encoder and the first secondary decoder are both composed of a first convolution block, an LSFE module, a plurality of down-sampling modules, a DPC module, a second convolution block, a first convolution block, and a plurality of up-sampling modules in sequence;

The LSFE module is used to extract the features of the breeding area within a large field of view, and includes a separable convolution and an output filter;

The DPC module is used to capture long-range context information, which includes separable convolution and output channels.
The optical remote sensing image marine raft culture method based on panoramic segmentation according to claim 1 A method for classifying regions, wherein the instance segmentation branch network includes an improved SOTR network, and the improved SOTR network includes at least a Transformer module; wherein the Transformer module includes a separable convolution and an iABN synchronization layer; and the Transformer module is used to predict the category of each instance.
According to the method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation according to claim 3, wherein the instance segmentation branch network further includes a feature extraction module, and the feature extraction module includes a moving reverse bottleneck unit and a bidirectional feature pyramid network;

Using the feature extraction module to extract features from the image to be segmented to obtain multi-scale features;

Based on the multi-scale features, instance segmentation is performed on the image to be segmented through the improved SOTR network to obtain an initial instance segmentation result.
According to any one of claims 1 to 4, the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images, wherein the pre-trained panoramic segmentation model is trained in the following manner:

Obtain a training data set and its corresponding labels, and construct a panoramic segmentation model; wherein the labels include semantic labels of raft aquaculture areas and non-raft aquaculture areas and instance labels of multiple aquaculture area categories;

Inputting the training data set into the semantic segmentation branch network, predicting a training semantic segmentation result, calculating the loss between the training semantic segmentation result and the semantic label, and obtaining a first loss;

Inputting the training data set into the instance segmentation branch network, predicting a training instance segmentation result, calculating the loss between the training instance segmentation result and the instance label, and obtaining a second loss;

Using the panoramic fusion module, the training semantic segmentation result and the training instance segmentation result are adaptively fused to obtain a training multi-classification result;

A total loss is obtained according to the first loss and the second loss, and the panoramic segmentation model is trained based on the training multi-classification result and the total loss until the panoramic segmentation model converges to obtain a trained panoramic segmentation model.
According to the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images according to claim 5, after obtaining the training data set and its corresponding labels, the The method also includes:

Respectively constructing the normalized vegetation index feature and the normalized water index feature of the training data set;

fusing the normalized vegetation index feature and the normalized water index feature with the training data set to obtain a shared synthetic data set;

Accordingly, inputting the training data set into the semantic segmentation branch network includes:

Inputting the shared synthetic dataset into the semantic segmentation branch network;

The step of inputting the training data set into the instance segmentation branch network comprises:

The shared synthetic dataset is input into the instance segmentation branch network.
According to the method for classifying marine raft aquaculture areas in optical remote sensing images based on panoramic segmentation in claim 5, the training data set includes a labeled data set and an adversarial sample set, the labeled data set is a data set with corresponding labels after annotation, and the adversarial sample set is obtained by adversarial training on the segmentation results of the training instances.
According to the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images according to claim 7, wherein the label data set is obtained by:

An optical remote sensing image of a training marine aquaculture area is obtained, and the optical remote sensing image is subjected to at least storage format unification, cloud removal, normalization, and cropping.
According to the method for classifying marine raft aquaculture areas based on panoramic segmentation of optical remote sensing images in claim 7, the multiple aquaculture area categories include at least fish, algae and shellfish.
An optical remote sensing image marine raft aquaculture area classification device based on panoramic segmentation, comprising:

An image acquisition module is used to acquire an image to be segmented, wherein the image to be segmented is an optical remote sensing image of a marine aquaculture area;

An image segmentation module is used to input the image to be segmented into a pre-trained panoramic segmentation model to predict a multi-classification segmentation result, wherein the multi-classification segmentation result includes raft aquaculture areas, non-raft aquaculture areas, and multiple aquaculture area categories;

The pre-trained panoramic segmentation model includes a semantic segmentation branch network, an instance segmentation branch network and a panoramic fusion module;

The semantic segmentation branch network is used to perform semantic segmentation on the image to be segmented to obtain An initial semantic segmentation result, wherein the initial semantic segmentation result includes an initial raft aquaculture area and an initial non-raft aquaculture area;

Using the instance segmentation branch network to perform instance segmentation on the image to be segmented to obtain an initial instance segmentation result, wherein the initial instance segmentation result includes a plurality of initial breeding area categories;

The panoramic fusion module is used to fuse the initial semantic segmentation result and the initial instance segmentation result to obtain a multi-classification segmentation result.