CN111145178A - High-resolution remote sensing image multi-scale segmentation method - Google Patents
High-resolution remote sensing image multi-scale segmentation method Download PDFInfo
- Publication number
- CN111145178A CN111145178A CN201811310536.8A CN201811310536A CN111145178A CN 111145178 A CN111145178 A CN 111145178A CN 201811310536 A CN201811310536 A CN 201811310536A CN 111145178 A CN111145178 A CN 111145178A
- Authority
- CN
- China
- Prior art keywords
- remote sensing
- sensing image
- image
- picture
- probability
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformation in the plane of the image
- G06T3/40—Scaling the whole image or part thereof
- G06T3/4038—Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Abstract
The invention belongs to the field of computer vision and remote sensing image processing, and particularly relates to a high-resolution remote sensing image multi-scale segmentation method. The method comprises the following steps: preprocessing a remote sensing image; training a multi-scale segmentation model; and (5) predicting the large-scale remote sensing image. The method can effectively relieve the negative influence caused by the edge effect and the inter-class competition problem, and improve the segmentation precision of the remote sensing image.
Description
Technical Field
The invention belongs to the fields of deep learning, computer vision and remote sensing image processing, and particularly relates to a high-resolution remote sensing image multi-scale segmentation method.
Background
Remote sensing image segmentation is an important component of digital image analysis and is widely applied to various fields such as homeland monitoring, geographical mapping, urban planning, disaster prevention and reduction and the like for a long time. With the development of deep learning technology, segmentation technologies based on remote sensing images, such as earth surface coverage classification, are also developed to a certain extent.
The segmentation is greatly challenged by some special properties of the remote sensing image, such as overlarge image scale, serious imbalance of categories and the like. When a large-scale high-resolution remote sensing image is segmented, an original image is generally firstly segmented into an image with a moderate scale, and then a prediction result of the segmented image is spliced back to the original size, so that edge effects of unsmooth edges and low accuracy at the spliced part are often brought. In addition, the class competition problem caused by the imbalance of the remote sensing image classes can lead to the serious inhibition of the classes with small area occupation ratio, and the segmentation precision is reduced. The high-resolution remote sensing image multi-scale segmentation method can effectively relieve the problems of edge effect and inter-class competition and improve the segmentation precision of the remote sensing image.
Disclosure of Invention
Aiming at the problems or the defects, and reducing the negative effects caused by the edge effect and the inter-class competition, the invention provides a high-resolution remote sensing image multi-scale segmentation method.
The technical scheme adopted by the invention is as follows:
(1) and cutting, normalizing and data enhancing the high-resolution remote sensing image.
(2) And (3) constructing a neural network, and training a multi-scale segmentation model on the remote sensing image processed in the step (1).
(3) And cutting the high-resolution remote sensing image to be tested, and performing multi-scale prediction by using the segmentation model to generate a segmentation result.
The cutting, normalization and data enhancement processing in the step (1) specifically comprise:
(11) randomly cutting out a picture with a moderate size from the high-resolution remote sensing image, and enhancing data such as up-down turning, left-right turning, right-angle rotation, random contrast, random saturation and the like on the cut picture to expand the diversity of a training sample; and carrying out the same processing on the label picture, and keeping the newly generated label picture and the training sample synchronous.
(12) And (3) normalizing the picture generated in the step (11), namely subtracting the mean value of the corresponding channel from each channel of the picture, and dividing the mean value by the standard deviation of the corresponding channel.
The multi-scale segmentation model training process in the step (2) specifically comprises the following steps:
(21) the segmentation network adopts an encoder-decoder structure, and ResNet101 with a full connection layer removed is used as an encoder. The decoder part is structured in such a way that the output of the fifth block of the ResNet101 is subjected to double upsampling and added with the output of the fourth block, the sum of the output of the fifth block is subjected to double upsampling and added with the output of the third block, the sum of the output of the third block and the output of the second block is subjected to double upsampling and added, and finally, the output of the second block is subjected to upsampling to the network input size. And (3) the original size pictures, the 0.75 time size pictures and the 1.25 time size pictures of the training samples are output after passing through the network and are scaled back to the original size, and then the outputs of the three sizes are spliced together. Assuming that n types are shared in the labeled picture, n binary branches are obtained from the output of the network through n convolution layers, and the ith (i is 0,1,2, …, i < n) branch represents the probability that the current pixel is the ith type.
(22) Training is carried out by using a random gradient descent method, and a composite loss function consisting of cross-entropy and jaccard approximation coefficients is adopted, and the calculation formula is as follows:
cross_entropy＝-∑(ytruelog ypred+(1-ytrue)log(1-ypred))
loss＝cross_entropy-log(jaccard_approximation)
the multi-scale prediction process in the step (3) specifically includes:
(31) cutting the high-resolution remote sensing image to be tested into three groups according to different sizes, wherein the height of the remote sensing image is h, the width of the remote sensing image is w, and the height of the jth (j is 0,1,2) group size is hjWidth of wjFirstly, the remote sensing image is reflected and filled, and the filled remote sensing image is high
Width is
(32) And (3) taking one image A from the remote sensing images filled in the jth group (j is 0), normalizing the image A as described in (12), inputting the image A into a trained segmentation model to obtain n probability maps as described in (21), and splicing the n probability maps into A'. And turning the image A up and down to obtain an image B, turning the image A left and right to obtain an image C, and performing the same operation on the images B and C to respectively obtain probability maps B 'and C'. And (4) solving the average value of the probability graph and A ' obtained by respectively turning B ' and C ' up and down and left and right to obtain the final prediction probability graph.
(33) Traversing the residual j (j ═ 0) th group of images, each performing the operation as described in (32), and splicing the final prediction probability map into high-level
Width is
And cutting the probability map into a probability map with the height h and the width w.
(34) The two groups of remote sensing images with j being 1 and j being 2 are respectively subjected to the operations (32) and (33), all three obtained probability maps are averaged, and the category to which the maximum probability belongs is the category of the pixel.
The invention has the beneficial effects that:
the invention provides a high-resolution remote sensing image multi-scale segmentation method, which is characterized in that a large-scale remote sensing image is cut into a plurality of groups of images with different sizes, a prediction result of the cut image is spliced back to the original size, and then the plurality of groups of results are fused together, so that the edge effect can be effectively relieved. In addition, the multi-classification problem of the pixel points is decomposed into a plurality of two-classification problems, so that the problem of inter-class competition caused by unbalanced classes is effectively solved, and the segmentation precision of the remote sensing image is improved.
Drawings
FIG. 1 is a high resolution remote sensing image to be predicted
FIG. 2 is a segmentation result of a high resolution remote sensing image
Detailed Description
The present invention will be described in detail below with reference to the accompanying drawings.
The invention discloses a high-resolution remote sensing image multi-scale segmentation method, which comprises the following specific implementation steps:
(1) and cutting, normalizing and data enhancing the high-resolution remote sensing image.
(2) And (3) constructing a neural network, and training a multi-scale segmentation model on the remote sensing image processed in the step (1).
(3) And cutting the high-resolution remote sensing image to be tested, and performing multi-scale prediction by using the segmentation model to generate a segmentation result.
The cutting, normalization and data enhancement processing in the step (1) specifically comprise:
(11) randomly cutting out a picture with a moderate size from the high-resolution remote sensing image, and enhancing data such as up-down turning, left-right turning, right-angle rotation, random contrast, random saturation and the like on the cut picture to expand the diversity of a training sample; and carrying out the same processing on the label picture, and keeping the newly generated label picture and the training sample synchronous.
(12) And (3) normalizing the picture generated in the step (11), namely subtracting the mean value of the corresponding channel from each channel of the picture, and dividing the mean value by the standard deviation of the corresponding channel.
The multi-scale segmentation model training process in the step (2) specifically comprises the following steps:
(21) the segmentation network adopts an encoder-decoder structure, and ResNet101 with a full connection layer removed is used as an encoder. The decoder part is structured in such a way that the output of the fifth block of the ResNet101 is subjected to double upsampling and added with the output of the fourth block, the sum of the output of the fifth block is subjected to double upsampling and added with the output of the third block, the sum of the output of the third block and the output of the second block is subjected to double upsampling and added, and finally, the output of the second block is subjected to upsampling to the network input size. And (3) the original size pictures, the 0.75 time size pictures and the 1.25 time size pictures of the training samples are output after passing through the network and are scaled back to the original size, and then the outputs of the three sizes are spliced together. Assuming that n types are shared in the labeled picture, n binary branches are obtained from the output of the network through n convolution layers, and the ith (i is 0,1,2, …, i < n) branch represents the probability that the current pixel is the ith type.
(22) Training is carried out by using a random gradient descent method, and a composite loss function consisting of cross-entropy and jaccard approximation coefficients is adopted, and the calculation formula is as follows:
cross_entropy＝-∑(ytruelog ypred+(1-ytrue)log(1-ypred))
loss＝cross_entropy-log(jaccard_approximation)
the multi-scale prediction process in the step (3) specifically includes:
(31) cutting the high-resolution remote sensing image to be tested into three groups according to different sizes, wherein the height of the remote sensing image is h, the width of the remote sensing image is w, and the height of the jth (j is 0,1,2) group size is hjWidth of wjFirstly, the remote sensing image is reflected and filled, and the filled remote sensing image is high
Width is
(32) And (3) taking one image A from the remote sensing images filled in the jth group (j is 0), normalizing the image A as described in (12), inputting the image A into a trained segmentation model to obtain n probability maps as described in (21), and splicing the n probability maps into A'. And turning the image A up and down to obtain an image B, turning the image A left and right to obtain an image C, and performing the same operation on the images B and C to respectively obtain probability maps B 'and C'. And (4) solving the average value of the probability graph and A ' obtained by respectively turning B ' and C ' up and down and left and right to obtain the final prediction probability graph.
(33) Traversing the residual j (j ═ 0) th group of images, each performing the operation as described in (32), and splicing the final prediction probability map into high-level
Width is
And cutting the probability map into a probability map with the height h and the width w.
(34) The two groups of remote sensing images with j being 1 and j being 2 are respectively subjected to the operations (32) and (33), all three obtained probability maps are averaged, and the category to which the maximum probability belongs is the category of the pixel.
The high-resolution remote sensing image to be predicted is shown in figure 1, and the segmentation result of the high-resolution remote sensing image is shown in figure 2. Experimental results show that the method can effectively relieve the problems of edge effect and inter-class competition, and improve the segmentation precision of the remote sensing image.
Claims (4)
1. A high-resolution remote sensing image multi-scale segmentation method is characterized by comprising the following steps:
(1) cutting, normalizing and data enhancing the high-resolution remote sensing image;
(2) constructing a neural network, and training a multi-scale segmentation model on the remote sensing image processed in the step (1);
(3) and cutting the high-resolution remote sensing image to be tested, and performing multi-scale prediction by using the segmentation model to generate a segmentation result.
2. The method according to claim 1, wherein the step (1) specifically comprises:
(11) randomly cutting out a picture with a moderate size from the high-resolution remote sensing image, and enhancing data such as up-down turning, left-right turning, right-angle rotation, random contrast, random saturation and the like on the cut picture to expand the diversity of a training sample; carrying out the same processing on the label picture to keep the newly generated label picture and the training sample synchronous;
(12) and (3) normalizing the picture generated in the step (11), namely subtracting the mean value of the corresponding channel from each channel of the picture, and dividing the mean value by the standard deviation of the corresponding channel.
3. The method according to claim 1, wherein the step (2) specifically comprises:
(21) the segmentation network adopts an encoder-decoder structure, ResNet101 with a fully connected layer removed is used as an encoder, the decoder part structure is that the output of the fifth block of ResNet101 is added with the output of the fourth block through double upsampling, the sum of the two upsampling and the output of the third block is added with the output of the second block through double upsampling, finally the input size of the network is upsampled, the output of the original size picture of a training sample, the 0.75-time size picture and the 1.25-time size picture after passing through the network is shrunk back to the original size, the outputs of the three sizes are spliced together, n types are set in the label picture, the outputs of the network respectively obtain n binary branches through n convolutional layers, the i (i is 0,1,2, …, i < n) th branch represents the probability that the current pixel is the ith type,
(22) training is carried out by using a random gradient descent method, and a composite loss function consisting of cross-entropy and jaccard approximation coefficients is adopted, and the calculation formula is as follows:
cross_entropy＝-∑(ytruelogypred+(1-ytrue)log(1-ypred))
loss＝cross_entropy-log(jaccard_approximation)。
4. the method according to claim 1, wherein the step (3) specifically comprises:
(31) cutting the high-resolution remote sensing image to be tested into three groups according to different sizes, wherein the height of the remote sensing image is h, the width of the remote sensing image is w, and the height of the jth (j is 0,1,2) group size is hjWidth of wjFirstly, the remote sensing image is reflected and filled, and the filled remote sensing image is high
Width is
(32) Taking one image A out of the remote sensing images filled in the jth (j is 0), normalizing the image A as described in (12), inputting the image A into a trained segmentation model to obtain n probability maps as described in (21), splicing the n probability maps into A ', turning the image A up and down to obtain an image B, turning the image A left and right to obtain an image C, and performing the same operation on the images B and C to respectively obtain probability maps B ' and C '. Calculating the mean value of the probability graph and A ' obtained by respectively turning B ' and C ' up and down and left and right to obtain a final prediction probability graph;
(33) traversing the residual j (j ═ 0) th group of images, each performing the operation as described in (32), and splicing the final prediction probability map into high-level
Width is
Cutting the probability graph into a probability graph with the height h and the width w;
(34) the two groups of remote sensing images with j being 1 and j being 2 are respectively subjected to the operations (32) and (33), all three obtained probability maps are averaged, and the category to which the maximum probability belongs is the category of the pixel.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811310536.8A CN111145178A (en) | 2018-11-06 | 2018-11-06 | High-resolution remote sensing image multi-scale segmentation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811310536.8A CN111145178A (en) | 2018-11-06 | 2018-11-06 | High-resolution remote sensing image multi-scale segmentation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111145178A true CN111145178A (en) | 2020-05-12 |
Family
ID=70515974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811310536.8A Pending CN111145178A (en) | 2018-11-06 | 2018-11-06 | High-resolution remote sensing image multi-scale segmentation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111145178A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709387A (en) * | 2020-06-22 | 2020-09-25 | 中国科学院空天信息创新研究院 | Building segmentation method and system for high-resolution remote sensing image |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170061249A1 (en) * | 2015-08-26 | 2017-03-02 | Digitalglobe, Inc. | Broad area geospatial object detection using autogenerated deep learning models |
| CN107610141A (en) * | 2017-09-05 | 2018-01-19 | 华南理工大学 | A kind of remote sensing images semantic segmentation method based on deep learning |
| CN107767380A (en) * | 2017-12-06 | 2018-03-06 | 电子科技大学 | A kind of compound visual field skin lens image dividing method of high-resolution based on global empty convolution |
| CN107862695A (en) * | 2017-12-06 | 2018-03-30 | 电子科技大学 | A kind of modified image segmentation training method based on full convolutional neural networks |
| CN107958271A (en) * | 2017-12-06 | 2018-04-24 | 电子科技大学 | The cutaneous lesions deep learning identifying system of Analysis On Multi-scale Features based on expansion convolution |
| CN108230329A (en) * | 2017-12-18 | 2018-06-29 | 孙颖 | Semantic segmentation method based on multiple dimensioned convolutional neural networks |
| CN108681692A (en) * | 2018-04-10 | 2018-10-19 | 华南理工大学 | Increase Building recognition method in a kind of remote sensing images based on deep learning newly |
-
2018
- 2018-11-06 CN CN201811310536.8A patent/CN111145178A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170061249A1 (en) * | 2015-08-26 | 2017-03-02 | Digitalglobe, Inc. | Broad area geospatial object detection using autogenerated deep learning models |
| CN107610141A (en) * | 2017-09-05 | 2018-01-19 | 华南理工大学 | A kind of remote sensing images semantic segmentation method based on deep learning |
| CN107767380A (en) * | 2017-12-06 | 2018-03-06 | 电子科技大学 | A kind of compound visual field skin lens image dividing method of high-resolution based on global empty convolution |
| CN107862695A (en) * | 2017-12-06 | 2018-03-30 | 电子科技大学 | A kind of modified image segmentation training method based on full convolutional neural networks |
| CN107958271A (en) * | 2017-12-06 | 2018-04-24 | 电子科技大学 | The cutaneous lesions deep learning identifying system of Analysis On Multi-scale Features based on expansion convolution |
| CN108230329A (en) * | 2017-12-18 | 2018-06-29 | 孙颖 | Semantic segmentation method based on multiple dimensioned convolutional neural networks |
| CN108681692A (en) * | 2018-04-10 | 2018-10-19 | 华南理工大学 | Increase Building recognition method in a kind of remote sensing images based on deep learning newly |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709387A (en) * | 2020-06-22 | 2020-09-25 | 中国科学院空天信息创新研究院 | Building segmentation method and system for high-resolution remote sensing image |
| CN111709387B (en) * | 2020-06-22 | 2023-05-12 | 中国科学院空天信息创新研究院 | Building segmentation method and system for high-resolution remote sensing image |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113850825B (en) | Remote sensing image road segmentation method based on context information and multi-scale feature fusion | |
| CN109299274B (en) | Natural scene text detection method based on full convolution neural network | |
| CN115049936B (en) | High-resolution remote sensing image-oriented boundary enhanced semantic segmentation method | |
| CN109858372B (en) | Lane-level precision automatic driving structured data analysis method | |
| CN108776772B (en) | Cross-time building change detection modeling method, detection device, method and storage medium | |
| CN113850824B (en) | Remote sensing image road network extraction method based on multi-scale feature fusion | |
| CN106897681B (en) | Remote sensing image contrast analysis method and system | |
| CN111723798B (en) | Multi-instance natural scene text detection method based on relevance hierarchy residual errors | |
| WO2021088101A1 (en) | Insulator segmentation method based on improved conditional generative adversarial network | |
| CN110020658B (en) | Salient object detection method based on multitask deep learning | |
| CN111951284B (en) | Optical remote sensing satellite image refined cloud detection method based on deep learning | |
| US20220398737A1 (en) | Medical image segmentation method based on u-network | |
| CN111524117A (en) | Tunnel surface defect detection method based on characteristic pyramid network | |
| CN110728640A (en) | Double-channel single-image fine rain removing method | |
| CN116258719A (en) | Flotation foam image segmentation method and device based on multi-mode data fusion | |
| CN113723377A (en) | Traffic sign detection method based on LD-SSD network | |
| CN110852327A (en) | Image processing method, image processing device, electronic equipment and storage medium | |
| CN111353396A (en) | Concrete crack segmentation method based on SCSEOCUnet | |
| CN114913493A (en) | Lane line detection method based on deep learning | |
| CN112819837A (en) | Semantic segmentation method based on multi-source heterogeneous remote sensing image | |
| CN116958827A (en) | Deep learning-based abandoned land area extraction method | |
| CN114445442A (en) | Multispectral image semantic segmentation method based on asymmetric cross fusion | |
| CN1252588C (en) | High spectrum remote sensing image combined weighting random sorting method | |
| CN111145178A (en) | High-resolution remote sensing image multi-scale segmentation method | |
| CN116012709B (en) | High-resolution remote sensing image building extraction method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200512 |
|
| WD01 | Invention patent application deemed withdrawn after publication |