CN114155253B

CN114155253B - Connection method for strip-shaped element fracture based on remote sensing image segmentation

Info

Publication number: CN114155253B
Application number: CN202111498196.8A
Authority: CN
Inventors: 袁晓军; 周乐乐
Original assignee: University of Electronic Science and Technology of China
Current assignee: University of Electronic Science and Technology of China
Priority date: 2021-12-09
Filing date: 2021-12-09
Publication date: 2023-04-07
Anticipated expiration: 2041-12-09
Also published as: CN114155253A

Abstract

The invention belongs to the field of computer vision image processing, and particularly relates to a method for connecting broken strip-shaped elements based on remote sensing image segmentation. In the semantic segmentation of the remote sensing image, the finally generated prediction result is generally a classification result gray-scale map. However, the prediction results of narrow and narrow strip-shaped elements such as roads and rivers are generally not ideal, and a road/river fracture phenomenon often occurs, and the fracture phenomenon is a great difficulty in the task of processing and segmenting the remote sensing image. The method comprises the steps of obtaining a remote sensing image segmentation map by using a deep learning semantic segmentation algorithm, and realizing the connection of narrow, long and thin or strip-shaped elements such as broken roads and rivers through nine steps of splicing, element extraction, expansion, element skeleton extraction, end point detection, end point extension search, expansion, overlapping coverage and the like. The technical achievement of the invention can be used for connecting roads and rivers in the remote sensing image segmentation task and connecting other strip-shaped broken elements.

Description

Connection method for strip-shaped element fracture based on remote sensing image segmentation

Technical Field

The invention belongs to the field of computer vision image processing, and particularly relates to a method for connecting broken strip-shaped elements based on remote sensing image segmentation.

Background

In a remote sensing image segmentation task, a deep learning algorithm based on a convolutional neural network is adopted in the current mainstream technology, the construction of a deep learning algorithm network model architecture is not a clear guiding principle, the deep learning algorithm network model architecture is still a black box and is usually based on an empirical rule, and the common method is to extract features by adopting a backbone network widely known in the field of computer vision at present and then self-define and output a result head network structure based on a specific task. Deep learning neural network models typically require a large amount of data to train and the distribution of data has a large impact on the performance of the model, which typically results in poor prediction performance for less-than-many classes if the training data classes are unbalanced. Two general solutions for dealing with such problems exist, one is from the perspective of Data, data enhancement is simply and roughly performed, that is, objects occupying fewer categories are copied and pasted (Golnaz Ghiasi, yin Cui, aravidin Srinivas, rui Qian, tsung-Yi Lin, ekin d. Cube, quoc v. Le, barret zoph. Simple Copy-Paste is a Strong Data amplification Method for Instance Segmentation), so that the proportion of elements occupying fewer categories is increased, and the model prediction performance is further improved. However, such copying and pasting is suitable for elements with specific shape boundaries, such as spheres, animals, etc., but is not suitable for elements without boundaries, such as roads or rivers, and it is difficult to achieve precise extraction and copying. Another method is to suppress the contribution of the class with a large proportion to the model total Loss and enhance the contribution of the class with a small proportion to the model total Loss from the viewpoint of the Loss function, such as Focalloss "Tsung-Yi Lin, priya Goyal, ross Girshick, kaiming He, piotr dontar. Focal local for detect Object Detection", in this way, the image of the data distribution class non-uniformity on the model performance is reduced, and the model total performance is enhanced. However, even with the above-mentioned solutions, the influence of the above on the model performance is only reduced to a certain extent but cannot be completely solved. In the task of segmenting the remote sensing image, the neural network model has less category occupation, and simultaneously, elements with long and narrow strip shapes, such as roads in the countryside or small rivers, are predicted to have fracture. Because such elements usually only occupy the width of a few pixels on an image, and due to the fact that trees on the two sides of a rural road or a river are shielded, a prediction result is broken, and the breaking problem is always a great difficulty in the task of deeply learning remote sensing image segmentation.

Disclosure of Invention

In view of the above problems, since such problems cannot be solved well on the network model, the present invention considers the connection of the broken elements by post-processing on the results output by the model. Sinkawa and the like (sinkawa, gayuou, velcade, broken road connection method based on high-resolution remote sensing images 2020.28 (2)) are also broken road connections from the perspective of post-processing, but the connection methods are different. The search direction determination of shenchuan and the like only uses the adjacent pixel points of eight neighborhoods of the end point to perform direction determination, and the direction is unstable. The direction of the invention adopts the vector determination between the end point and the backtracking point which backs a certain number of pixels along the skeleton line, and the direction of the skeleton line is used for assisting the confirmation of the detection direction of the end point, so that the direction is more accurate and reasonable, and the robustness is better.

The invention is based on the remote sensing image segmentation task, and carries out post-processing on the condition that long, narrow and thin strip-shaped elements appearing in the remote sensing image segmentation result are broken, so as to realize the connection of the broken elements. The method well realizes the connection of the strip-shaped broken elements such as roads and rivers through 9 steps of remote sensing image segmentation, splicing, element extraction, segmentation result expansion, element skeleton extraction, skeleton end point detection, end point extension scanning, expansion, superposition and the like.

The steps and the specific scheme of the invention are as follows:

step 1: and obtaining a remote sensing image segmentation result graph. The remote sensing image is segmented mainly by using a deep learning algorithm. The invention uses DeepLabv3+ (Chen, L.C., zhu, Y.K., papandreuu, G., schroff, F., adam, H.: encoder-Decoder with associated Separable convergence for Semantic Image Segmentation) to extract Image features from remote sensing images, and two network heads are grafted on the feature map output by DeepLabv3+ for obtaining the final Segmentation result gray map. The two network heads in the invention respectively correspond to different classification numbers, one is a classification network head of 8 categories, and the other is a classification network head of 25 categories, and can be flexibly switched according to the requirements. Different classification codes are represented as pixel values on the segmentation result map, namely the pixel values are classification code values. The concrete classification is as follows:

the numerical codes and corresponding categories of the 8 major classes are as follows: 1, ploughing, 2, garden, 3, forest land, 4, grassland, 5, a construction land, 6, a transportation land, 7, a water area and an auxiliary device, and 8, the rest;

the 25 subclasses of numerical codes and corresponding classes are as follows: 11: paddy field, 12: water-irrigated land, 13: dry land, 21: orchard, 22: tea garden, 23: other parks, 31: woodland, 32: shrub forest land, 33: other woodland, 41: the natural pasture land comprises 42 parts of artificial pasture land, 43 parts of other pasture land, 51 parts of land for urban construction, 52 parts of land for rural construction, 53 parts of mining land, 54 parts of land for other construction, 61 parts of rural road, 62 parts of land for other transportation, 71 parts of river, lake and pond, 72 parts of: wetland, 73: glacier and permanent snow, 81: saline-alkali soil, 82: sand, 83: bare land 84: bare rock gravel ground.

Among the above categories, rivers in water and roads for transportation are narrow and small elements. As described above, due to the defect of the deep learning algorithm, the result of this step is that, in the case where the prediction of the long, narrow and thin strip-like elements has a break, the broken elements need to be connected.

Step 2: and (5) splicing gray level images. Since the remote sensing image is usually a large-size image such as 16000x24000, etc., but is limited by computer hardware resources such as memory, video memory, etc., it is impossible to process such a large image at one time, and in practice, the processing is to cut the large-size image into many small-size images with fixed sizes. Such as 512x512, and then fed into the neural network for prediction. After the network partition, the size of the obtained result is the same as the input, for example, the input is 512x512, and the output is still 512x512. However, in order to restore the large-scale segmentation result, the individual small-scale segmentation results need to be spliced according to the positions of the individual small-scale segmentation results in the original large-scale image, and finally the segmentation results of the large-scale image are spliced.

And step 3: and extracting fracture elements. And (3) extracting the broken elements according to the image gray values on the basis of the splicing result of the step (2), and mapping the positions of the broken elements to a binary image with the same size as the original image. A pixel value of 1 indicates an element to be connected, and a pixel value of 0 indicates a background. The main purpose of mapping onto a binary image is to enable dilation-erosion operations using morphological techniques in conventional image processing.

And 4, step 4: morphological dilation. And (4) selecting a proper expansion value on the binary image extracted in the step (3) according to the degree of element fracture, wherein a dilate function in OpenCV is used for expansion operation, and the function parameter kernel value is set to be (13, 13). After expansion, a large fraction of fractured fine debris at distances less than 13 pixels will be connected together, which greatly reduces the number of fractured patches. At the same time, the number of end points of the skeleton is also reduced. And for the part with longer missing part, an end point extension detection method is adopted for solving the problem.

And 5: extracting the skeleton of the element after expansion. This step will refine the strip-like elements in the result of step 4 to have a width of only one pixel. The result after refinement is that the neighboring pixels have either only one common edge neighbor or only two kinds of common diagonal vertices, and there are no other neighbors.

Step 6: and detecting an end point. After the morphological dilation is carried out in the step 4, most of the fracture parts are connected, so that the number of the bone end points after the bone is thinned in the step 5 is reduced, the number of the end points needing to be traversed is greatly reduced, and the processing speed is further improved.

The end points of the bone need to be located after bone extraction. The following is the identification of endpoints on the bone, and our endpoint determination algorithm is as follows:

the coordinates of points on all bones are found. Taking these points as the center, in the eight neighborhood range of 3x3, if the pixel value of only one point except the center point is 1 and the other pixel values are all 0, then the center point of the 3x3 region is the bone end point.

After endpoint detection, a set of coordinates for all endpoints is obtained.

And 7: the endpoint extension scan. After the set of endpoints is obtained, the coordinates of the backtracking point are obtained for each endpoint by a distance of 13 pixels back along the bone line. This point is used to assist in constructing the extension direction vector.

Next, a square area with length of side 2l +1 is constructed with the end point as the center, and in this square area, points on the skeleton satisfying the following conditions are found. The specific L value can be adjusted according to the fracture condition, and L =30 in the invention. That is, a point satisfying the following condition is searched for within a square area of 61 × 61 centered on the end point.

1) The vector formed by the line connecting the point and the end point

Constitutes a vector ≥ with an endpoint and a backtracking point>

The included angle between the two is less than 60 degrees, namely the cosine value is more than 0.5.

2) The distance from the point to the end point is smaller than the set end point extension detection distance, and the end point extension detection distance is set to be 45 pixels.

Among the point sets satisfying the above condition, the point closest to the end point is selected for connection.

And step 8: morphological dilation. Morphological dilation is performed on top of the result of step 7, where the dilation kernel parameter is set to (4, 4). The main purpose of this expansion operation is to give the bone parts connected in step 7 a certain pixel width.

And step 9: and (6) overlapping and covering. And (4) carrying out pixel-by-pixel logical OR operation on the result of the step (8) and the binary result image of the step (3) to obtain a binary image result of which the connection of the broken elements is finished. Then, the coordinates of all non-0 pixels in the binary image result are extracted, and the pixels at these coordinate positions in the image resulting from step 2 are replaced with the pixel values of the broken elements.

After the nine-step operation, the connection of the broken elements in the segmentation result is realized.

The method has the advantages that on the basis of the remote sensing image segmentation result, the method realizes the connection of broken strip long and narrow small elements by using the traditional image processing means and adopting the modes of expansion, thinning and extracting the skeleton, end point extension scanning, re-expansion and overlapping coverage.

The invention is based on the post-processing carried out on the classification result of the remote sensing image segmentation output. It can also be used to connect other elongated strip-like elements. After connecting the long and narrow strip-shaped elements, the method is greatly helpful for downstream tasks such as road and river centerline extraction and the like.

Drawings

Fig. 1/3/5 is a binary image of a fracture element in the original segmentation result of the remote sensing image extracted in step 3.

Fig. 2/4/6 is a binary image of the extracted connected broken elements after the operation of step 9 is completed.

Detailed Description

The best mode for carrying out the invention having been described in detail in the summary of the invention, the following description is provided in conjunction with the accompanying drawings to show the effect of the invention in practical application to show the technical progress made by the invention.

Fig. 1/3/5 show the case of broken elements (white pixels in the figure) extracted from the segmentation result of the deep learning algorithm without using the method of the present invention.

FIGS. 2/4/6 show the situation after the connection of the broken elements by the method of the present invention.

From the above, after the processing of the invention, the connection of the narrow and small or strip-shaped elements such as broken roads and rivers is realized, and the extraction effect of the downstream task of the remote sensing image processing, such as the central line of roads and rivers, is greatly improved.

Claims

1. A method for connecting strip-shaped element fracture based on remote sensing image segmentation is characterized by comprising the following steps:

step 1: segmenting the obtained remote sensing image through a deep learning algorithm to obtain a gray level image;

step 2: splicing the obtained gray level images to form a segmentation result of a large-size image;

and 3, step 3: on the result of splicing in the step 2, extracting the broken elements according to the gray value of the image, mapping the positions of the required pixels to a binary image with the same size as the original image, defining that the pixel value is 1 to represent the elements to be connected, and defining that the pixel value is 0 to represent the background;

and 4, step 4: selecting a proper expansion value to perform morphological expansion according to the degree of element fracture, specifically using a dilate function in OpenCV, and setting an expansion parameter kernel to be (13, 13);

and 5: extracting the skeleton of the dilated elements, thereby refining the strip-like elements to have a width of only one pixel, and the result after refinement is a case where adjacent pixels have either only one common edge or only two adjacent diagonal common vertices;

step 6: the specific endpoint determination algorithm is as follows:

finding out coordinates of points on all bones, and taking the points as centers, in an eight-neighborhood range of 3x3, if the pixel value of only one point except a central point is 1 and the other pixel values are 0, then the central point of the 3x3 area is a bone endpoint;

obtaining a coordinate set of all endpoints through endpoint detection;

and 7: after the end point set is obtained, for each end point, backing a distance of 13 pixels along a skeleton line, and obtaining coordinates of a backtracking point, wherein the coordinates of the backtracking point are used for assisting in constructing an extending direction vector;

taking an end point as the center, constructing a square area with the side length of 2L +1, wherein the value of L is set to be 30, and searching points on bones meeting the following conditions in the square area:

1) The vector formed by the line connecting the point and the end point

Constitutes a vector ≥ with an endpoint and a backtracking point>

The included angle between the two is less than 60 degrees, namely the cosine value is more than 0.5;

2) The distance from the point to the end point is less than the set end point extension detection distance;

selecting a point closest to the endpoint from the point set meeting the condition to connect with the endpoint;

and step 8: on top of the result of step 7, morphological dilation is performed with the dilation parameter kernel set to (4, 4);

and step 9: performing pixel-by-pixel logic or operation on the result of the step 8 and the binary result image of the step 3 to obtain a binary image result of which the connection of the broken elements is completed; then, the coordinates of all non-0 pixels in the binary image result are extracted, and the pixels at the coordinate positions in the image obtained in step 2 are replaced by the pixel values of the broken elements, so that the connection processing of the broken elements is completed.