AU2020103026A4 - A Single Tree Crown Segmentation Algorithm Based on Super-pixels and Topological Features in Aerial Images - Google Patents

Abstract

The invention discloses an individual tree crown(ITC) segmentation algorithm based on aerial images using super-pixel and topological features, which comprises the following steps: performing Simple Linear Iterative Clustering (SLIC) superpixel segmentation on original aerial images, and simultaneously acquiring the coronal boundaries of the images via holistically-nested edge detection (HED) network; Calculating three similarity measurement indexes between two adjacent superpixels, i.e., the difference between RGB average values of two adjacent superpixels, the number of intersecting pixels of two adjacent superpixels, and the number of intersecting boundary pixels obtained from HED network, by which the similarity weights between two adjacent superpixels are constructed; Conducting a superpixel neighborhood connected graph based on the center point of each superpixel, and the minimum spanning tree(MST) is extracted from the connected graph to generate the connected tree of aerial images. Then, the superpixels are merged according to the calculated similarity weight matrix between the superpixels, in order to realize the ITC segmentation. According to the invention, the superpixel segmentation and the topological graph method are combined, so that the ITC can be accurately and effectively separated from the aerial images with high segmentationprecision. 3/6 stage 1 VGG16 network architecture stage 2 stage 3 stage 4 stg5sae6 K stae ge4 IlK~li~i11tagege convolution+ReLU pooling layer fully connected+ReLU softmax HED network architecture stage stage 2 stage 3 stage stage econvolution+ReLU ... I. .. .. .pooling layer (7Jside-output layer i transposed transposed transposed transposed transposed onouonconvolution convolution Oilusi convolution (b) xx4 x8 x16 concatenationOtt Figure 4 (a) (b) (c) Figure 5

Description

3/6

stage 1 VGG16 network architecture

stage 2

stage 3

IlK~li~i11tagege stage 4 K stae stg5sae6 ge4

convolution+ReLU pooling layer fully connected+ReLU softmax

HED network architecture stage

stage 2

stage 3 stage stage

econvolution+ReLU ... I. .....pooling layer (7Jside-output layer

i transposed transposed transposed transposed transposed onouonconvolution convolution Oilusiconvolution (b) xx4 x8 x16

concatenationOtt

Figure 4

(a) (b) (c)

Figure 5

A Single Tree Crown Segmentation Algorithm Based on Super-pixels and

Topological Features in Aerial Images

TECHNICAL FIELD

[01] The invention relates to the technical field of forest detection, in particular to an individual tree crown segmentation algorithm based on aerial images using super pixels and topological features.

BACKGROUND

[02] As one of the key links of the whole ecological cycle, forests have significant environmental and economic benefits. Obtaining individual information of forest trees is the basis of forest resources analysis. Analyzing the structure and characteristics of trees on the scale of a single tree is beneficial to forest inventory and management planning. Individual tree segmentation conveniently describes the distribution of forest vegetation, it also provides strong support for tree species and calculation of trunk volume, biomass and carbon storage.

[03] Traditional field investigation is time-consuming, expensive and labor intensive, which is not conducive to large-scale and complex forestry monitoring. With the rapid development of remote sensing technology, using remote sensing technology to obtain data sources has become an effective alternative method for traditional field measurements, such as obtaining point cloud data by airborne lidar and aerial images by unmanned aerial vehicle (UAV). In recent years, high-resolution aerial images obtained by unmanned aerial vehicles have been widely used in forestry and agricultural surveying and mapping, which is mainly provided by relevant institutions. Compared with satellite images, aerial images are less affected by weather and can obtain detailed tree information. In addition, compared with traditional aircraft, UAV is safer, more convenient and faster in acquiring data, and its cost is also lower. Therefore, unmanned aerial vehicle (UAV) has become one of the priority choices for researchers to obtain remote sensing data in high altitude or remote areas. The research trend has gradually changed from simple artificial forest to complex mixed forest or natural forest based on the application and development of aerial images.

[04] Typical image processing algorithms, such as marker-based watershed algorithm, region growing algorithm and local maximum filtering algorithm, are usually used for tree crown segmentation of aerial images. The mark-controlled watershed algorithm can reduce the excessive segmentation in traditional watershed algorithms. The core goal of this algorithm is to mark and divide the image into watersheds according to prior knowledge. The region growing algorithm can directly extract the crowns from the original lidar data, which is a commonly used method to segment 3D point clouds, but this method can only identify the spatial distribution of objects to a limited extent. The local maximum filtering algorithm is that each crown has a single "bright spot", and the area between the crowns is darker than the sun irradiated part of the crowns. However, the noise in the image can easily lead to excessive segmentation. Image segmentation method based on gray threshold and edge detection is a traditional image segmentation method. The concept of superpixel was first proposed by Ren et al. in 2003. Superpixel is a region composed of a series of pixels with similar characteristics, which retains effective information and is convenient for further image segmentation. Besides, the boundary information of objects in the image will not be damaged under normal conditions. In 2012, Achanta et al. proposed a simple linear iterative clustering (SLIC) super-pixel algorithm, which has been widely used in machine vision and image processing. It is a superpixel segmentation algorithm based on k-means clustering. In addition, SLIC is easy to use and understand, faster and more efficient than existing methods, and has the most advanced image boundary dependency. Although the ITC segmentation algorithm mentioned above has achieved success in some studies, its segmentation effect is poor in complex environments such as continuous overlapping of crowns, different sizes of crowns, and oblique growth of trees. The overlapping of the adjacent crowns makes the boundary of crowns unclear, and trees with smaller height and width and occluded by other larger trees are often ignored. These factors may lead to under-segmentation of crowns. In previous studies, most of the study areas only contained a small amount of tree species diversity, but the segmentation results of different tree species in different environments were quite different. In addition, most of the existing researches on crown segmentation choose coniferous forest as the research object, mainly because the canopy structure of the coniferous forest is relatively regular, and the forest canopy center is easier to be detected and identified.

[05] In addition to traditional image processing algorithms, machine learning methods and topological features can be used for individual tree crown segmentation. For example, Xie(2017) first proposed fully-nested edge detection (HED) based on fully convolutional neural (FCN) network and deep supervision network. In addition, under the guidance of in-depth monitoring, the HED network automatically learns multi-layer representation and gradually generates detailed edge mapping to achieve better edge detection. As a graph-based method, the minimum spanning tree (MST) has a topological structure and has been widely used in image segmentation.

[06] For some existing tree crown segmentation methods, such as the circular Hough transform, which combines canny edge detection method and multi-scale filtering and segmentation (MSF) method. In this method, the canny edge detection algorithm is used to extract canopy boundaries from the same tree species, and circular Hough transform algorithm can be used to obtain the circular shapes in the image, so as to identify a single crown. The MFS method generates multi-scale representation of the image via multi-scale analysis, and then uses watershed segmentation method to generate multi-scale segmentation map to acquire accurate forest canopy segmentation boundaries and effectively reduce over-segmentation. However, the circular Hough transform method is limited to tree species with regular shape (such as citrus). This method has a good effect only on plantations with the same afforestation method and similar phenotypic characteristics of crowns, i.e., trees are arranged neatly and the planting spacing is fixed (stand density is low). The crown shape is roughly round, which is not suitable for natural forest stands composed of various tree species with irregular crown shapes. The segmentation results of the MFS method are easily interfered by shadows under the irradiation of solar radiation with fixed zenith angle. Because the segmentation boundary of the filtered image is refined in the process of multi-scale filtering, the computational complexity of MFS method is high. In addition, many oblique branches of a larger tree may be mistaken as a small crown.

[07] The tree crown segmentation algorithm based on aerial images is a prerequisite for understanding tree growth, tree species competition and biomass assessment. For different types of forests, how to provide a new tree crown segmentation algorithm to improve the segmentation accuracy is still a problem to be solved.

SUMMARY

[08] The technical problem to be solved by the present invention is to provide an aerial image individual tree crown segmentation algorithm based on superpixel and topological features, which combines superpixel segmentation with topological graph method, and can accurately and effectively separate individual tree crown from aerial images with high segmentation accuracy.

[09] In order to achieve the above technical purpose, the technical scheme adopted by the invention is as follows:

[010] The tree crown segmentation algorithm for aerial images based on superpixels and topological features comprises the following steps:

[011] (1) SLIC super-pixel segmentation is performed on the original aerial images, and the coronal boundary of the image is obtained by HED network;

[012] (2) Calculating three similarity measurement indexes between two adjacent superpixels, i.e., the difference of RGB average values of two adjacent superpixels, the number of intersecting pixels of two adjacent superpixels and the number of boundary pixels obtained by HED network, which is used to construct similarity weights between two adjacent superpixels;

[013] (3) Based on the center point of each superpixel, constructing a superpixel neighborhood connected graph, extracting the MST from the superpixel neighborhood connected graph to generate the connected tree of aerial images, and merging the superpixels according to the calculated similarity weights to realize the ITC segmentation.

[014] As a further improved technical scheme of the present invention, the step (1) further comprises the following content:

[015] CCD camera and optical camera are used to collect aerial images of the study area.

[016] As a further improved technical scheme of the present invention, the step (2) comprises the following content:

[017] A superpixel neighborhood connected graph G = (V, B) is constructed based on the center point of each superpixel, and the node vi E V corresponds to a superpixel region, b(vi, vj) E B is the branch connecting two adjacent superpixels, and

the weight of wb(vi, vj) represents the similarity weight between the two adjacent

superpixel regions vi and vj connected by the branch b(vi, vj). The expression of the

similarity weight is as below:

[018] wb = awl + 4 +yw (1) Wb

[019] Where a, P, y are weight coefficients, wl, w', w' are three similarity measurement indexes of neighbouring superpixels, respectively.

[020] wj represents the measure of the difference between RGB mean value of two adjacent superpixels vi and v, and the color mean value of each superpixel region

is recorded as the color value of the corresponding superpixel, and the color value of the superpixel region of vi is recorded as (rv,, g,,, bv,), the color value of the superpixel

region vj is recorded as (rvj, gj, bv,). The calculation formula of wl is as follows:

[021] w = (r - ry)+(g- gv) 2 + (bv, - bvj)z (2)

[022] wb represents the similarity measure of the number of intersecting pixels of two adjacent superpixel regions vi and v. The calculation formula ofw2 is as follows:

[023] w = internumber(vi,vj) (3)

[024] wb represents the similarity measurement based on the edges of the tree

crowns generated by HED network in the intersecting area of two adjacent superpixel vi and v1 , the calculation formula of w3 is expressed as follows:

[025] wb = Edge(vi,v) lE, (4)

[026] where Edge(vi, vj) represents the intersected boundaries of two adjacent

superpixel regions, and E, denotes the number of boundary pixels in their intersecting area of two adjacent superpixel regions defined by HED network.

[027] The individual tree crown segmentation based on aerial images using superpixel and topological features according to claim 1, characterized in that (3) comprises the following content:

[028] The maximum weighted branch of the local minimum spanning tree is defined as the maximum weight of the MST in the individual region. The expression is as follows:

[029] Int(vi) max wb (5) b(vi,vj)EMST(V,B)

[030] MST(V, B) represents the minimum spanning tree composed of the branch set in the individual region;

[031] The intra-class difference between two adjacent superpixel regions vi and vj is defined as follows:

[032] D if (vi, vj) min wb(vi, vj) (6) Vi,VjEV,b(vi,vj)EB

[033] When the difference between two adjacent superpixel areas vi and vj is less than MInt(vi, vj), merging the two superpixels, otherwise, the superpixels are not merged:

( true, Dif(vi,v;) < MInt(vi,v;) (

false , Dif(vi,vj) MInt(vi,vj)

[035] MInt(vi, v1 ) = min{Int(vi) + T(vi), Int(vj) + T( vj)} is the minimum

value of intra-class differences; T( vi) is a threshold function, the expression is T( vi) -, and m is a constant parameter. Iv~i

[036] The method provided by the invention has the beneficial effects that the individual tree crown segmentation method combined with SLIC super-pixel segmentation algorithm, HED network and MST method in topology can accurately and effectively segment forest crowns in aerial remote sensing images. Firstly, the aerial remote sensing images are segmented by superpixels based on color measurement, and the coronal boundaries are obtained by HED network. At the same time, three measurement indexes, that is, the RGB color value, the number of intersecting pixels and the difference of the number of boundary pixels defined by HED network in the intersecting area, are used to calculate the similarity weights between two adjacent superpixels. Finally, the connected tree of aerial images at super-pixel scale is generated by using the MST method, and the superpixels are merged according to the calculated similarity weights, so as to realize ITC segmentation with high segmentation accuracy. The method provided by the invention has a good application prospect for ITC segmentation of forest aerial images, and provides a new concept based on image processing technology for adapting to different types of forests.

BRIEF DESCRIPTION OF THE FIGURES

[037] Fig. 1 shows aerial images of three corresponding research areas acquired by unmanned aerial vehicles from Guangxi Gaofeng Forest Farm.

[038] Fig. 1(a) is the aerial image of Cunninghamialanceolatamainly in withered or deciduous state in forest plot Al.

[039] Fig. 1(b) is the aerial image of Pinus massonianain forest plot A2.

[040] Fig. 1(c) is the aerial image of forest plot A3 composed of eucalyptus trees affected by uneven sunlight.

[041] Fig. 2 is the gray scale diagram of Fig. 1.

[042] Fig. 2(a) is the gray scale diagram of Fig. 1(a).

[043] Fig. 2(b) is the gray scale map of Fig. 1(b).

[044] Fig. 2(c) is the gray scale map of Fig. 1(c).

[045] Fig. 3 is the architecture diagram of VGG16 network architecture (a) and HED network architecture (b).

[046] Fig. 4 is the gray scale diagram of Fig. 3.

[047] Fig. 5 is the results of edge detection of aerial images of three experimental areas using HED network and histogram equalization method.

[048] Fig. 5(a) is the result of edge detection on aerial images of forest plot Al.

[049] Fig. 5(b) is the result of edge detection on aerial images of forest plot A2.

[050] Fig. 5(c) is the result of edge detection on aerial images of forest plot A3.

[051] Fig. 6 is the explanatory diagram of the results of the minimum spanning tree algorithm.

[052] Fig. 6(a) is the superpixel neighborhood connected graph G constructed based on the center points of the superpixels.

[053] Fig. 6(b) is the schematic diagram of the minimum spanning tree extracted from graph G.

[054] Fig. 7 is the gray scale diagram of Fig. 6.

[055] Fig. 7(a) is the gray scale diagram of Fig. 6(a).

[056] Fig. 7(b) is the gray scale diagram of Fig. 6(b).

[057] Fig. 8 is the results of super-pixel segmentation of aerial remote sensing images of three research sites by using SLIC super-pixel algorithm.

[058] Fig. 8(a) is the result of super-pixel segmentation of aerial remote sensing image of forest plot Al by using SLIC super-pixel algorithm.

[059] Fig. 8(b) is the result of super-pixel segmentation of aerial remote sensing image of forest plot A2 by using SLIC super-pixel algorithm.

[060] Fig. 8(c) is the result of super-pixel segmentation of aerial remote sensing image of forest plot A3 by using SLIC super-pixel algorithm.

[061] Fig. 9 is the gray scale diagram of Fig. 8.

[062] Fig. 9(a) is the gray scale diagram of Fig. 8(a).

[063] Fig. 9(b) is the gray scale diagram of Fig. 8(b).

[064] Fig. 9(c) is the gray scale diagram of Fig. 8(c).

[065] Fig. 10 is the schematic diagram of MSTs extracted from connected graph G at superpixel scale.

[066] Fig. 10(a) is the schematic diagram of MSTs extracted from the connected graph G at superpixel scale of forest plot Al.

[067] Fig. 10(b) is the schematic diagram of MSTs extracted from the connected graph G at superpixel scale of forest plot A2.

[068] Fig. 10(c) is a schematic diagram of MSTs extracted from the connected graph G at superpixel scale of forest plot A3.

[069] Fig. 11 is the gray scale diagram of Fig. 10.

[070] Fig. 11(a) is the gray scale diagram of Fig. 10(a).

[071] Fig. 11(b) is the gray scale diagram of Fig. 10(b).

[072] Fig. 11(c) is the gray scale diagram of Fig. 10(c).

[073] Fig. 12 is the schematic diagram of the final results of individual tree crown segmentation after superpixel merging based on MST results.

[074] Fig. 12(a) is the schematic diagram of the individual tree crown segmentation result of the image of the forest plot Al.

[075] Fig. 12(b) is the schematic diagram of the individual tree crown segmentation result of the image of the forest plot A2.

[076] Fig. 12(c) is the schematic diagram of the individual tree crown segmentation result of the image of the forest plot A3.

[077] Fig. 13 is the gray scale diagram of Fig. 12.

[078] Fig. 13(a) is the gray scale diagram of Fig. 12(a).

[079] Fig. 13(b) is the gray scale diagram of Fig. 12(b).

[080] Fig. 13(c) is the gray scale diagram of Fig. 12(c).

DESCRIPTION OF THE INVENTION

[081] The specific embodiments of the present invention will be further explained below according to the drawings.

[082] The invention provides an aerial image individual tree crown segmentation algorithm based on superpixel and topological features, which combines superpixel segmentation and topological graph method to effectively separate the ITC from aerial images. Firstly, aerial images of forest plots aquired by unmanned aerial vehicles are segmented by using the simple superpixel linear iterative clustering(SLIC) algorithm, and the tree crown boundries of aerial images are obtained by the deep learning concept of holographic-nested edge detection (HED) network. Secondly, the similarity weights of adjacent superpixels are measured by three indicators: the difference of RGB color value, the number of intersectinng pixels and the number of boundary pixels defined by HED network. Finally, the minimum spanning tree of topological method is used to generate the connected tree of super-pixel aerial images, and the superpixels are merged according to the calculated similarity weights, so as to realize ITC segmentation.

[083] Details are as follows:

[084] 1. Area under study and image acquisition;

[085] (1.1) Area under study;

[086] The research of this example was carried out in Gaofeng Forest Farm, Guangxi Zhuang Autonomous Region (22 49 '-23 5 'N, 108 7 '-108 38 'E). The total land area of Gaofeng Forest Farm is 593.34 km2 , including 320 km 2 internal area and 273.34 km2 external greening area. It is located on the south side of Tropic of Cancer and on the northern edge of Nanning Basin in Guangxi, belonging to Daming Mountain Range. The terrain is dominated by hills, which are high in the east and low in the west, with hills accounting for 55.5% and mountains accounting for 38.7%. The watershed in the central region has an altitude of about 450 meters (higher than the average sea level) and a relative height of 100-200 meters. The average elevation of woodland on both sides of watershed is below 300 meters. The study area belongs to the tropical northern edge climate with abundant rainfall. The annual average temperature is about 21°C, the extreme minimum and maximum temperatures are -2°C and 40°C respectively, and there is ice in the upper part of the mountain. The annual accumulated temperature is about 7500°C, and the annual rainfall is 1200 - 1500mm, mainly concentrated in June ~ September every year. The climate is hot and humid. In this embodiment, three representative forest farms are selected as study areas. These data are obtained by aerial remote sensing and field investigation. The forest land areas of 2 forest plot Al, forest plot A2 and forest plot A3 are 1433.45 M 2 , 1300 m and 925.93 m2 respectively, which are composed of Cunninghamialanceolata,Pinus massoniana and Eucalyptus respectively.

[087] (1.2) Aerial images;

[088] High resolution CCD camera and optical camera are used to collect aerial images of the study area. The CCD camera used in the study has a field of view of 70 and a focal length of 50 mm. In the process of data acquisition, the flight speed and altitude are set to 18m/s and 55m respectively (higher than the altitude of the takeoff position). Fig. 1 shows the remote sensing digital aerial images of three research forest plots. Fig. 2 is the gray scale diagram of Fig. 1.

[089] Research methods;

[090] (2.1) Superpixel segmentation;

[091] In this embodiment, the SLIC superpixel algorithm is used to generate compact and approximately consistent superpixels, so as to perform superpixel segmentation on the remote sensing image. The algorithm selects multiple cluster centers on the image, and divides pixels into the most similar cluster ranges by using the similarity between pixels and cluster centers.

[092] Firstly, SLIC algorithm converts the image from RGB color space to CIE Lab color space. The color value (L,a,b) and coordinate (x,y) corresponding to each pixel constitute a five-dimensional vector [L,a,b,x,y], and the similarity of two pixels can be measured by their vector distance. The greater the distance, the smaller the similarity.

[093] The specific implementation process of SLIC segmentation is as follows: (1) Initializing cluster centers: according to the set number of superpixels, the cluster centers (seed points), that is, the centers of divided regions, are evenly distributed in the image. In this embodiment, the total number of N pixels of the image can be assumed and divided into K superpixels with the same size in advance. The size of each superpixel is N/K, and the distance between two adjacent seed points is approximately S = sqrt (N/K). (2) Re-selecting the seed points in the original seed point n*n square matrix (n = 3 is generally selected). The specific method is as follows: calculating the gradient values of all pixels in the neighborhood of the seed point, and moving the seed point to the point with the smallest gradient in the neighborhood; (3) Assigning a class label to each pixel around each seed point (that is, the cluster center the seed point belongs to). Different from the standard of searching the whole image by K- means, the search range of SLIC for pixels is limited to 2S*2S, which can accelerate the convergence of the algorithm; (4) Distance measurement: including color distance (L,a,b) and space distance (x,y). For each searched pixel, the distance from it to the seed point needs to be calculated separately, and then the seed point corresponding to the minimum value is selected as the clustering center of the pixel. Cluster centers are iterated and updated continuously until there is no obvious change in cluster centers. SLIC superpixel algorithm is used to divide pixels with similar features into the same superpixel area in the segmented aerial image. Transforming image pixel set I into superpixel set V: I -- V.

[094] (2.2) Holistically-nested edge detection (HED) network for aerial images of forest plots;

[095] HED network is a multi-scale, multi-level and multi-fusion network structure realized by using VGG16 convolutional neural network (CNN) and depth supervision technology. With deep learning model, the detected edges are more coherent and the internal noise points are reduced. HED network is an improvement based on VGG16 network. The VGG16 network is composed of 13 convolution layers and 3 fully connected layers, in addition to 5 pooling layers and 1 softmax layer. The network architecture of VGG16 is divided into six stages, as shown in Fig. 3(a). As shown in Fig. 3 (Fig. 4 is the gray map of Fig. 3), based on VGG16 network, HED network eliminates the last stage (the fifth pooling layer, three fully connected layers and softmax layer), and adds an edge output layer to output inheritance after convolution of the last layer of each stage, so as to gradually refine the edge mapping of different scales. HED network only needs to extract the image features, so it retains the previous convolution and pooling layers. The specific architecture of HED network is shown in Figure 3(b). In addition, due to the influence of pooling layer, from the second stage, the length and width of the input image in each stage are half of the input image in the previous stage (i.e., multi-scale description). Therefore, the images obtained in each stage need to be recalculated by using transposition convolution, which is equivalent to enlarging the length and width of the images extracted from the second stage to the fifth stage by 2 times, 4 times, 8 times and 16 times respectively (i.e., the step sizes are set to 1, 2, 4, 8 and 16 respectively), so that the images obtained at each scale have the same size. Finally, the outputs of each stage are connected to obtain the final result. In this paper, the DNN module of OpenCV is used to transform the pre trained HED model for edge detection. The convolution kernel size of HED network is 3*3, and the pooling type is mean pooling. The above parameters are used for edge detection to prepare for the calculation of similarity measurement in the later stage.

[096] Therefore, in this embodiment, the HED network is used for effective image edge detection. In this embodiment, SLIC superpixel algorithm is used to segment the original aerial remote sensing image, and the number of boundary pixels between adjacent superpixels in the intersection area is defined by HED network, which is denoted as E. The boundary line with larger gray value (brighter boundary line in the figure) indicates that the boundary between canopy areas is stronger (refer to Figure 5).

[097] At the same time, it is necessary to properly adjust the size of the input image so that the edge outline of the whole image can be displayed coherently. Increasing the size of the image properly is beneficial to display the dim edge outline inside the image. Therefore, the length I of the input image is adjusted to 200, 400, 600, 800, 1000, 1200 and 1400, respectively for edge detection (refer to Fig. 5), and the widths of seven images are calculated according to the aspect ratio of the original image. The seven detected images are represented as edge images E, , E ,

E 1 6 ,0E , 800 E 1 00 m, E 100 Then the edge detection results of the above seven

images are superimposed. In addition, the histogram equalization method is used to process the superimposed images, so as to increase the deviation of gray values of different boundaries, improve the contrast of gray images, and obtain the final edge detection results with more prominent contour features. This result is defined as HE, -

H(E 2 0 0 + E E 60+ I800+ +EI10 + E +E2o +E 1400 ), where H represents the process of histogram equalization on the superimposed image and the final result after histogram equalization. Figure 5 shows the aerial image edge detection results of three research areas by using HED network and histogram equalization method. Fig. 5(a), Fig. 5(b) and Fig. 5(c) show the final results of three forest plots Al, A2 and A3, respectively. Firstly, the input aerial images are adjusted, and the lengths are set to 200, 400, 600, 800, 1000, 1200 and 1400 respectively for edge detection. Secondly, the detected images are superimposed, and the boundary between canopy regions is extracted. The histogram equalization method is used to stretch the HED boundary map to improve the contrast of the image. The boundary with larger gray value (brighter boundary line in the figure) indicates that the boundary between canopy areas is stronger.

[098] (2.3) Calculation of similarity weights between superpixels;

[099] Firstly, SLIC segmentation algorithm is applied to image segmentation, which is used to generate superpixels as computing units and locally cluster images. This weakens the texture details of images to a certain extent, and blurs images while retaining important edges. A set of superpixels is obtained after segmentation, and a neighborhood connected graph G = (V, B) of superpixels is constructed based on the center point of each superpixel. A node corresponds to a superpixel area, which is a branch connecting two adjacent superpixels. The weight of represents the similarity weight between two adjacent super-pixel regions connected by the channel, and the expression is as follows:

[0100] w = awb+-4+yW3 (1); Wb

[0101] Here a, , y are weight coefficients, wl, w , w 1 are the three similarity

measures of adjacent superpixels.

[0102] wl is a measure of the difference between the RGB average values of two adjacent superpixels. Each pixel in the image has its own color space, which can be represented by RGB values between 0 and 255. In the pre-segmented image, each segmented super-pixel region contains a certain number of pixels. The average color value of each super-pixel area can be recorded as the color value of the super-pixel, so the calculation formula is as follows:

[0103] w (r - r + g- )2 + (b, - by,) 2 (2);

[0104] The smaller the value of w', the higher the color similarity between the two super-pixel regions, so the two regions should belong to the same crown.

[0105] w2 indicates the similarity measure of the number of intersecting pixels of the sum of two adjacent superpixel regions. In this embodiment, the area of two adjacent superpixels is expanded by one pixel, and the number of pixels in the overlapped part of the two adjacent areas after expansion is counted. The similarity measurement is expressed by w2, which is written as follows:

[0106] w2 = int ernumber(vi, vj) (3);

[0107] The larger the value ofw', the larger the number of intersecting pixels of two super-pixel regions, and the greater the probability that the two regions belong to the same crown.

[0108] w' is the similarity measure of crown edge generated by HED network in the intersection area of two adjacent superpixels vi and vj. In this embodiment, the

number of intersecting pixels between the intersecting boundaries of two adjacent superpixels is calculated, and the tree crown edge is calculated by the corresponding HED network, and the obtained values are as follows:

[0109] w3 = Edge(vi,v) lE, (4);

[0110] Edge(vi, vj) represents the intersection boundary of adjacent superpixels.

Indicates the number of boundary pixels in the intersection area of two adjacent superpixel areas defined by HED network. The smaller the value ofw3, the weaker the edge of the crown area between two superpixels, and the greater the probability that two superpixels are classified into the same crown area.

[0111] In addition, in order to ensure that the three similarity measurement indexes (i.e., w', w 2and w3 ) of adjacent superpixels in formula (1) have enough influence on the segmentation results, corresponding weight coefficients should be set to achieve the goal of balancing. According to the three similarity measurement indexes, three weight coefficients are determined. For example, if the sum is set to (3, 2, 0.003) in a certain operation, the weight (y) of the third part needs to be enlarged, while the weights of the first and second parts are appropriately reduced, that is, it is recommended that the three weight coefficients a, P, y can be set to (1, 1, 1000).

[0112] (2.4) Minimum spanning tree and superpixel merging;

[0113] In (2.3) the minimum spanning tree (MST) extracted from the connected graph G = (V, B) of superpixel neighborhood is used to generate the connected tree of aerial image. This topological method considers all the superpixels of the image in order to segment the image from a global perspective. Then, according to the similarity weight calculated above, the superpixels are merged to realize single tree crown segmentation. According to the adjacency relationship and attribute difference between regions, MSTs of each super-pixel region of aerial image is generated and connected on the super-pixel scale.

[0114] To sum up, the weight coefficients a, P, y are determined, and we can calculate wb. The results show that the smaller the w b is, the higher the similarity between two adjacent superpixels will be, and the two regions are more likely to be divided into the same crown. Then, according to the weight (similarity or difference between adjacent superpixels), the path set B of MST is sorted in non-reduced order. Finally, whether the pre-segmented image areas are merged can be judged by the defined criteria.

[0115] (2.4.1) Class differences within superpixels;

[0116] In this section, under the initialization condition, a superpixel is specified to represent an area. Intra-class differences are characterized by the maximum weighted path of local minimum spanning tree, which is defined as the maximum weight of a single region of MST. The expression is as follows:

[0117] Int(vi) = max wb (5); b(vi,vj)EMST(V,B)

[0118] The MST(V, B) is an MST composed of a channel set in a single area. For example, as shown in Fig. 6 (Fig. 7 is the gray image of Fig. 6), a superpixel neighborhood connected graph G (Fig. 6(a)) is constructed, and MST is extracted from the graph g (Fig. 6(b)). Fig. 6(a) is a superpixel neighborhood connected graph G constructed based on superpixel center points, wherein the center points are represented by circles (red circles), and the paths between two superpixel center points are represented by lines (blue lines); Fig. 6(b) extracts the minimum spanning tree from graph G, in which the path of MST is described by a line (yellow line), and the weight of each edge is marked by a number (blue number).

[0119] (2.4.2) Superpixel merging based on intra-class differences;

[0120] The intra-class difference between two adjacent superpixel regions is defined as follows:

[0121] D if (vi, vj) min wb (vi, vj) (6) Vi,VjEV,b(vi,vj)EB

[0122] Whether the image areas before segmentation are merged can be judged by the following criteria. The results show that when the two regions are similar and meet the merging conditions, that is, the difference between the two superpixel regions

Dif (vi, vj) is less than Dif(vi, vj), they are merged, otherwise they are not merged.

( true, Dif(vi,vj) < MInt(vi,vj) (

0v, V f) false , Dif(vi,vj) > MInt(vi,vj)

[0124] MInt(vi, vj) = min{Int(vi) + T(vi), Int(vj) + T( vj)} is the minimum value of intra-class differences. T(vi) is a threshold function, and the expression is

T(vi) = , and m is a constant parameter. When m is close to 0, it will lead to lvii excessive image segmentation. When m approaches infinity, the images will be merged into a large area. Therefore, the size of the constant m determines the size of the image segmentation sub-region. The larger m is, the larger the subregion will be, and vice versa. lvii indicates the total number of pixels contained in the superpixel area vi. Setting T( vi) to a smaller area that appears during merging and splitting. In addition, because the aerial image of forest contains a large number of pixels, when a single super-pixel area is large, that is, there are many pixels in a single super-pixel region, the influence can be ignored. Therefore, the aerial images combined by this method contain relatively uniform super-pixel regions.

[0125] Results;

[0126] The super-pixel segmentation results of aerial remote sensing images of three forest research plots are shown in Figure 8 (Figure 9 is the gray map of Figure 8). Fig. 8 shows the result of super-pixel segmentation of aerial remote sensing images of three research sites by using SLIC super-pixel algorithm, in which super-pixel areas are separated by sky blue lines. Fig. 8(a), Fig. 8(b) and Fig. 8(c) respectively show the segmentation results of the images of forest plots Al, A2 and A3. Through this step, the expression form of aerial remote sensing image of forest plot is simplified, that is, each sub-pixel in super-pixel area has similar color and brightness characteristics. Then, three similarity measurement indexes between two adjacent superpixels are defined, which are the difference of RGB color values, the number of intersecting pixels and the number of boundary pixels in the intersecting area defined by HED network, and the similarity weights are calculated according to these three standards. In addition, MSTs is extracted from the connected graph G at superpixel scale (refer to Fig. 10, and Fig. 11 is the gray scale of Fig. 10). Fig. 10 shows that MSTs is extracted from connected graph G based on the result of superpixel segmentation, in which the center of superpixel area is marked by a circle (red circle), MSTs path is represented by a line (yellow line), and the weight of each path is represented by numbers (pink numbers). Fig. 10(a), Fig. 10(b) and Fig. 10(c) show MSTs of forest plots Al, A2 and A3 respectively. Finally, the final result of single tree crown segmentation after superpixel merging is shown in Figure 12 (Figure 13 is the gray map of Figure 12). Fig. 12 is the result of single tree crown segmentation using the superpixel merging algorithm based on MST results, in which each tree crown area is segmented by lines (yellow lines). Fig. 12(a), Fig. 12(b) and Fig. 12(c) show the segmentation results of the images of forest plots Al, A2 and A3, respectively.

[0127] Table 1 below shows three important parameters (i.e., image size, number of pre-segmented superpixels and threshold constant) selected by the algorithm in this embodiment in aerial images of different forest plots. In addition, the quantitative evaluation of experimental results is shown in table 2, wherein TP(true positive) is the number of correctly detected trees, FN(false negative) is the number of undetected trees

(missing errors), and FP(false positive) is the number of trees that do not exist in the region but are added incorrectly (misclassification errors). r(recall) is the detection rate of the tree, p(precision) is the correct rate of the detected tree, and f is the overall precision of the detected tree. F is the harmonic average of the tree detection rate and the correct rate of the detected tree, which combines and balances r and p. Finally, the overall accuracies of tree crown segmentation of forest plots Al, A2 and A3 are 86%, 92% and 87%, respectively. The results show that this method has high accuracy for three aerial images.

[0128] Table 1. Three parameters selected by the algorithm for aerial images of three forest plots:

Forest plot number Image size Number of pre- m divided superpixels Al 960 *719 500 1000 A2 425*319 500 500 A3 370 *240 500 300

[0129] Table 2. Evaluation of tree segmentation accuracy in three forest plots;

Forest Number of trees TP FN FP r p f plot Algorithm number result/field measurement Al 301/258 240 18 61 0.93 0.80 0.86 A2 208/195 186 9 22 0.95 0.89 0.92 A3 138/125 114 11 24 0.91 0.83 0.87

[0130] 4. Comparison with existing methods;

[0131] The method of this embodiment is novel, because it combines SLIC superpixel algorithm, HED network and MST which uses topological method to segment single crown of aerial image, it is different from some existing single crown segmentation methods, such as circular Hough transform combining canny edge detection method and multi-scale filtering and segmentation (MSF) method. In this method, canny edge detection algorithm is used to extract canopy boundary from the same tree species, and circular Hough transform algorithm is used to obtain the circular shape in the image, so as to identify a single crown. The MFS method is generated by multi-scale analysis. Multi-scale representation of the image, then it adopts watershed segmentation method to generate multi-scale segmentation map, in order to obtain accurate forest canopy segmentation boundaries, and effectively reduce the phenomenon of over-segmentation. However, the circular Hough transform method is limited to tree species with regular shape (such as citrus). This method only works well on plantations with the same afforestation methods and similar tree crown phenotypes, that is, the trees are arranged neatly and the planting distance is fixed (low stand density). The crown shape is roughly round, which is not suitable for natural forest stands composed of various tree species with irregular crown shape. The segmentation results of MFS method are easily interfered by shadows under the irradiation of solar radiation with fixed zenith angle. Because the segmentation boundary of the filtered image is refined in the process of multi-scale filtering, the computational complexity of MFS method is high. In addition, many oblique branches of a larger tree may be mistaken for a small crown. In this embodiment, the circular Hough transform algorithm, the MFS method and the method proposed by this technology are used to test the aerial images of three study sites Al, A2 and A3, of the same forest plot. The accuracy of the three methods for tree crown segmentation is shown in Table 3. Due to the different aerial photography angles and the limitation of the biological characteristics of the crown itself, the overall segmentation accuracy of the circular Hough transform method is low. Because the crown texture changes with the uneven sunlight, shading and different shooting angles, the accuracy of MFS method decreases, and there is no effective fusion strategy of crown parts. In contrast, the segmentation accuracy of aerial images of three forest plots has been improved in general.

[0132] Table 3 shows the accuracy comparison of single tree crown segmentation by using circular Hough transform method, multi-scale filtering and segmentation (MFS) method and the technical method for aerial images of three forest plot research points Al, A2 and A3:

Method Forest Number of TP FN FP r p f plot trees number Algorithm result/field measurement Circular Al 211/258 162 96 49 0.63 0.77 0.69 Hough A2 173/195 138 57 35 0.71 0.80 0.75 transform method A3 114/125 87 38 27 0.70 0.76 0.73 MFS Al 312/258 223 33 87 0.87 0.72 0.79 method A2 213/195 174 21 38 0.89 0.82 0.85 A3 140/125 106 19 34 0.85 0.76 0.80 The Al 301/258 240 18 61 0.93 0.80 0.86 technical A2 208/195 186 nine 22 0.95 0.89 0.92 method ________

A3 138/125 114 11 24 0.91 0.83 0.87

[0133] TP: the number of correctly detected trees. FP: the number of extra trees that do not exist in the region (misclassification error). FN: the number of undetected trees (omission error). r: tree detection rate. p: correctness of the checked-out tree. f: the overall accuracy of the detected tree.

[0134] This method transforms pixels into superpixels, and enhances the segmentation boundary with HED network, which can simply and effectively improve the segmentation accuracy. Specifically, SLIC is one of the best super-pixel segmentation methods at present, which has good flexibility, compactness and good noise resistance when generating superpixels. It improves the segmentation speed and performance of single tree crown segmentation algorithm. In addition, SLIC algorithm can artificially control the number of superpixels generated and convert pixels into superpixels for processing, so as to segment the ITC faster and more effectively while maintaining the crown contour. HED network is one of the most advanced edge detection methods at present. This method solves the problem of object boundary acquisition and ambiguity, and obtains the detection results obviously superior to other edge detection methods, such as edge detection methods based on robert operator, sobel operator, prewitt operator, log operator and canny operator. Applying HED network to canopy segmentation can extract clear and coherent canopy boundary, which lays the foundation for more accurate canopy segmentation. At the same time, this technology draws lessons from the efficient image segmentation algorithm based on graph (EGBIS) in order to further merge superpixels by MST method. MST method not only improves the tree crown segmentation efficiency of aerial survey images of forest land by reducing the frequency of super-pixel fusion, but also solves the problem that the typical image segmentation methods are difficult to determine the appropriate image segmentation scale. In a word, the method proposed by this technology has the best segmentation effect on the stand structure features with regular distribution and clear boundary, and the overall accuracy of the detected tree can reach to 92%.

[0135] If there is a need to apply this technical method, the following parameters should be considered:

[0136] Firstly, SLIC algorithm can generate compact and unified superpixels. Because it only needs to determine the number of pre-segmented superpixels, which usually depends on the number of tree crowns and pixels in aerial images, it is convenient to use. If the set number of superpixels is too small, the image segmentation may be insufficient and uneven; If the parameter setting is too large, the image may be over-divided. According to the field calculation, the number of trees in the three forest plots is 258, 195 and 125, respectively. In this embodiment, it is hoped that each large crown will be divided into 3-4 superpixels. Each small tree crown is divided into 2-3 superpixels. Each aerial image of the three forest plots contains more than 80,000 pixels and more than 100 trees on average. Therefore, according to the above criteria, the number of pre-segmented superpixels k was set as 500 in this embodiment. The results show that the crown segmentation method is feasible, and the phenomenon of over segmentation is obviously reduced.

[0137] Secondly, when using similarity weights for super-pixel fusion, the key problem is to set appropriate threshold parameters according to the resolution and size of aerial images. The value ofmin the threshold function-c(vi) is highly correlated with the regional merging of adjacent superpixels. The size of merging region increases with the increase of m value. Therefore, in order to accurately identify the forest canopy, we can reduce the under-segmentation and over-segmentation by choosing the appropriate m value. In low-resolution (100*100) images, the value range of m is usually 100 to 200; In a high-resolution (300*300) image, m usually ranges from 200 to 300; In images with high resolution (900*900), m = 1000 or other approximate values are generally selected. The aerial images of forest plots Al, A2 and A3 in Fig. 1 are 960*719 (high resolution), 425*319 and 370*240, respectively. Therefore, in this embodiment, m = 1000, m = 500 and m = 300 were set for superpixel merging.

[0138] In addition, Table 2 analyzes the ITC segmentation results of forest plots Al, A2 and A3 from three aspects: detection rate of the three, correct rate of detected trees and overall accuracy of detected trees. The results show that the overall segmentation accuracy of forest plots A2 and A3 of single tree species is higher than that of forest plots Al of mixed forest. The number of trees detected by this algorithm is 301, 208 and 138 respectively, while the corresponding number of trees measured on the spot is 258, 195 and 125 respectively. The results show that the algorithm has the best segmentation effect on forest plot A2, and the overall accuracy rate is 92%. The forest plot A2 is an artificially managed Pinus massoniana forest with the same tree species and similar crown size. Evenly distributed Pinus massoniana has obvious crown characteristics and regular stand structure characteristics, which is easy to realize single tree crown identification and segmentation. The tree crown size of forest plot A3 is different, which is affected by the sunlight in the upper right corner, and it increases the difficulty of tree crown segmentation by this algorithm. Therefore, the single tree crown segmentation accuracy of forest plot A3 is only 87%. Dead trees and healthy trees grow together in forest plot Al. The color and texture characteristics of dead trees and deciduous trees are difficult to distinguish, and the lateral branches of deciduous trees are often mistaken for a single crown, which easily leads to excessive segmentation. Therefore, the stand structure characteristics of forest plot Al make the process of tree crown segmentation complicated, and the accuracy rate is only 86%. Here, this embodiment suggests that the parameters of relatively complex mixed forest can be adjusted appropriately according to the characteristics of tree species or morphological structure, so as to increase the applicability of this technical method.

[0139] In a word, the method proposed in this embodiment has a good application prospect for tree crown segmentation of forest aerial images, it provides a novel concept based on image processing technology to adapt to different types of forests.

[0140] To sum up, this embodiment proposes a single tree crown segmentation method that combines SLIC superpixel segmentation algorithm, HED network and MST method in topology, which can accurately and effectively segment forest crowns in aerial remote sensing images. Firstly, the aerial remote sensing image is segmented by superpixels based on color measurement, and the coronal boundary is obtained by HED network. At the same time, three indexes, namely RGB color value, the number of intersecting pixels and the difference of the number of boundary pixels defined by HED network in the intersecting area, are used to measure the similarity weight between two adjacent superpixels. Finally, MST method is used to generate the connected tree of aerial images at superpixel scale, and the superpixels are fused according to the calculated similarity weights, so as to realize single tree crown segmentation.

[0141] The research results show that this method has a good application prospect, and the experimental results are encouraging. The overall segmentation accuracy rates of the three forest plots are 86%, 92% and 87%, respectively. Compared with the circular Hough transform method and multi-scale filtering segmentation method, the detection accuracy of this method is improved by 7%-17%, and the detection error is reduced by 3%-10 % . The tree characteristics obtained by this method provide comprehensive and necessary information for various forest applications.

[0142] Although the invention has been described with reference to specific examples, it will be appreciated by those skilled in the art that the invention may be embodied in many other forms, in keeping with the broad principles and the spirit of the invention described herein.

[0143] The present invention and the described embodiments specifically include the best method known to the applicant of performing the invention. The present invention and the described preferred embodiments specifically include at least one feature that is industrially applicable

Claims (4)

THE CLAIMS DEFINING THE INVENTION ARE AS FOLLOWS:

1. The single tree crown segmentation algorithm based on superpixel and topological features in aerial images, characterized by comprising the following steps:

(1) SLIC super-pixel segmentation is performed on the original aerial image, and simultaneously the crown boundary of the image is obtained by using HED network;

(2) Calculating three similarity measurement indexes between two adjacent superpixels, i.e., the difference between RGB average values of two adjacent superpixels, the number of intersecting pixels of two adjacent superpixels and the number of boundary pixels obtained from HED network, which are used to construct similarity weights between two adjacent superpixels;

(3) Constructing a superpixel neighborhood connected graph based on the center point of each superpixel, extracting the minimum spanning tree from the superpixel neighborhood connected graph to generate the connected tree of aerial images, and merging the superpixels according to the calculated similarity weights to realize the ITC segmentation.

2. The individual tree crown segmentation algorithm based on aerial images using superpixel and topological features according to claim 1, characterized in that (1) further comprises the following content:

CCD camera and optical camera are used to collect aerial images of the study area.

3. The individual tree crown segmentation algorithm based on aerial images using superpixel and topological features according to claim 1, characterized in that (2) comprises the following content:

A superpixel neighborhood connected graph G = (V, B) is constructed based on the center point of each superpixel, and the node vi E V corresponds to a superpixel region, b(vi, vj) E B is the branch connecting two adjacent superpixels, and the weight

of wb(vi,vj) represents the similarity weight between the two adjacent superpixel regions vi and vj connected by the branch b(vi, vj), and the expression of similarity weight is as follows:

Wb aWbj,+ wb+ yw, (1);

Where a, P, y are weight coefficients, w1, w2, w' are three similarity measures between adjacent superpixels.

wj represents the similarity measurement of the difference between RGB mean values of the two adjacent superpixels vi and vj, and the color mean value of each

superpixel area is recorded as the color value of the corresponding superpixel, and the color value of the superpixel area vi is recorded as (rv,, gv,, b,), the color value of the

superpixel area vj is recorded as (rvj, gj, bv,). The calculation formula of wl is as

follows:

w (r - b +(bv- bv,)z (2)

wb2 represents the similarity measurement of the number of intersecting pixels

of the two adjacent superpixel regions vi and vj. The calculation formula ofw2 is as

follows:

wb = internumber(vi,vj) (3)

wb represents the similarity measurement of crown edge generated by HED network in the intersection area of two adjacent superpixel vi and vj, the calculation

formula ofw3 is expressed as follows:

wb = Edge(vi,v)fE, (4)

Edge(vi, vj) represents the intersecting boundaries of the two adjacent super

pixel regions, and E, represents the number of boundary pixels in the intersected regions of two adjacent super-pixel defined by HED network.

4. The individual tree crown segmentation algorithm based on aerial images using superpixel and topological features according to claim 1, characterized in that (3) comprises the following content:

The maximum weighted branch of the local minimum spanning tree is defined as the maximum weight of the MST in the individual region. The expression is as follows:

Int(vi) max wb (5) b(vi,vj)EMST(V,B)

MST(V,B) represents the minimum spanning tree composed of the set of branches in a single superpixel region;

The intra-class difference between two adjacent superpixel regions vi and v

is defined as follows:

D if (vi, vj) min wb(vi, vj) (6) vi,vjEV,b(vi,vj)EB

When the difference between two adjacent superpixel areas vi and vj is less

than MInt(vi, vj), merging the two superpixels, otherwise, the superpixels are not

merged:

(g( true, Dif(vi,vj) < MInt(vi,vj) (

Meg1V = false , Dif(vi,v) > MInt(vi,vy)

MInt(vi, v1 ) = min(Int(vi) + T(vi), Int(vj) +T(vj)} is the minimum value of intra-class differences; T( vi) is a threshold function, the expression is T( vi) -, and m is a constant parameter. Iv~i