CN117036370A - Plant organ point cloud segmentation method based on attention mechanism and graph convolution - Google Patents

Abstract

A plant organ point cloud segmentation method based on an attention mechanism and graph convolution belongs to the technical field of three-dimensional point cloud instance segmentation. The method comprises the steps of dividing a network TRGCN based on a point attention mechanism and a double-branch parallel example of space diagram convolution, directly inputting a three-dimensional point cloud, respectively focusing on local feature extraction and global feature extraction by double branches, and fusing the two features through a T-G feature coupling layer. The method takes five plant point cloud data of tomatoes, corns, tobaccos, sorghum and wheat as research objects, and the dual-branch parallel neural network architecture TRGCN can capture local characteristics and global characteristics of the point cloud at the same time, is used for training a high-robustness example segmentation model, can improve the segmentation precision of the plant point cloud, has good generalization capability, and can provide good data support for rapid, efficient and accurate plant phenotype analysis.

Description

A plant organ point cloud segmentation method based on attention mechanism and graph convolution

Technical field

The invention belongs to the technical field of three-dimensional point cloud instance segmentation, and specifically relates to a dual-branch parallel plant organ point cloud segmentation method based on attention mechanism and spatial graph convolution.

Background technique

With the popularization of lidar equipment and the emergence of various consumer-grade depth sensors, point cloud data is increasingly used in various fields, such as robotics, autonomous driving, and urban planning. In phenotypic analysis research, three-dimensional point cloud, as a low-resolution representation of the real world, has become the most direct and effective data form for studying plant structure and morphology. Many studies have used plant three-dimensional structures to segment organs, monitor growth, and evaluate varieties. Points in a three-dimensional coordinate system, as the most basic units that constitute point clouds, are similar to pixels in two-dimensional pictures, but can accommodate more high-dimensional semantic information. In phenotypic research, the morphological structure of plant organs is an intuitive and important trait, which can reflect the plant's ability to adapt to external conditions and growth, such as photosynthesis efficiency and water absorption efficiency. Plant organ point cloud segmentation refers to the process of semantically dividing plants according to different organs (such as stems, leaves, fruits, etc.). It is the basis for subsequent in-depth understanding of point cloud data and is of great significance for understanding the functional structure of plants. It is currently a challenging research direction.

The traditional plant point cloud segmentation algorithm requires manual feature description in advance. This segmentation process is complex and cumbersome. With the advent of the big data era, traditional processing methods cannot meet the needs of fast and accurate analysis. Therefore, there is an increasing need for automated segmentation methods. With the rapid growth of computer graphics processor performance, deep learning, as the leading technology of artificial intelligence, has been successfully used to solve various two-dimensional vision problems. However, due to the disorder and complexity of point clouds in spatial aggregation, the application of deep learning methods on point clouds still faces many challenges. Convolutional neural networks (CNN), which have good performance in visual segmentation tasks, extract features through shared kernel convolution and improve model efficiency. The inherent translation invariance of CNN makes it more accurate to control local features. However, the receptive field of CNN itself is usually relatively small, the ability to capture global features is relatively weak, and it cannot directly act on the original point cloud data. Another neural network structure applied to point cloud data is the graph convolutional network (GCN), which treats each independent point in the point cloud as a vertex in the graph data structure, which can be directly implemented in the point cloud data. Convolution-like operations to extract local features. Transformer, which has outstanding performance in the field of natural language processing, can also capture global features well, and its core idea attention mechanism is also very suitable for processing point cloud data. These deep learning methods have achieved satisfactory segmentation results on many public point cloud data sets, demonstrating the effectiveness of deep learning methods for point cloud data segmentation.

However, relatively speaking, the structural complexity of plant point clouds results in a greater amount of semantic information that needs to be recognized in the organ segmentation task. When obtaining point clouds, the occlusion problem between leaves often causes part of the point cloud to be lost, resulting in holes and sparse problems. In addition, the similarity between plant organs is very high, and different leaf instances often have the same characteristics such as color, morphological structure, and texture. This highly repetitive feature is not friendly to the learning of neural networks. Finally, different species of plants have different geometric morphological characteristics. Even the phenotypic characteristics of plants of the same species in different growth environments are different, and may even be very different, which requires high generalization capabilities of the network. In summary, the current segmentation accuracy of plant point clouds cannot meet the requirements.

Contents of the invention

The purpose of this invention is to solve the problem of accurate organ segmentation in complex plant point clouds and provide a two-branch parallel plant organ point cloud segmentation method based on attention mechanism and spatial graph convolution. This method is a three-dimensional plant point cloud with complex structure. The cloud provides a reliable and efficient organ segmentation method. Taking five plant point cloud data of tomato, corn, tobacco, sorghum, and wheat as the research object, a dual-branch parallel neural network architecture is newly designed based on the attention mechanism and spatial graph convolution. TRGCN can simultaneously capture local features and global features of point clouds and is used to train highly robust instance segmentation models. It can improve the segmentation accuracy of plant point clouds and provide data support for fast, efficient, and accurate plant phenotypic analysis.

In order to achieve the above objects, the technical solutions adopted by the present invention are as follows:

A plant organ point cloud segmentation method based on attention mechanism and graph convolution. The method is:

Step 1: The feature encoder takes the original point cloud as input, uses a multi-layer perceptron to map the features to a high-dimensional space, and uses the point cloud attention mechanism to initially extract features, and then inputs the initial feature data into the TRGCN block. This module can cascade superposition to continuously deepen the understanding of high-dimensional features; the feature aggregation layer in the TRGCN block extracts neighborhood features while downsampling point clouds, and then enters the dual-branch parallel network part, which is composed of spatial graph convolution The local feature capture branch composed of the local feature capture branch and the global feature learning branch composed of the point attention mechanism finally input the feature data into the T-G feature coupling layer to obtain the target number of point clouds and corresponding high-dimensional abstract features. The encoder part superimposes the TRGCN block , extract high-dimensional feature information from the original plant point cloud for segmentation tasks;

Step 2: The feature decoder part also stacks three cascaded TRGCN blocks and receives the output of the three TRGCN blocks in the encoder respectively, but replaces the feature aggregation layer with an interpolation layer. The interpolation layer restores the features of the high-dimensional point set to Low-dimensional points are concentrated, but the grouping results of the K nearest neighbor algorithm are still output for calculation by the two branches of TRGCN; for segmentation result prediction, the decoder sets an independent interpolation layer after the TRGCN block and uses a single point attention layer to ensure Information integrity, the network finally uses a multi-layer perceptron to output the segmentation results of the point cloud;

Step 3: Network training: All experiments in this study were conducted on an independent server equipped with a 12-core 20-thread CPU, 64GB memory, and an Nvidia GeForce RTX 3090Ti GPU; an independent server was used for neural network training. During the training phase, all The plant point cloud segmentation model uses the same hyperparameters. The hyperparameters are specifically: the training batch size is set to 32, the initial learning rate is set to 0.001, the Adam method is used to optimize the network, and a total of 100 cycles are trained, every 20 The learning rate is halved every cycle, the weight decay is set to 0.0001, the momentum is set to 0.9, the K value of the K nearest neighbor algorithm is set to 12, and the feature dimension of the point attention layer is set to 256.

Further, the step one is specifically:

(1) Feature aggregation layer

The specific process of feature aggregation is: input x points with feature dimension, first randomly sample the farthest point, then use the K nearest neighbor algorithm to group the point cloud, input it into the multi-layer perceptron to aggregate the neighbor point features to the center point, and finally The maximum pooling operation is used to calculate y points with feature' dimension characteristics;

The feature aggregation layer uses the K nearest neighbor algorithm to sample and group the input point set; the feature aggregation layer outputs the calculated K nearest neighbor matrix and shares it with subsequent parallel branches;

(2) Local feature capture branch

This branch is built based on dynamic spatial graph convolution and is used to extract local features in the input plant point cloud; first, a feature map G = (V, E) is constructed based on the point set V and neighbor information E, and edge convolution is used to extract the local features in the input plant point cloud. Feature extraction is performed on the feature space of ; the formula for extracting features of a certain point x _i is:

f _i =? h(x _i ,y _i )

Among them, x _j is one of the neighbor points of point _xi ,? and h respectively represent a certain aggregation function and a certain relational operation; that is, a relational operation is used to aggregate the features of neighbor points around the candidate point, that is, the characteristic information of the candidate point is obtained. The relational operation is defined as edge convolution;

Maximum pooling is used as the aggregation function. The specific process is:

conv _i =Max(MLP(h( _xi , _xi -x _j )))

The relational operation h is defined as the linear combination between the feature differences of point x _i , x _i and its neighbor point x _j and the output value of point x _i ;

(3) Global feature learning branch

The vector attention mechanism in the local neighborhood is used to extract features. The calculation formula is:

Among them, x _j is one of the K neighbor points of point x _i , In this study, it is defined as the difference between the neighborhood point and the attention point, φ, α is a point-level feature transformation method, which is obtained from Q, K, and V in the attention mechanism (Query, Key, and Value respectively, which are proper nouns in the attention mechanism, corresponding to query, key, and value in Chinese) value, δ is the position encoding function. Based on the above attention mechanism, a point attention layer is proposed. The improved calculation formula is:

(4)T-G characteristic coupling layer

After the above processing, two feature matrices with identical dimensions and shapes are obtained: a matrix G with significant local features and a matrix T with complete global features; G and T are spliced and input into the feature coupling layer to obtain the target feature matrix:

TG＝Linear(ReLU(Linear(T,G)))

The T-G feature coupling layer is designed using two linear layers and a ReLU activation layer, allowing the network to learn the more important information of the two matrices and combine them into the target feature matrix.

The beneficial effects of the present invention compared to the existing technology are: the present invention designs a brand new dual-branch parallel instance segmentation network TRGCN based on point attention mechanism and spatial graph convolution, which directly inputs the three-dimensional point cloud, and the dual branches focus on local features respectively. extraction and global feature extraction, and fuse the two features through the T-G feature coupling layer. The results show that TRGCN has achieved excellent performance on different plant point clouds, with an accuracy higher than other mainstream point cloud segmentation networks. It has good generalization ability and can provide fast, efficient and accurate plant phenotype analysis. Good data support.

Description of the drawings

Figure 1 is a network architecture diagram of the TRGCN of the present invention;

Figure 2 is a structural diagram of the TRGCN block of the present invention;

Figure 3 is an architectural diagram of the global feature learning layer of the TRGCN block of the present invention;

Figure 4 is a diagram showing the segmentation results of five plant point clouds according to the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below in conjunction with specific embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and do not limit the scope of the present invention.

Example 1:

Based on the point cloud self-attention mechanism and spatial graph convolution, this research innovatively proposes a dual-branch parallel network Transformer Graph Convolution Network (TRGCN) designed using an encoder-decoder architecture (Figure 1).

The feature encoder takes the original point cloud as input, uses a multi-layer perceptron to map features to a high-dimensional space (default is 32 dimensions), and uses a point cloud attention mechanism to initially extract features. The initial feature data is then input into the TRGCN module (Figure 2), which can be superimposed in multiple cascades to continuously deepen the understanding of high-dimensional features. Specifically, the feature aggregation layer in the TRGCN block extracts neighborhood features while downsampling the point cloud, and then enters the dual-branch parallel network part, which is the local feature capture branch composed of spatial graph convolution and the point attention mechanism. The global feature learning branch. Finally, the feature data is input into the specially designed T-G feature coupling layer to obtain the target number of point clouds and corresponding high-dimensional abstract features. The encoder part extracts high-dimensional feature information from the original plant point cloud by superimposing TRGCN blocks for segmentation tasks.

(1) Feature aggregation layer

The function of the feature aggregation layer in the TRGCN block is to reduce the input point set base and abstract high-dimensional feature vectors during the stacking process of multiple modules. For example, from the original input to the first TRGCN block, the number of points is reduced from N to N/4, and the point cloud feature dimension is increased from F to 2F;

The specific process of feature aggregation is as follows: input x points with feature dimension, first randomly sample the farthest point, then use the K nearest neighbor algorithm to group the point cloud, input it into the multi-layer perceptron to aggregate the neighbor point features to the center point, and finally The maximum pooling operation is used to calculate y points with feature' dimension characteristics (default y=x/4, feature'=2*feature).

The feature aggregation layer uses the K nearest neighbor algorithm to sample and group the input point set. Moreover, in order to save video memory space during training, this layer will output the calculated K nearest neighbor matrix and share it with subsequent parallel branches.

(2) Local feature capture branch

This branch is built based on dynamic spatial graph convolution and is used to extract local features in the input plant point cloud. First, a feature map G = (V, E) is constructed based on the point set V and neighbor information E, and edge convolution is used to extract features on the input feature space. The formula for extracting features of a certain point x _i is as follows:

f _i =? h(x _i ,y _i )

Among them, x _j represents one of the neighbor points of point _xi ,? and h represent some kind of aggregate function and some kind of relational operation respectively. That is, a relational operation is used to aggregate the features of neighbor points around the candidate point to obtain the feature information of the candidate point. This relational operation is defined as edge convolution. In order to enhance the understanding of local features in point clouds, this study uses max pooling as the aggregation function. The specific process is as follows:

conv _i =Max(MLP(h( _xi , _xi -x _j )))

The relational operation h is defined as the linear combination between the characteristic difference between point x _i x _i and its neighbor point x _j and the output value of point x _i . This choice not only retains the characteristics of local point sets that influence each other, but also partially takes into account the overall global characteristics.

(3) Global feature learning branch

As shown in Figure 3, this branch is built based on the point cloud attention mechanism and is very suitable for processing point cloud data. In essence, point cloud data can be regarded as word vectors embedded in the attention space. This study uses Vector attention mechanism is used to extract features. The calculation formula is as follows:

Among them, x _j is one of the K neighbor points of point x _i , φ, α is a point-level feature transformation method, which is obtained from the Q, K, and V values in the attention mechanism respectively. δ is the position encoding function, ρ is the regularization function, γ is the mapping function, and β is some kind of relational operation. In this paper In the study, it is defined as the difference between the neighborhood point and the attention point. Based on the above attention mechanism, this study proposes a point attention layer. The improved calculation formula is as follows:

Different from the ordinary attention mechanism, position coding is also added to the α function to enhance the understanding of features. Based on the point attention layer, the TRGCN encoder constructs a residual structure in the global feature learning branch. Add a linear layer before and after the point attention layer, and conduct residual connection between the final output and the input to promote information exchange, accelerate network convergence, and provide the possibility to train deep networks.

(4)T-G characteristic coupling layer

After the above processing, two feature matrices with exactly the same dimensions and shapes can be obtained: a matrix G with significant local features and a matrix T with complete global features. After splicing G and T, input into the feature coupling layer to obtain the target feature matrix:

TG＝Linear(ReLU(Linear(T,G)))

The T-G feature coupling layer is designed using two linear layers and a ReLU activation layer, so that the network can learn the more important information of the two matrices and combine them into the target feature matrix.

In summary, the feature encoder part of the TRGCN network can design models adapted to different visual tasks by changing the stacking number of TRGCN blocks. Fewer TRGCN blocks can be used for lightweight classification networks, while more cascaded TRGCN blocks can be used for finer-grained tasks such as point cloud segmentation and object recognition.

The feature decoder part also stacks three cascaded TRGCN blocks and receives the outputs of the three TRGCN blocks in the encoder respectively, but replaces the feature aggregation layer with an interpolation layer. Contrary to the feature aggregation layer, the interpolation layer in the decoder restores the features of the high-dimensional point set to the low-dimensional point set, but still outputs the grouping results of the K nearest neighbor algorithm for the calculation of the two branches of TRGCN. For instance segmentation result prediction, the decoder sets an independent interpolation layer after the TRGCN block and uses a single point attention layer to ensure information integrity. The network finally uses a multi-layer perceptron to output the segmentation result of the point cloud.

Network training. All experiments in this study were conducted on a dedicated server equipped with a 12-core 20-thread CPU, 64GB memory, and an Nvidia GeForce RTX 3090Ti GPU. In the training phase, the five plant point cloud segmentation models use the same hyperparameters, specifically: the training batch size is set to 32, the initial learning rate is set to 0.001, the Adam method is used to optimize the network, and a total of 100 cycles are trained. The learning rate is halved for 20 epochs, the weight decay is set to 0.0001, the momentum is set to 0.9, the K value of the K nearest neighbor algorithm is set to 12, and the feature dimension of the point attention layer is set to 256.

The organ instance segmentation test was conducted on five kinds of plant point cloud data, and the highest average intersection and union ratio of 86.38% and an average accuracy of 88.58% were achieved. In order to verify the segmentation ability of TRGCN, three mainstream point cloud segmentation networks were selected for comparison with TRGCN. Among the 5 segmentation tasks, TRGCN has 9 indicators leading the other three methods, and has achieved the best results in most segmentation tasks. The optimal accuracy, especially the improvement in accuracy on sorghum leaves, is more obvious, which shows that TRGCN is better at processing monocot point clouds. Since the canopy structure of dicotyledonous crops is relatively crowded and can easily cause occlusion problems, the segmentation effect of tobacco and tomato point clouds is not as good as that of monocot crops, but the segmentation effect is still better than the other three segmentation networks. The specific test results are shown in Table 1. Figure 4 is a segmentation rendering of five plant point clouds.

This invention also uses sorghum point cloud as the research object to discuss the stacking number of TRGCN pooling layers and cascaded TRGCN blocks. The results show that the network segmentation performance using maximum pooling is the best, and the accuracy is better than average pooling and summation pooling. The chemical is about 2% higher. When the number of cascaded TRGCN blocks is 3, the network achieves optimal training time and segmentation effect, sacrificing a certain amount of time to obtain higher segmentation accuracy. The specific test results are shown in Tables 2 and 3.

Table 1 is a comparison table of segmentation accuracy between the TRGCN network of the present invention and other mainstream networks.

Pooling layer Training time (seconds) Average intersection ratio (%) Average accuracy (%) max pooling 2082 78.9292 84.9198 average pooling 2085 75.7748 80.6104 sum pooling 2086 76.3709 82.9647

Table 2 Ablation experiment 1 of the present invention, segmentation effect table of different pooling layers

TRGCN block number Training time (seconds) Average intersection ratio (%) 2 1728 73.7120 3 2082 78.9292 4 2202 75.5498

Table 3: Ablation experiment 2 of the present invention, segmentation effect table with different stacking numbers of TRGCN blocks.

Claims (2)

1. A plant organ point cloud segmentation method based on an attention mechanism and graph convolution is characterized by comprising the following steps of: the method comprises the following steps:

step one: the feature encoder takes an original point cloud as input, adopts a multi-layer perceptron to map features to a high-dimensional space, adopts a point cloud attention mechanism to extract the features preliminarily, and then inputs initial feature data into a TRGCN block, and the module can be used for cascade superposition to deepen understanding of the high-dimensional features; the feature aggregation layer in the TRGCN block extracts neighborhood features and downsamples point clouds at the same time, then enters a double-branch parallel network part, is a local feature capturing branch formed by space graph convolution and a global feature learning branch formed by a point attention mechanism respectively, and finally inputs feature data into the T-G feature coupling layer to obtain target number of point clouds and corresponding high-dimensional abstract features;

step two: the feature decoder part stacks three cascaded TRGCN blocks, and receives the outputs of the three TRGCN blocks in the encoder respectively, but replaces the feature aggregation layer with the interpolation layer, and the interpolation layer restores the features of the high-dimensional point set to the low-dimensional point set, and still outputs the grouping result of the K nearest neighbor algorithm for two branches of the TRGCN to calculate; for segmentation result prediction, the decoder sets an independent interpolation layer behind the TRGCN block, adopts a single point attention layer to ensure information integrity, and finally adopts a multi-layer perceptron to output a segmentation result of point cloud;

step three: training a network: a CPU with 12 cores and 20 threads, a 64GB memory and a Nvidia GeForce RTX 3090Ti GPU are arranged on an independent server; the neural network training is carried out by using an independent server, and in the training stage, all plant point cloud segmentation models adopt the same super parameters, wherein the super parameters are specifically as follows: training batch size was set to 32, initial learning rate was set to 0.001, the network was optimized using Adam method for 100 cycles, learning rate was halved every 20 cycles, weight decay was set to 0.0001, momentum was set to 0.9, K value of K nearest neighbor algorithm was set to 12, and feature dimension of point attention layer was set to 256.

2. The method for segmenting the plant organ point cloud based on the attention mechanism and the graph convolution according to claim 1, wherein the method comprises the following steps of: the first step is specifically as follows:

(1) Feature polymeric layer

The specific process of feature polymerization is as follows: inputting x points with feature dimension, firstly sampling the points at the most distant point by using a random, grouping the point clouds by adopting a K nearest neighbor algorithm, inputting the point clouds into a multi-layer perceptron to aggregate the neighbor point features to a central point, and finally obtaining y points with feature' dimension features by adopting a maximum pooling operation;

the characteristic aggregation layer adopts a K nearest neighbor algorithm to sample and group the input point set; the feature aggregation layer outputs the calculated K neighbor matrix and shares the K neighbor matrix with the subsequent parallel branches;

(2) Local feature capture branching

The branch is constructed based on a dynamic space diagram convolution and is used for extracting local features in an input plant point cloud; firstly, constructing a feature graph G= (V, E) based on a point set V and neighbor information E, and carrying out feature extraction on an input feature space by adopting edge convolution; extracting a certain point x _i The formula of the characteristics is as follows:

f _i ＝？h(x _i ,y _i )

wherein x is _j For point x _i Is one of the neighbor points? And h represents a certain aggregation function and a certain relational operation, respectively; the method comprises the steps that neighbor point features around candidate points are aggregated through a relation operation, so that feature information of the candidate points is obtained, and the relation operation is defined as edge convolution;

the maximum pooling is adopted as an aggregation function, and the specific process is as follows:

conv _i ＝Max(MLP(h(x _i ,x _i -x _j )))

the relational operation h is defined as point x _i ，x _i And its neighbor point x _j Feature difference value and point x of (2) _i Linear combinations between the output values;

(3) Global feature learning branching

The feature is extracted by adopting a vector attention mechanism in a local neighborhood, and the calculation formula is as follows:

wherein x is _j Is the point x _i X is an independent point set in each single plant point cloud, ρ is a regularization function, γ is a mapping function, β is a difference between a neighborhood point and a point of interest, φ,Alpha is a characteristic transformation method of point level, Q, K, V values in a self-attention mechanism are respectively obtained, delta is a position coding function, a point attention layer is provided according to the attention mechanism, and an improved calculation formula is as follows:

(4) T-G feature coupling layer

Through the processing, the feature matrix with two dimensions and identical shapes is obtained: a matrix G with significant local features and a matrix T with complete global features; and (3) inputting the spliced G and T into a feature coupling layer to obtain a target feature matrix:

TG＝Linear(ReLU(Linear(T,G)))

the T-G characteristic coupling layer is designed by adopting two linear layers and one ReLU activation layer, so that the network can learn more important information of each of the two parts of matrixes and combine the two parts of matrixes into a target characteristic matrix.