CN109410238B  Wolfberry identification and counting method based on PointNet + + network  Google Patents
Wolfberry identification and counting method based on PointNet + + network Download PDFInfo
 Publication number
 CN109410238B CN109410238B CN201811098583.0A CN201811098583A CN109410238B CN 109410238 B CN109410238 B CN 109410238B CN 201811098583 A CN201811098583 A CN 201811098583A CN 109410238 B CN109410238 B CN 109410238B
 Authority
 CN
 China
 Prior art keywords
 pointnet
 point
 layer
 point cloud
 network
 Prior art date
 Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
 Active
Links
 235000015468 Lycium chinense Nutrition 0.000 title claims abstract description 29
 235000015459 Lycium barbarum Nutrition 0.000 title claims abstract description 15
 240000001313 Lycium barbarum Species 0.000 title claims description 13
 235000017784 Mespilus germanica Nutrition 0.000 claims abstract description 45
 240000002645 Mimusops elengi Species 0.000 claims abstract description 45
 235000000560 Mimusops elengi Nutrition 0.000 claims abstract description 45
 235000007837 Vangueria infausta Nutrition 0.000 claims abstract description 45
 230000011218 segmentation Effects 0.000 claims abstract description 38
 240000002012 Lycium chinense Species 0.000 claims abstract description 20
 238000007781 preprocessing Methods 0.000 claims abstract description 8
 230000004927 fusion Effects 0.000 claims abstract description 4
 230000000875 corresponding Effects 0.000 claims description 21
 238000005070 sampling Methods 0.000 claims description 14
 210000002320 Radius Anatomy 0.000 claims description 12
 238000000605 extraction Methods 0.000 claims description 6
 241001106041 Lycium Species 0.000 claims description 4
 238000006243 chemical reaction Methods 0.000 claims description 3
 239000000126 substance Substances 0.000 claims description 3
 235000013399 edible fruits Nutrition 0.000 abstract description 4
 230000001131 transforming Effects 0.000 abstract description 2
 238000004458 analytical method Methods 0.000 description 1
 238000001514 detection method Methods 0.000 description 1
 238000010586 diagram Methods 0.000 description 1
 238000009826 distribution Methods 0.000 description 1
 230000000694 effects Effects 0.000 description 1
 238000005516 engineering process Methods 0.000 description 1
 238000003306 harvesting Methods 0.000 description 1
 238000004519 manufacturing process Methods 0.000 description 1
 239000000463 material Substances 0.000 description 1
 238000005259 measurement Methods 0.000 description 1
 238000000034 method Methods 0.000 description 1
 230000004048 modification Effects 0.000 description 1
 238000006011 modification reaction Methods 0.000 description 1
 238000003012 network analysis Methods 0.000 description 1
 239000002245 particle Substances 0.000 description 1
 239000002689 soil Substances 0.000 description 1
 238000003860 storage Methods 0.000 description 1
 XLYOFNOQVPJJNPUHFFFAOYSAN water Substances O XLYOFNOQVPJJNPUHFFFAOYSAN 0.000 description 1
Classifications

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T7/00—Image analysis
 G06T7/10—Segmentation; Edge detection
 G06T7/136—Segmentation; Edge detection involving thresholding

 G06F18/23213—

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/10—Image acquisition modality
 G06T2207/10028—Range image; Depth image; 3D point clouds

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/20—Special algorithmic details
 G06T2207/20112—Image segmentation details
 G06T2207/20152—Watershed segmentation

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30181—Earth observation
 G06T2207/30188—Vegetation; Agriculture

 G—PHYSICS
 G06—COMPUTING; CALCULATING; COUNTING
 G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
 G06T2207/00—Indexing scheme for image analysis or image enhancement
 G06T2207/30—Subject of image; Context of image processing
 G06T2207/30242—Counting objects in image
Abstract
The invention relates to a wolfberry identification and counting method based on a PointNet + + network, which solves the problem that a twodimensional image identification method is difficult to accurately identify due to shielding overlapping before picking of wolfberry fruits in the prior art. The invention comprises the following steps: collecting and preprocessing a PointNet + + network training sample; acquiring a PointNet + + model based on the PointNet + + network fusion context information; acquiring and preprocessing point cloud data to be identified; and identifying and counting the number of the Chinese wolfberry. The invention integrates a PointNet + + network based on context information and an optimal threshold watershed segmentation algorithm based on distance transformation, and realizes accurate segmentation and counting of the medlar.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a medlar recognition counting method based on a PointNet + + network.
Background
In the planting process of the medlar, the region with low yield can be modified pertinently by accurately predicting the medlar yield, factors such as soil, variety or water content are adjusted, and resources such as manpower, material resources, storage and the like required in harvesting are reasonably arranged.
Currently, in fruit target identification and yield estimation, twodimensional images are mostly used as data input for yield measurement. However, in practical application, because the medlar particles are small, and most of medlar fruits have the problems of shielding and overlapping before picking, the depth learning method based on the twodimensional image is low in production efficiency and poor in robustness, and the accuracy of the method is difficult to guarantee.
Therefore, how to design a new identification and counting method to improve the accuracy of identification and counting of the medlar becomes a technical problem to be solved urgently.
Disclosure of Invention
The invention aims to solve the defect that the twodimensional image recognition method is difficult to accurately recognize due to shielding overlapping before picking of medlar fruits in the prior art, and provides a medlar recognition and counting method based on a PointNet + + network to solve the problem.
In order to achieve the purpose, the technical scheme of the invention is as follows:
a wolfberry identification and counting method based on a PointNet + + network comprises the following steps:
11) collecting and preprocessing PointNet + + network training samples: acquiring 18 pictures and corresponding context information of each Chinese wolfberry tree, wherein the context information comprises time, space, temperature and phenological period information, constructing a threedimensional model through the 18 pictures to obtain threedimensional point cloud, and using the threedimensional point cloud together with the context information as training data;
12) acquiring a PointNet + + model based on the PointNet + + network fusion context information:
setting a PointNet + + network, and setting a sampling layer, a grouping layer, a PointNet layer and a segmentation layer of the PointNet + + network; training a PointNet + + network, introducing time, space, temperature and phenological period context information when point cloud data are obtained as characteristic data, and training a PointNet + + model for segmenting the threedimensional point cloud data of the Chinese wolfberry;
13) acquiring and preprocessing point cloud data to be identified: acquiring 18 pictures of a Chinese wolfberry tree to be identified and corresponding time, space, temperature and phenological period context information of the pictures, constructing a threedimensional model and acquiring a threedimensional point cloud to be identified;
14) identification and counting of the number of the medlar:
inputting the threedimensional point cloud to be identified and the contextual information as the characteristics into a PointNet + + model to obtain the score of the medlar target, and counting the number of the medlar by a marking method.
The method for setting the PointNet + + network comprises the following steps:
21) setting a PointNet + + network to sequentially comprise a sampling layer, a grouping layer and a hierarchy combination of PointNet layers;
22) setting a sampling layer as a selection center;
obtaining central points by using a kmeans + + clustering method, wherein the number of the central points in the first layer is m;
X＝{x_{1},x_{2},......,x_{n}is the input set of labeled points,
wherein x is_{i}∈R^{d+C}D is a point cloud dimension, and C is a characteristic dimension;
if x is taken randomly_{k}∈R^{d+C}The first cluster center is then followed by centeringThe probability reaches a maximum value, where D (x)_{j}) Is x_{j}Distance from the nearest one of the cluster centers;
23) setting a grouping layer, wherein the grouping layer is used for creating a plurality of subpoint clouds by adjacent points in a given radius;
231) obtaining a given radius;
232) searching for fixed adjacent points by using a kNN algorithm and taking the clustering center as a reference, wherein the radiuses are respectively 0.1, 0.2 and 0.4, and the corresponding maximum points in the circle are 16, 32 and 128;
the input point set of the first layer is n x (d + C), the output point set is n' × kx (d + C), wherein k is the number of points contained in the neighborhood;
24) setting a PointNet layer, wherein the PointNet layer is used for obtaining higher dimensional representation of the subpoint cloud;
setting a point set n ' × kx (d + C) as the input of a PointNet + + network, performing feature extraction on the point set, and finally outputting n ' × (d + C ');
repeating the steps 22), 23) and 24) three times, i.e. combining 3 network layers_{1}、l_{2}、l_{3}A module for continuously extracting the features;
25) setting a segmentation layer, and obtaining the score of each original point by adopting an interpolation and backtracking mode based on context information;
251) input l_{3}Point set of layers n '× (d + C');
252) for l_{2}Point of layer consisting of_{3}After interpolation of the layer points, carrying out convolution of 1 x 1 with the corresponding characteristics obtained in 24) to finally obtain l_{2}The values of the layer points are traced all the time to finally obtain l_{1}A score of the origin;
253) while interpolation is carried out, context information of input corresponding points is taken as characteristic data and introduced into the Unit PointNet + + network of the last layer;
H_{1}＝σ(W_{1}x_{i}+UD+b_{1})，
The rest layers are H_{m}＝σ(W_{m}H_{m1}+b_{m}) Wherein M ∈ {2,..., M };
254) acquisition Point (x)_{p},y_{p},z_{p})∈R^{C}Probability P of belonging to target medlar class or background class_{c}'(x_{p},y_{p},z_{p}) It is expressed as follows:
wherein c is { target matrimony vine class, background class },for the weights of the last layer belonging to class c, all parameters (W, U, b) are learned by minimizing the cross entropy loss function.
The PointNet + + network training method comprises the following steps:
31) inputting the threedimensional point cloud into a sampling layer to obtain a central point of the threedimensional point cloud;
32) inputting the central point of the threedimensional point cloud into a grouping layer to obtain a plurality of subpoint clouds in the radius of the central point of the threedimensional point cloud;
33) inputting a plurality of subpoint clouds into a PointNet + + network, and inputting corresponding characteristic data into a multilayer perceptron of the PointNet + + network for characteristic extraction;
34) repeating step 31), step 32) and step 33) three times in this order, i.e. 3 modules are assembled with such a network layer;
35) and obtaining the probability of the original point serving as the medlar or the background by adopting an interpolation and backtracking mode.
The identification and counting of the number of the medlar also comprises the step of accurately segmenting point cloud: on the basis of distance conversion, a point cloud is segmented by using an optimal threshold watershed segmentation algorithm, and the number of the Chinese wolfberries is counted by using a marking method;
the accurate segmentation of the point cloud comprises the following steps:
41) reconstructing the distance point cloud to distinguish the boundary points and the internal points of the medlar;
setting N (P) as an eight neighborhood point set, P (x)_{p},y_{p},z_{p}) Representing the gray value of the point P, P' (x)_{p},y_{p},z_{p}) Representing the probability, P ″ (x), that point P belongs to the target Lycium class_{p},y_{p},z_{p}) A value representing the distance of the point p from its nearest background point;
reconstructing a point cloud of distances, an order
Wherein the content of the first and second substances,
 represents the euclidean distance between points p and q;
if dist_{min}＜P”(x_{p},y_{p},z_{p}) Let P "(x)_{p},y_{p},z_{p})＝dist_{min}The distance point cloud constructed at this time has a value of P ″ (x)_{p},y_{p},z_{p}) A set of (a);
42) establishing a watershed segmentation algorithm based on an optimal threshold;
421) set of distances P "(x) to reconstruct_{p},y_{p},z_{p}) The distance value of the internal points of the Chinese wolfberry is large, the distance value corresponding to the boundary point is small, and the distance value of the background point is zero; finding out local maximum value points on all distances and endowing the local maximum value points with unique identification;
422) adopting a descending method to descend on plain, descending all points until meeting the marked area, and adding the points into the marked area;
423) combining the areas, and combining the areas with the depth lower than a threshold value;
424) determining a watershed segmentation result;
43) and (4) a target mark counting method is adopted to obtain a segmentation result in the final point cloud, and the number is counted to obtain the number of the actual medlar in the point cloud.
The area merging comprises the following steps:
51) for each region, finding out an inner curvature minimum point, a region adjacent to the region and a minimum point on the boundary of each adjacent region;
52) calculating the watershed depth of each region;
53) calculating an optimal threshold value;
let the point cloud have L reconstructed distance values, the point with distance value i has n_{i}Number of individuals and total pointsIs N, the probability of each distance value occurring isAnd is
The threshold value is set as t, the point cloud is divided into 2 parts, namely a background class a {0, 1, 2,. and t } and a medlar target class B { t +1, t +2,. and L1} based on the distance value, and the probability of occurrence of the two classes is respectively:
the distance mean values of the two types A and B are respectivelyThe total distance mean in the point cloud is:
from this, the betweenclass variance of the two parts A and B is obtained
The larger the interclass variance is, the larger the difference of the distance values between the two classes is, that is, the optimal threshold is determined as follows:
54) when the depth is lower than the threshold value, combining the region and the region adjacent to the boundary curvature minimum value point and updating the related information;
55) repeating 54) until the depth of all the areas is greater than a given threshold.
Advantageous effects
Compared with the prior art, the wolfberry identification and counting method based on the PointNet + + network integrates the PointNet + + network based on context information and the optimal threshold watershed segmentation algorithm based on distance transformation, and realizes accurate segmentation and counting of the wolfberry.
The method is based on threedimensional point cloud, utilizes a PointNet + + network based on context information and an improved watershed segmentation algorithm to carry out accurate segmentation on the medlar, overcomes the defect that twodimensional image processing and recognition technology is simply utilized to carry out prediction, and can accurately predict the yield of the medlar in the planting base.
The invention not only considers relevant context information (time, space, temperature and phenological period), but also provides an optimal watershed segmentation algorithm improved based on a reconstruction distance to solve the problem of accurate segmentation of the medlar aiming at the problem of inaccuracy of segmentation of overlapped and shielded medlar by using a PointNet + + network, thereby improving the medlar detection capability of a medlar planting base under a complex natural condition and improving the accuracy of medlar yield prediction.
Drawings
FIG. 1 is a sequence diagram of the method of the present invention.
Detailed Description
In order to further understand and appreciate the structural features and advantages achieved by the present invention, a preferred embodiment is described in detail with reference to the accompanying drawings, in which:
as shown in fig. 1, the method for identifying and counting medlar based on PointNet + + network according to the present invention includes the following steps:
the first step, collecting and preprocessing a PointNet + + network training sample.
The method comprises the steps of obtaining 18 pictures of each Chinese wolfberry tree and corresponding context information, wherein the context information comprises time, space, temperature and phenological period information, constructing a threedimensional model through the 18 pictures according to the existing method to obtain threedimensional point cloud, and using the threedimensional point cloud together with the context information as training data.
And secondly, acquiring a PointNet + + model based on the PointNet + + network fusion context information.
First, a PointNet + + network is set, and a sampling layer, a grouping layer, a PointNet layer, and a splitting layer of the PointNet + + network are set. Secondly, training a PointNet + + network, introducing time, space, temperature and phenological period context information when the point cloud data are obtained as characteristic data, and training a PointNet + + model for segmenting the threedimensional point cloud data of the Chinese wolfberry.
The specific steps for setting the PointNet + + network are as follows:
(1) the method is characterized in that a sampling layer, a grouping layer, a PointNet layer and a segmentation layer which are sequentially included in a PointNet + + network are set, and the hierarchical combination relationship of the sampling layer, the grouping layer and the PointNet layer of the PointNet + + network is set firstly.
(2) Setting a sampling layer as a selection center;
obtaining central points by using a kmeans + + clustering method, wherein the number of the central points in the first layer is m;
X＝{x_{1},x_{2},......,x_{n}is the input set of labeled points,
wherein x is_{i}∈R^{d+C}D is a point cloud dimension, and C is a characteristic dimension;
if x is taken randomly_{k}∈R^{d+C}The first cluster center is then followed by centeringThe probability reaches a maximum value, where D (x)_{j}) Is x_{j}Distance from the nearest one of the cluster centers.
(3) Setting a grouping layer, wherein the grouping layer is used for creating a plurality of subpoint clouds by adjacent points in a given radius;
A1) obtaining a given radius;
A2) searching for fixed adjacent points by using a kNN algorithm and taking the clustering center as a reference, wherein the radiuses are respectively 0.1, 0.2 and 0.4, and the corresponding maximum points in the circle are 16, 32 and 128;
the input point set of the first layer is n x (d + C), the output point set is n' × kx (d + C), wherein k is the number of points contained in the neighborhood;
(4) setting a PointNet layer, wherein the PointNet layer is used for obtaining higher dimensional representation of the subpoint cloud;
setting a point set n ' × kx (d + C) as the input of a PointNet + + network, performing feature extraction on the point set, and finally outputting n ' × (d + C ');
repeating the step (2) (setting sampling layer), the step (3) (setting grouping layer) and the step (4) (setting PointNet layer) three times, namely combining the network layers into 3 l_{1}、l_{2}、l_{3}And the module is used for continuously extracting the features.
(5) Setting a segmentation layer, and obtaining the score of each original point by adopting an interpolation and backtracking mode based on context information;
B1) input l_{3}Point set of layers n '× (d + C');
B2) for l_{2}Point of layer consisting of_{3}After interpolation of the layer points, carrying out convolution of 1 x 1 with the corresponding characteristics obtained in 24) to finally obtain l_{2}The values of the layer points are traced all the time to finally obtain l_{1}A score of the origin;
B3) while interpolation is carried out, context information of input corresponding points is taken as characteristic data and introduced into the Unit PointNet + + network of the last layer;
H_{1}＝σ(W_{1}x_{i}+UD+b_{1})，
The rest layers are H_{m}＝σ(W_{m}H_{m1}+b_{m}) Wherein M ∈ {2,..., M };
B4) acquisition Point (x)_{p},y_{p},z_{p})∈R^{C}Probability P of belonging to target medlar class or background class_{c}'(x_{p},y_{p},z_{p}) It is expressed as follows:
wherein, c ═ target lycium barbarumThe background class },for the weights of the last layer belonging to class c, all parameters (W, U, b) are learned by minimizing the cross entropy loss function.
Secondly, training the PointNet + + network comprises the following steps:
(1) and inputting the threedimensional point cloud into a sampling layer to obtain a central point of the threedimensional point cloud.
(2) And inputting the central point of the threedimensional point cloud into a grouping layer to obtain a plurality of subpoint clouds in the radius of the central point of the threedimensional point cloud.
(3) And inputting a plurality of subpoint clouds into a PointNet + + network, and inputting corresponding characteristic data into a multilayer perceptron of the PointNet + + network for characteristic extraction.
(4) And (3) repeating the steps (1), (2) and (3) in the step of training the PointNet + + network three times, namely combining 3 modules by using the network layer.
(5) And obtaining the probability of the original point serving as the medlar or the background by adopting an interpolation and backtracking mode.
And thirdly, acquiring and preprocessing point cloud data to be identified.
Acquiring 18 pictures of the Chinese wolfberry tree to be identified and corresponding time, space, temperature and phenological period context information of the pictures, constructing a threedimensional model and acquiring a threedimensional point cloud to be identified.
And fourthly, identifying and counting the number of the Chinese wolfberry.
Inputting the threedimensional point cloud to be identified and the contextual information as the characteristics into a PointNet + + model to obtain the score of the medlar target, and counting the number of the medlar by a marking method.
The number of the medlar can be obtained, and counting identification of the medlar is realized. However, in practical applications, although the analysis is performed on the threedimensional point cloud data according to the characteristic that the distribution of the medlar is easy to overlap, a certain error still occurs. In order to further effectively distinguish the sticky and blocked wolfberries, the scores of the wolfberries target obtained in the fourth step of identifying and counting the number of the wolfberries are utilized, namely, the points in the 3D point cloud with the scores are precisely segmented by utilizing an improved watershed segmentation algorithm. The improved watershed segmentation algorithm is utilized, weak edges generated due to small pixel difference can be processed, the edges are accurately positioned, and even a single Chinese wolfberry in a cluster can be effectively distinguished.
And fifthly, performing accurate segmentation on the point cloud.
On the basis of distance conversion, the point cloud is segmented by using an optimal threshold watershed segmentation algorithm, and the number of the Chinese wolfberries is counted by using a marking method. The method specifically comprises the following steps:
(1) reconstructing the distance point cloud to distinguish the boundary points and the internal points of the medlar;
setting N (P) as an eight neighborhood point set, P (x)_{p},y_{p},z_{p}) Representing the gray value of the point P, P' (x)_{p},y_{p},z_{p}) Representing the probability, P ″ (x), that point P belongs to the target Lycium class_{p},y_{p},z_{p}) A value representing the distance of the point p from its nearest background point;
reconstructing a point cloud of distances, an order
Wherein the content of the first and second substances,
 represents the euclidean distance between points p and q;
if dist_{min}＜P”(x_{p},y_{p},z_{p}) Let P "(x)_{p},y_{p},z_{p})＝dist_{min}The distance point cloud constructed at this time has a value of P ″ (x)_{p},y_{p},z_{p}) A collection of (a).
(2) And establishing a watershed segmentation algorithm based on an optimal threshold value. In the method, the selection of the extreme points in the watershed segmentation algorithm is updated, so that the result of PointNet + + network analysis can be utilized, the calculated amount is reduced, the efficiency is improved, and both large data processing and accurate segmentation can be realized. The method comprises the following specific steps:
C1) set of distances P "(x) to reconstruct_{p},y_{p},z_{p}) And as input, the distance value of the internal points of the Chinese wolfberry is larger, the distance value corresponding to the boundary point is smaller, and the distance value of the background point is zero. Finding out local maximum value points on all distances and endowing the local maximum value points with unique identification;
C2) adopting a descending method to descend on plain, descending all points until meeting the marked area, and adding the points into the marked area;
C3) combining the areas, and combining the areas with the depth lower than a threshold value;
C4) and determining a watershed segmentation result.
Here, the regions are subjected to the merging step individually as follows:
C31) for each region, finding out an inner curvature minimum point, a region adjacent to the region and a minimum point on the boundary of each adjacent region;
C32) calculating the watershed depth of each region;
C33) calculating an optimal threshold value;
let the point cloud have L reconstructed distance values, the point with distance value i has n_{i}N, the total number of points, and the probability of each distance value appearing isAnd is
The threshold value is set as t, the point cloud is divided into 2 parts, namely a background class a {0, 1, 2,. and t } and a medlar target class B { t +1, t +2,. and L1} based on the distance value, and the probability of occurrence of the two classes is respectively:
the distance mean values of the two types A and B are respectivelyThe total distance mean in the point cloud is:
from this, the betweenclass variance of the two parts A and B is obtained
The larger the interclass variance is, the larger the difference of the distance values between the two classes is, that is, the optimal threshold is determined as follows:
C34) when the depth is lower than the threshold value, combining the region and the region adjacent to the boundary curvature minimum value point and updating the related information;
C35) repeating the step C34) until the depths of all the areas are greater than the given threshold.
(3) And (4) a target mark counting method is adopted to obtain a segmentation result in the final point cloud, and the number is counted to obtain the number of the actual medlar in the point cloud.
The segmentation result in the final point cloud can be obtained by using the segmentation method, and in practical application, the segmentation result is each basin (each medlar target object), and the actual number of the medlar in the point cloud can be accurately obtained only by counting the number of the basins.
The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are merely illustrative of the principles of the invention, but that various changes and modifications may be made without departing from the spirit and scope of the invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (3)
1. A wolfberry identification and counting method based on a PointNet + + network is characterized by comprising the following steps:
11) collecting and preprocessing PointNet + + network training samples: acquiring 18 pictures and corresponding context information of each Chinese wolfberry tree, wherein the context information comprises time, space, temperature and phenological period information, constructing a threedimensional model through the 18 pictures to obtain threedimensional point cloud, and using the threedimensional point cloud together with the context information as training data;
12) acquiring a PointNet + + model based on the PointNet + + network fusion context information:
setting a PointNet + + network, and setting a sampling layer, a grouping layer, a PointNet layer and a segmentation layer of the PointNet + + network; training a PointNet + + network, introducing time, space, temperature and phenological period context information when point cloud data are obtained as characteristic data, and training a PointNet + + model for segmenting the threedimensional point cloud data of the Chinese wolfberry;
the method for setting the PointNet + + network comprises the following steps:
121) setting a PointNet + + network to sequentially comprise a sampling layer, a grouping layer and a hierarchy combination of PointNet layers;
122) setting a sampling layer as a selection center;
obtaining central points by using a kmeans + + clustering method, wherein the number of the central points in the first layer is m;
X＝{x_{1},x_{2},......,x_{n}is the input set of labeled points,
wherein x is_{i}∈R^{d+C}D is a point cloud dimension, and C is a characteristic dimension;
if x is taken randomly_{k}∈R^{d+C}The first cluster center is then followed by centeringThe probability reaches a maximum value, where D (x)_{j}) Is x_{j}Distance from the nearest one of the cluster centers;
123) setting a grouping layer, wherein the grouping layer is used for creating a plurality of subpoint clouds by adjacent points in a given radius;
1231) obtaining a given radius;
1232) searching for fixed adjacent points by using a kNN algorithm and taking the clustering center as a reference, wherein the radiuses are respectively 0.1, 0.2 and 0.4, and the corresponding maximum points in the circle are 16, 32 and 128;
the input point set of the first layer is n x (d + C), the output point set is n' × kx (d + C), wherein k is the number of points contained in the neighborhood;
124) setting a PointNet layer, wherein the PointNet layer is used for obtaining higher dimensional representation of the subpoint cloud;
setting a point set n ' × kx (d + C) as the input of a PointNet + + network, performing feature extraction on the point set, and finally outputting n ' × (d + C ');
repeating the steps 122), 123), 124) three times in this way, i.e. 3 l of such network layer combinations_{1}、l_{2}、l_{3}A module for continuously extracting the features;
125) setting a segmentation layer, and obtaining the score of each original point by adopting an interpolation and backtracking mode based on context information;
1251) input l_{3}Point set of layers n '× (d + C');
1252) for l_{2}Point of layer consisting of_{3}After interpolation of the layer points, carrying out convolution of 1 x 1 with the corresponding characteristics obtained in 124) to finally obtain l_{2}The values of the layer points are traced all the time to finally obtain l_{1}A score of the origin;
1253) while interpolation is carried out, context information of input corresponding points is taken as characteristic data and introduced into the Unit PointNet + + network of the last layer;
for all the context information fused, the first layer input of the multilayer perceptron is
H_{1}＝σ(W_{1}x_{i}+UD+b_{1})，
The rest layers are H_{m}＝σ(W_{m}H_{m1}+b_{m}) Wherein M ∈ {2,..., M };
1254) acquisition Point (x)_{p},y_{p},z_{p})∈R^{C}Probability P 'of belonging to target matrimony vine or background'_{c}(x_{p},y_{p},z_{p}) It is expressed as follows:
wherein c is { target matrimony vine class, background class },for the weights of the last layer belonging to class c, all parameters (W, U, b) are learned by minimizing the cross entropy loss function;
the PointNet + + network training method comprises the following steps:
131) inputting the threedimensional point cloud into a sampling layer to obtain a central point of the threedimensional point cloud;
132) inputting the central point of the threedimensional point cloud into a grouping layer to obtain a plurality of subpoint clouds in the radius of the central point of the threedimensional point cloud;
133) inputting a plurality of subpoint clouds into a PointNet + + network, and inputting corresponding characteristic data into a multilayer perceptron of the PointNet + + network for characteristic extraction;
134) repeating step 131), step 132), step 133) three times in this order, i.e. 3 modules are assembled with such a network layer;
135) obtaining the probability of the original point as the medlar or the background by adopting an interpolation and backtracking mode;
13) acquiring and preprocessing point cloud data to be identified: acquiring 18 pictures of a Chinese wolfberry tree to be identified and corresponding time, space, temperature and phenological period context information of the pictures, constructing a threedimensional model and acquiring a threedimensional point cloud to be identified;
14) identification and counting of the number of the medlar: inputting the threedimensional point cloud to be identified and the contextual information as the characteristics into a PointNet + + model to obtain the score of the medlar target, and counting the number of the medlar by a marking method.
2. The method according to claim 1, wherein the identification and counting of the number of the lycium barbarum comprises the following steps of performing accurate segmentation on the point cloud: on the basis of distance conversion, a point cloud is segmented by using an optimal threshold watershed segmentation algorithm, and the number of the Chinese wolfberries is counted by using a marking method;
the accurate segmentation of the point cloud comprises the following steps:
21) reconstructing the distance point cloud to distinguish the boundary points and the internal points of the medlar;
setting N (P) as an eight neighborhood point set, P (x)_{p},y_{p},z_{p}) Representing the gray value of the point P, P' (x)_{p},y_{p},z_{p}) Representing the probability, P ″ (x), that point P belongs to the target Lycium class_{p},y_{p},z_{p}) A value representing the distance of the point p from its nearest background point;
reconstructing a point cloud of distances, an order
Wherein the content of the first and second substances,
 represents the euclidean distance between points p and q;
if dist_{min}<P”(x_{p},y_{p},z_{p}) Let P "(x)_{p},y_{p},z_{p})＝dist_{min}The distance point cloud constructed at this time has a value of P ″ (x)_{p},y_{p},z_{p}) A set of (a);
22) establishing a watershed segmentation algorithm based on an optimal threshold;
221) set of distances P "(x) to reconstruct_{p},y_{p},z_{p}) The distance value of the internal points of the Chinese wolfberry is large, the distance value corresponding to the boundary point is small, and the distance value of the background point is zero; finding out all distancesThe local maximum value point on the distance is given with a unique identifier;
222) adopting a descending method to descend on plain, descending all points until meeting the marked area, and adding the points into the marked area;
223) combining the areas, and combining the areas with the depth lower than a threshold value;
224) determining a watershed segmentation result;
23) and (4) a target mark counting method is adopted to obtain a segmentation result in the final point cloud, and the number is counted to obtain the number of the actual medlar in the point cloud.
3. The method according to claim 2, wherein the region merging comprises the following steps:
31) for each region, finding out an inner curvature minimum point, a region adjacent to the region and a minimum point on the boundary of each adjacent region;
32) calculating the watershed depth of each region;
33) calculating an optimal threshold value;
let the point cloud have L reconstructed distance values, the point with distance value i has n_{i}N, the total number of points, and the probability of each distance value appearing isAnd is
The threshold value is set as t, the point cloud is divided into 2 parts, namely a background class a {0, 1, 2,. and t } and a medlar target class B { t +1, t +2,. and L1} based on the distance value, and the probability of occurrence of the two classes is respectively:
the distance mean values of the two types A and B are respectivelyThe total distance mean in the point cloud is:
from this, the betweenclass variance of the two parts A and B is obtained
The larger the interclass variance is, the larger the difference of the distance values between the two classes is, that is, the optimal threshold is determined as follows:
34) when the depth is lower than the threshold value, combining the region and the region adjacent to the boundary curvature minimum value point and updating the related information;
35) repeating 34) until the depth of all the areas is greater than a given threshold.
Priority Applications (1)
Application Number  Priority Date  Filing Date  Title 

CN201811098583.0A CN109410238B (en)  20180920  20180920  Wolfberry identification and counting method based on PointNet + + network 
Applications Claiming Priority (1)
Application Number  Priority Date  Filing Date  Title 

CN201811098583.0A CN109410238B (en)  20180920  20180920  Wolfberry identification and counting method based on PointNet + + network 
Publications (2)
Publication Number  Publication Date 

CN109410238A CN109410238A (en)  20190301 
CN109410238B true CN109410238B (en)  20211026 
Family
ID=65465779
Family Applications (1)
Application Number  Title  Priority Date  Filing Date 

CN201811098583.0A Active CN109410238B (en)  20180920  20180920  Wolfberry identification and counting method based on PointNet + + network 
Country Status (1)
Country  Link 

CN (1)  CN109410238B (en) 
Families Citing this family (5)
Publication number  Priority date  Publication date  Assignee  Title 

CN109993748A (en) *  20190330  20190709  华南理工大学  A kind of threedimensional grid method for segmenting objects based on points cloud processing network 
CN110032962B (en) *  20190403  20220708  腾讯科技（深圳）有限公司  Object detection method, device, network equipment and storage medium 
CN110276266B (en) *  20190528  20210910  暗物智能科技(广州)有限公司  Rotationbased point cloud data processing method and device and terminal equipment 
CN110210431B (en) *  20190606  20210511  上海黑塞智能科技有限公司  Point cloud semantic labeling and optimizationbased point cloud classification method 
CN114973006B (en) *  20220802  20221018  四川省机械研究设计院(集团)有限公司  Method, device and system for picking Chinese prickly ash and storage medium 
Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

CN105740798A (en) *  20160127  20160706  西安理工大学  Structure analysis based identification method for object in point cloud scene 
CN106919901A (en) *  20170120  20170704  江苏大学  A kind of depth ball transversal method of spherefruit identification in strain 
CN107679562A (en) *  20170920  20180209  北京航空航天大学  The analyzing and processing method and device of threedimensional model 
CN108182704A (en) *  20161208  20180619  广州映博智能科技有限公司  Localization method based on Shape context feature 
CN108196535A (en) *  20171212  20180622  清华大学苏州汽车研究院（吴江）  Automated driving system based on enhancing study and Multisensor Fusion 
CN108319959A (en) *  20171229  20180724  中国科学院合肥物质科学研究院  A kind of corps diseases imagerecognizing method compressed based on characteristics of image with retrieval 
CN108389251A (en) *  20180321  20180810  南京大学  The full convolutional network threedimensional model dividing method of projection based on fusion various visual angles feature 
Family Cites Families (1)
Publication number  Priority date  Publication date  Assignee  Title 

US20040013749A1 (en) *  20020722  20040122  Young D. Gary  Antioxidant and immune boosting composition and methods of using 

2018
 20180920 CN CN201811098583.0A patent/CN109410238B/en active Active
Patent Citations (7)
Publication number  Priority date  Publication date  Assignee  Title 

CN105740798A (en) *  20160127  20160706  西安理工大学  Structure analysis based identification method for object in point cloud scene 
CN108182704A (en) *  20161208  20180619  广州映博智能科技有限公司  Localization method based on Shape context feature 
CN106919901A (en) *  20170120  20170704  江苏大学  A kind of depth ball transversal method of spherefruit identification in strain 
CN107679562A (en) *  20170920  20180209  北京航空航天大学  The analyzing and processing method and device of threedimensional model 
CN108196535A (en) *  20171212  20180622  清华大学苏州汽车研究院（吴江）  Automated driving system based on enhancing study and Multisensor Fusion 
CN108319959A (en) *  20171229  20180724  中国科学院合肥物质科学研究院  A kind of corps diseases imagerecognizing method compressed based on characteristics of image with retrieval 
CN108389251A (en) *  20180321  20180810  南京大学  The full convolutional network threedimensional model dividing method of projection based on fusion various visual angles feature 
NonPatent Citations (3)
Title 

《PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space》;Charles R. Qi等;《31st Conference on Neural Information Processing Systems (NIPS 2017)》;20171231;第110页 * 
《基于WebGIS的农场生产管理信息系统的设计与实现》;魏圆圆 等;《农业工程学报》;20180822;第34卷(第16期);第139147页 * 
《枸杞分级自动计数系统的设计》;李好博;《工业控制计算机》;20080925;第09卷;第8788页 * 
Also Published As
Publication number  Publication date 

CN109410238A (en)  20190301 
Similar Documents
Publication  Publication Date  Title 

CN109410238B (en)  Wolfberry identification and counting method based on PointNet + + network  
Chaudhuri et al.  Multilabel remote sensing image retrieval using a semisupervised graphtheoretic method  
Peng et al.  Automatic image segmentation by dynamic region merging  
CN111259786B (en)  Pedestrian reidentification method based on synchronous enhancement of appearance and motion information of video  
CN109671102B (en)  Comprehensive target tracking method based on depth feature fusion convolutional neural network  
CN110956185A (en)  Method for detecting image salient object  
CN108280396B (en)  Hyperspectral image classification method based on depth multifeature active migration network  
Alidoost et al.  A CNNbased approach for automatic building detection and recognition of roof types using a single aerial image  
Zhang et al.  Road recognition from remote sensing imagery using incremental learning  
Lovell et al.  Performance evaluation of indirect immunofluorescence image analysis systems  
CN111968124B (en)  Shoulder musculoskeletal ultrasonic structure segmentation method based on semisupervised semantic segmentation  
CN108171119B (en)  SAR image change detection method based on residual error network  
CN112633382A (en)  Mutualneighborbased fewsample image classification method and system  
CN111310622A (en)  Fish swarm target identification method for intelligent operation of underwater robot  
Huynh et al.  Plant identification using new architecture convolutional neural networks combine with replacing the red of color channel image by vein morphology leaf  
Ge et al.  Coarsetofine foraminifera image segmentation through 3D and deep features  
Li et al.  Deep LearningBased Object Detection Techniques for Remote Sensing Images: A Survey  
CN109033944B (en)  Method and system for classifying allsky aurora images and positioning key local structure  
CN110728694A (en)  Longterm visual target tracking method based on continuous learning  
Florinabel  Realtime image processing method to implement object detection and classification for remote sensing images  
CN113157678B (en)  Multisource heterogeneous data association method  
Liu et al.  Contouroriented cropland extraction from high resolution remote sensing imagery using richer convolution features network  
CN113988147A (en)  Multilabel classification method and device for remote sensing image scene based on graph network, and multilabel retrieval method and device  
Ahmad et al.  A fusion of labeledgrid shape descriptors with weighted ranking algorithm for shapes recognition  
Zanwar et al.  A comprehensive survey on soft computing based optical character recognition techniques 
Legal Events
Date  Code  Title  Description 

PB01  Publication  
PB01  Publication  
SE01  Entry into force of request for substantive examination  
SE01  Entry into force of request for substantive examination  
GR01  Patent grant  
GR01  Patent grant 