CN109684967A - A kind of soybean plant strain stem pod recognition methods based on SSD convolutional network - Google Patents

Abstract

The invention provides a method for identifying stems and pods of soybean plants based on SSD convolutional networks, comprising the following steps: collecting a single soybean sample image to obtain a soybean plant image database; manually labeling the stems and pods, and labeling the pods with unobstructed pod tips The exposed part of the stem is marked, and the image database is divided into training set, validation set, and test set without repetition; random image enhancement and data augmentation are performed on the marked images of the training set, and the new images are automatically re-marked. image; construct an SSD convolutional network, use feature maps of different levels to perform multi-scale detection; randomly extract the training set for the training of the SSD convolutional neural network, and determine the learning parameters in the SSD convolutional neural network ; Send the test set to the trained SSD convolutional neural network for recognition test, and mark the recognition result in the original image in the test sample. Provided is a soybean plant stem pod identification method based on SSD convolution network, which can intelligently identify soybean plant stem pods through network training, has a high degree of automation, and effectively improves the detection efficiency of soybean plant stem pods.

Description

A method for identifying stems and pods of soybean plants based on SSD convolutional network

technical field

The invention relates to the technical field of computer image processing and identification methods, and more particularly, to a method for identifying stems and pods of soybean plants based on SSD convolutional networks.

Background technique

Soybean is an important grain and oil crop in the world, and it is also the main source of high-quality protein for human beings. It is not only one of the main crops in my country, but also the most economical crop. Soybean seed testing is an important link in the process of analyzing the genetics and breeding of soybean crops, which collects, organizes and counts the data on the traits of the whole soybean plant. At present, the soybean seed testing work mainly adopts manual operation. However, manual operation not only consumes a lot of manpower and material resources, but also has human errors, which brings inaccuracy of later data analysis.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art, and to provide a method for identifying stems and pods of soybean plants based on a convolutional network of SSD. The application of machine vision to the work of soybean seed testing can not only improve the accuracy of soybean plant character data, but also reduce artificial Errors, shorten the test cycle, but also reduce labor intensity, and develop in the direction of intelligence, speed, and accuracy.

In order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

Provided is a soybean plant stem pod identification method based on SSD convolution network, comprising the following steps:

S1. The Canon 5D Mark II camera is fixed at a distance of 120cm from the blue background cloth to collect the image of a single soybean sample to obtain a soybean plant image library;

S2. Traverse all the sample images in the image library described in step S1, carry out manual labeling of pods and stems for each sample image, label the unobstructed pod tips as pods, and label the bare stems as stems class, get the original image set;

S3. Perform random image enhancement and data augmentation on the labeled training set images described in step S2. Adaptive histogram equalization is used for image enhancement; random adjustment of RGB color channels within a certain threshold, horizontal and vertical mirror flips, random rotation and translation operations are used for data amplification, and the rotated and translated images are centered and intercepted , if the processed image exceeds the boundary, the label is discarded, and the enhanced training set is obtained;

S4. Construct an SSD convolutional network, and use feature maps of different levels to perform multi-scale detection;

S5. Transport the training samples in steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and determine the learning parameters in the SSD convolutional neural network simultaneously;

S6. Send the test set to the trained SSD convolutional neural network for recognition test, and use the classification result with a confidence greater than 40% in the recognition result as the output recognition result of the test sample.

The method for identifying stems and pods of soybean plants based on the SSD convolution network of the present invention adds a residual structure in which the fusion layer directly transmits the details of the lower layers to the higher layers, and makes full use of the SSD network to select feature maps of different levels for multi-scale detection. It makes up for the shortcomings of the existing methods that the detection accuracy of deformation, occlusion, and continuous overlapping objects is poor. The invention has the advantages of a convolutional neural network, reduces the interference of image background and environmental brightness, has strong anti-interference ability to occlusion and overlap, and improves the accuracy of soybean plant stem and pod detection.

Preferably, as described in step S2, pods and stems are manually labeled for each sample image, and the unobstructed tip of the pod is labeled as a pod, and the exposed part of the stem is labeled as a stem. Annotating only some features of the target can improve the recognition accuracy in the case of occlusion and overlap.

Preferably, random image enhancement and data augmentation are performed on the labeled training set images described in step S2. Adaptive histogram equalization is used for image enhancement; random adjustment of RGB color channels within a certain threshold, horizontal and vertical mirror flips, random rotation and translation operations are used for data amplification, and the rotated and translated images are centered and intercepted , a method for discarding annotations if the object is out of bounds in the processed image.

Preferably, the SSD model described in step S4 is constructed by adding one layer of fusion layer and four convolution layers in the VGG-16 network, and the steps of establishing the training model in step S4 are as follows:

S41. Taking the soybean plant sample image as input, the feature map obtained by convolution operation on the image in the convolution layer;

S42. Add the Add4_3 layer to the VGG-16 network. Add4_3 is composed of two feature maps of Maxpool3 and Conv4_2 after fusion (Add), activated by ReLU, and normalized by Batch Normalization (BN). Add4_3 is used as the Conv4_3 layer. Input, the feature maps of the Conv4_3 layer, Fc7 layer, and Conv8_2 to Conv11_2 layers in the network are convolved with a 3×3 convolution kernel to output the confidence for classification and the positioning information for output regression respectively;

S43. Combine all output structures, and obtain detection results through non-maximum suppression processing.

By selecting six feature maps of different levels for multi-scale detection, on the basis of retaining the detection of high-level feature maps, the fusion of low-level feature maps is added, which not only makes full use of the rich image detail information of low-level feature maps, but also achieves anti- The robustness effect of target detection with interference ability solves the detection and positioning problems in the case of deformation, occlusion, continuous overlap, etc.

Preferably, when outputting the confidence for classification, each frame generates two categories of confidence; when outputting the positioning information for regression, each frame generates four coordinate values (x, y, w, h).

Preferably, in step S41, the feature map is calculated according to the following method:

Step 1: Divide the feature map output by the Conv4_3 layer into 76 × 38 units, use four default bounding boxes on each unit, and use a convolution kernel of size 3 × 3 on each default bounding box for convolution operation, output The four elements of the frame are the horizontal and vertical coordinates x and y of the upper left corner of the output frame, the width w and height h of the frame output by the frame regression layer, and the confidence that the objects in the frame belong to pods and stalks respectively;

Step 2: According to the same method in step S411, the calculation is performed on the feature maps output by the Fc7 layer and the Conv8_2~Conv11_2 layers in turn; wherein, the feature maps of each layer are respectively divided into 38×19, 20×10, 1×5, 6 ×3, 1×1 units, and the default number of bounding boxes used by each unit is 6, 6, 6, 4, and 4, respectively.

Preferably, the training error of the pre-trained model in step S4 is less than 15%, and the average value of the test error is less than 20%.

Compared with the prior art, the beneficial effects of the present invention are:

(1) The invention can realize the detection and positioning of soybean plant stems and pods, has high accuracy, and has the advantages of good stability, strong anti-interference ability, high versatility, and high detection accuracy for deformation, occlusion, and continuous overlapping targets. , it is enough to apply the soybean plant trait detection system.

(2) The present invention has the advantages of convolutional neural network, reduces the interference of image background and environmental brightness, has strong anti-interference ability against occlusion and mutual overlap, and improves the accuracy of soybean plant stem and pod detection.

Description of drawings

Fig. 1 is the flow chart of the soybean plant stem pod identification method based on SSD convolution network;

Fig. 2 is the concrete flow chart of step S4;

Figure 3 is a sample image of a soybean plant identified in the present invention.

Detailed ways

The accompanying drawings are for illustrative purposes only, and should not be construed as limitations on this patent; in order to better illustrate the present embodiment, the accompanying drawings may be omitted, enlarged or reduced, and do not represent the actual size; for those skilled in the art, It is understood that certain well-known structures and their descriptions may be omitted in the drawings.

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

Please refer to FIG. 1, the present embodiment is the first embodiment of the SSD-based convolutional network soybean plant stem and pod identification method of the present invention, comprising the following steps:

S1. The Canon 5D Mark II camera is fixed at a position 120cm away from the blue background cloth to collect the image of a single soybean sample to obtain a soybean plant image library;

S2. Traverse all the sample images in the image set described in step S1, manually label the pods and stems for each sample image, label the unobstructed tip of the pods as pods, and label the exposed parts of the stem as stems , to obtain the original training set;

S3. Perform random image enhancement and data augmentation on the labeled training set images described in step S2. Adaptive histogram equalization is used for image enhancement; random adjustment of RGB color channels within a certain threshold, horizontal and vertical mirror flips, random rotation and translation operations are used for data amplification, and the rotated and translated images are centered and intercepted , if the processed image exceeds the boundary, the label is discarded, and the enhanced training set is obtained;

Please refer to FIG. 2, step S4 of this embodiment. Construct an SSD convolution network, and use feature maps of different levels to perform multi-scale detection;

S5. Transport the training samples in steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and determine the learning parameters in the SSD convolutional neural network simultaneously;

S6. Send the test set to the trained SSD convolutional neural network for recognition test, and use the classification result with a confidence greater than 40% in the recognition result as the output recognition result of the test sample.

In step S1, a Canon 5D Mark II camera is fixed at a position 120 cm away from the blue background cloth to collect an image of a single soybean sample to obtain a soybean plant image library. Specifically, the plant sample image set stores sample data in the following form:

{image_name, x, y}

Among them, image_name represents the name of the soybean plant image, x represents the horizontal pixel value of the image, and y represents the vertical pixel value of the image.

In step S2, all the sample images in the image library described in step S1 are traversed, and pods and stems are manually labeled for each sample image, and the unobstructed pod tips are labeled as pods, and the bare stems are labeled. Stem class, get the original image set. Specifically, the plant sample image annotates each real frame to form an image annotation set, and the image annotation set stores the marked data in the following form:

{label, xmin, ymin, xmax, ymax}

Among them, label represents the category of the annotation, xmin represents the abscissa of the smallest pixel of the annotation, ymin represents the ordinate of the smallest pixel of the annotation, xmax represents the abscissa of the largest pixel of the annotation, and ymax represents the vertical of the largest pixel of the annotation. coordinate.

In step S4, the establishment steps of the model after pre-training are as follows:

S41. Take the soybean plant sample image as input, and perform the convolution operation on the image in the convolution layer to obtain the feature map;

S42. Add the Add4_3 layer to the VGG-16 network. Add4_3 is composed of two feature maps of Maxpool3 and Conv4_2 after fusion (Add), activated by ReLU, and normalized by Batch Normalization (BN). Add4_3 is used as the Conv4_3 layer. Input, the feature maps of the Conv4_3 layer, Fc7 layer, and Conv8_2 to Conv11_2 layers in the network are convolved with a 3×3 convolution kernel to output the confidence for classification and the positioning information for output regression respectively;

S43. Combine all output structures, and obtain detection results through non-maximum suppression processing; wherein, the confidence level for output classification is the confidence level of each prediction box category; the positioning information for output regression is each prediction box The four coordinate values (x, y, w, h) of .

Wherein, the feature map in step S41 is calculated according to the following method:

Step 1: Divide the feature map output by the Conv4_3 layer into 76 × 38 units, use four default bounding boxes on each unit, and use a convolution kernel of size 3 × 3 on each default bounding box for convolution operation, output The four elements of the frame are the horizontal and vertical coordinates x and y of the upper left corner of the output frame, the width w and height h of the frame output by the frame regression layer, and the confidence that the objects in the frame belong to pods and stalks respectively;

Step 2: According to the same method in step S411, the calculation is performed on the feature maps output by the Fc7 layer and the Conv8_2 to Conv11_2 layers in turn; wherein, the feature maps of each layer are respectively divided into 38×19, 20×10, 10×5, 6 ×3, 1×1 units, and the default number of bounding boxes used by each unit is 6, 6, 6, 4, and 4, respectively.

By selecting six feature maps of different levels for multi-scale detection, on the basis of retaining the detection of high-level feature maps, the fusion of low-level feature maps is added, which not only makes full use of the rich image detail information of low-level feature maps, but also achieves anti- The robustness effect of target detection with interference ability solves the detection and positioning problems in the case of deformation, occlusion, continuous overlap, etc.

The partial network structure of VGG-16 with residual structure added in this embodiment is:

In the first layer, 64 convolutional filters of size 3×3 are used twice in a row, with a stride of 1 and a padding of 1, resulting in two 600×300×64 convolutional layers (Conv1_1, Conv1_2) , after obtaining the output of the convolutional layer, use the BN layer (batch normalization) for normalization, then use the ReLU function (Rectified Linear Units) as the nonlinear activation function for activation, and finally use a window size of 2 × 2. The max pooling layer (Maxpooling) performs pooling, and the sampling stride of the max pooling layer (Maxpooling) is 2.

In the second layer, 128 convolutional filters of size 3×3 are used twice consecutively, the stride is 1, and the padding is 1, resulting in two convolutional layers of 300×150×128 (Conv2_1, Conv2_2) , after obtaining the output of the convolutional layer, use the BN layer (batch normalization) for normalization, then use the ReLU function (Rectified LinearUnits) as the nonlinear activation function for activation, and finally use a window size of 2 × 2 maximum The pooling layer (Maxpooling) performs pooling, and the sampling stride of the maximum pooling layer (Maxpooling) is 2.

In the third layer, 256 convolutional filters of size 3×3 are used continuously three times, the stride is 1, and the padding is 1, and three convolutional layers of 150×75×256 are obtained (Conv3_1, Conv3_2, Conv3_3 ), after obtaining the output of the convolutional layer, use the BN layer (batch normalization) for normalization, then use the ReLU function (Rectified LinearUnits) as the nonlinear activation function for activation, and finally use a window size of 2 × 2. The maximum pooling layer (Maxpooling) performs pooling, and the sampling stride of the maximum pooling layer (Maxpooling) is 2.

The fourth layer uses 512 convolution filters with a secondary size of 3×3, 512 convolution filters with a primary size of 1×1, and 512 convolution filters with a primary size of 3×3. The width is 1, the padding is 1, and three 76×38×512 convolutional layers (Conv4_1, Conv4_2, Add4_3, Conv4_3) are obtained. After obtaining the output of the convolutional layer, the BN layer (batch normalization) is used for normalization. Unification processing, and then use the ReLU function (Rectified LinearUnits) as the nonlinear activation function for activation, and finally use a maximum pooling layer (Maxpooling) with a window size of 2 × 2 for pooling, the maximum pooling layer (Maxpooling) The sampling stride is 2.

In the fifth layer, 512 convolutional filters of size 3×3 are used three times in a row, the stride is 1, and the padding is 1, and three convolutional layers of 38×19×512 are obtained (Conv5_1, Conv5_2, Conv5_3 ), after obtaining the output of the convolutional layer, use the BN layer (batch normalization) for normalization, and then use the ReLU function (Rectified LinearUnits) as the nonlinear activation function for activation.

Next, use 1024 convolution filters of size 3×3 for the output of Conv5_3, with a stride of 1 and a padding of 1, to obtain an Fc6 layer of size 38×19×1024, and then use size for the Fc6 layer is 1 × 1 1024 convolutional filters with stride of 1 and padding of 1 to obtain an Fc7 layer of size 38 × 19 × 1024.

Finally, four convolutional layers are added after the Fc7 layer, namely the Conv8 layer of size 20×10×512, the Conv9 layer of 10×5×256, the Conv10 layer of 6×3×256, and the size of 1×1×256. Conv11 layer.

The training error of the pre-trained model in step S4 is less than 15%, and the average value of the test error is less than 20%. The model training error is calculated as follows:

Step 1: Match each ground-truth box with the default bounding box corresponding to the maximum jaccard coefficient overlap, and match the default bounding box with any ground-truth box whose jaccard coefficient overlap is greater than 0.5;

Step 2: i represents the default box number, j represents the real box number, p represents the category number, 0 is the background, 1 is the pod, 2 is the stem, For matching, if the overlap with the maximum jaccard coefficient of the real box is greater than the threshold, the matching value is 1, otherwise it is 0.

Step 3: The total objective loss function L(x, c, l, g) is obtained by the weighted summation of the localization loss L _loc and the confidence loss L _conf :

where N is the number of default bounding boxes matching the ground truth, L _loc is the localization loss, L _conf is the confidence loss, x is the training sample, c is the confidence of each type of object, and l is the prediction frame , g represents the real frame, α represents the weight, and α in this embodiment is set to 0.8;

In the localization loss _Lloc , f(x) is a piecewise smoothing function controlled by σ, d represents the default box, w represents the width of the ground truth or default bounding box, h represents the height of the ground truth or default bounding box, and i represents the The i-th default box, j represents the j-th real box, and m represents the location information of the real box or the default bounding box (where cx represents the x-axis coordinate of the center point; cy represents the y-axis coordinate of the center point; w represents the width of the box; h represents the height of the box), p represents the p-th category:

In the formula,

The confidence loss L _conf is a multi-class softmax loss function, as shown in Equation 8. in, Represents the i-th default box and the j-th real box corresponding to the predicted probability of category p. The calculation formula is as follows:

The method of the invention can accurately detect and locate the pods and stems in the soybean plant image, has strong anti-interference ability against interference such as occlusion and overlap, and improves the accuracy of detection of the soybean plant stems and pods. The identification result of the sample image is shown in FIG. 3 .

Obviously, the above-mentioned embodiments of the present invention are only examples for clearly illustrating the present invention, rather than limiting the embodiments of the present invention. For those of ordinary skill in the art, changes or modifications in other different forms can also be made on the basis of the above description. There is no need and cannot be exhaustive of all implementations here. Any modifications, equivalent replacements and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the claims of the present invention.

Claims (7)

1. A soybean plant stem pod identification method based on a convolution network of an SSD is characterized by comprising the following steps:

s1, fixing a Canon 5D Mark II camera at a position 120cm away from blue background cloth to acquire a single soybean sample image to obtain a soybean plant image library;

s2, traversing all sample images in the image library in the step S1, manually marking pods and stems for each sample image, marking the tips of the pods which are not shielded as pods, marking the exposed parts of the stems as stems, obtaining an original image library, and repeatedly dividing the image library into a training set, a verification set and a test set;

and S3, carrying out random image enhancement and data amplification on the labeled training set image in the step S2. Carrying out image enhancement by adopting self-adaptive histogram equalization; randomly adjusting RGB color channels within a certain threshold value, turning horizontal and vertical mirror images, randomly rotating and translating to amplify data, centrally intercepting the rotated and translated images, discarding labels if the target of the processed images exceeds the boundary, and obtaining an enhanced amplification training set;

s4, constructing an SSD convolutional network, and carrying out multi-scale detection by using feature maps of different layers;

s5, conveying the training samples in the steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and meanwhile determining learning parameters in the SSD convolutional neural network;

and S6, transmitting the test set to the trained SSD convolutional neural network for recognition test, and taking a classification result with the confidence level of more than 40% in a recognition result as an output recognition result of the test sample.

2. The method of claim 1, wherein in step S2, each sample image is manually labeled with pods and stalks, wherein the pod tips that are not blocked are labeled with pods and the stalk bare parts are labeled with stalks.

3. The SSD-based convolutional network soybean plant stem pod identification method of claim 1, wherein the labeled training set images of step S3 are subjected to random image enhancement and data amplification. Carrying out image enhancement by adopting self-adaptive histogram equalization; the method comprises the steps of carrying out data amplification by random adjustment of RGB color channels within a certain threshold value and horizontal and vertical mirror image turning and random rotation and translation operations, carrying out centered interception on images subjected to rotation and translation, and discarding labels if targets of the images subjected to processing exceed boundaries.

4. The method for identifying soybean plant stems based on the SSD convolutional network as claimed in claim 1, wherein the SSD model in step S4 is constructed by adding a fusion layer and four convolutional layers in the VGG-16 network, and the training model in step S4 is constructed by the following steps:

s41, taking the soybean plant sample image as input, and carrying out convolution operation on the image in a convolution layer to obtain a characteristic diagram;

s42, adding an Add4_3 layer to the VGG-16 network, wherein the Add4_3 is formed by fusing (adding) two feature maps of Maxpool3 and Conv4_2, activating by ReLU and normalizing by Batch Normalization (BN), the Add4_3 is used as the input of the Conv4_3 layer, the feature maps of the Conv4_3 layer, the Fc7 layer and the Conv8_ 2-Conv 11_2 layers in the network are convolved by a 3 x 3 convolution kernel, and the confidence coefficient for output classification and the positioning information for output regression are respectively realized;

and S43, combining all the output structures, and obtaining a detection result through non-maximum suppression processing.

5. The SSD-based convolutional network soybean plant stem pod identification method of claim 4, wherein upon outputting confidence for classification, each frame generates two categories of confidence; when the regression positioning information is output, four coordinate values (x, y, w, h) are generated for each frame.

6. The SSD-based convolutional network soybean plant stem pod identification method of claim 4, wherein the profile in step S41 is calculated as follows:

step 1: dividing a characteristic diagram output by a Conv4_3 layer into 76 × 38 units, wherein each unit uses four default boundary boxes, each default boundary box uses a convolution kernel with the size of 3 × 3 to perform convolution operation, and outputs four elements of a frame, namely, a horizontal coordinate x and a vertical coordinate y at the upper left corner of the output frame, the width w and the height h of the frame output by a frame regression layer, and the confidence degrees that objects in the frame belong to pods and stalks respectively;

step 2: calculating the characteristic diagrams output by the Fc7 layer and the Conv8_ 2-Conv 11_2 layer in sequence according to the same method in the step S411; each layer feature map is divided into 38 × 19, 20 × 10, 10 × 5, 6 × 3 and 1 × 1 units, and the default bounding boxes used by each unit are 6, 4 and 4 respectively.

7. The SSD-convolutional-network-based soybean plant stem pod identification method of claim 1, wherein the pre-trained model in step S4 has training errors of less than 15% and the average of the test errors is less than 20%.