CN109684967A - A kind of soybean plant strain stem pod recognition methods based on SSD convolutional network - Google Patents

Abstract

The present invention provides a kind of soybean plant strain stem pod recognition methods based on SSD convolutional network, comprising the following steps: acquisition single plant soybean sample image obtains soybean plant strain image library；Manual mark is carried out to stem pod, beanpod marks the beanpod tip not being blocked, and stalk marks stalk exposed part, and image library is divided into training set, verifying collection, test set without duplicate；Random image enhancing and data amplification are carried out to the training set image marked, and automatic mark again increases image newly；SSD convolutional network is constructed, with the characteristic pattern of different levels, carries out multiple scale detecting；The training set is randomly selected to the training for being used for SSD convolutional neural networks, and determines the learning parameter in the SSD convolutional neural networks；Test set is transported in trained SSD convolutional neural networks and carries out identification test, and by recognition result label in the original image in the test sample.A kind of soybean plant strain stem pod recognition methods based on SSD convolutional network is provided, intelligentized identification is carried out to soybean plant strain stem pod by network training, high degree of automation effectively improves the efficiency to the detection of soybean plant strain stem pod.

Description

Soybean plant stem pod identification method based on SSD convolutional network

Technical Field

The invention relates to the technical field of computer image processing and identification methods, in particular to a soybean plant stem pod identification method based on a convolution network of an SSD.

Background

Soybean is an important crop of grain and oil in the world and also a main source of high-quality protein for human beings. Is one of the main crops in China and is the crop with the most economic benefit. The soybean seed testing work collects, sorts and counts the data of the whole soybean character, and is an important link in the genetic breeding process of soybean crops. At present, the work of soybean seed test mainly adopts manual operation, however, manual operation not only consumes a large amount of manpower and materials, has human error simultaneously, brings later data analysis's inaccuracy.

Disclosure of Invention

The invention aims to overcome the defects of the prior art and provides a soybean plant stem pod identification method based on a convolution network of an SSD (solid State disk), which applies machine vision to soybean seed test work, can improve the character data precision of soybean plants, reduce personal errors, shorten the seed test period, reduce the labor intensity and develop towards intellectualization, rapidness and accuracy.

In order to solve the technical problems, the invention adopts the technical scheme that:

provided is a soybean plant stem pod identification method based on a convolution network of an SSD, comprising the following steps:

s1, fixing a Canon 5D Mark II camera at a position 120cm away from blue background cloth to acquire a single soybean sample image to obtain a soybean plant image library;

s2, traversing all sample images in the image library in the step S1, manually marking pods and stems for each sample image, marking the tips of the pods which are not shielded as pods, and marking the exposed parts of the stems as stems to obtain an original image set;

and S3, carrying out random image enhancement and data amplification on the labeled training set image in the step S2. Carrying out image enhancement by adopting self-adaptive histogram equalization; randomly adjusting RGB color channels within a certain threshold value, turning horizontal and vertical mirror images, randomly rotating and translating to amplify data, centrally intercepting the rotated and translated images, discarding labels if the target of the processed images exceeds the boundary, and obtaining an enhanced amplification training set;

s4, constructing an SSD convolutional network, and carrying out multi-scale detection by using feature maps of different layers;

s5, conveying the training samples in the steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and meanwhile determining learning parameters in the SSD convolutional neural network;

and S6, transmitting the test set to the trained SSD convolutional neural network for recognition test, and taking a classification result with the confidence level of more than 40% in a recognition result as an output recognition result of the test sample.

According to the soybean plant stem pod identification method based on the SSD convolutional network, a fusion layer is added to directly transfer details of a lower layer to a residual error structure of a higher layer, the characteristic that the SSD network selects characteristic graphs of different layers to perform multi-scale detection is fully utilized, and the defect that the existing method is poor in detection accuracy of deformed, shielded and continuously overlapped objects is overcome. The method has the advantages of a convolutional neural network, reduces the interference of image background and environment brightness, has stronger anti-interference capability on shielding and overlapping, and improves the accuracy of detecting the soybean plant stem pods.

Preferably, in step S2, the pods and stems of each sample image are manually labeled, the pod tips that are not shielded are labeled as pods, and the stem exposed parts are labeled as stems. Only part of characteristics of the target are marked, so that the identification accuracy under the shielding and overlapping conditions can be improved.

Preferably, random image enhancement and data augmentation are performed on the labeled training set images described in step S2. Carrying out image enhancement by adopting self-adaptive histogram equalization; the method comprises the steps of carrying out data amplification by random adjustment of RGB color channels within a certain threshold value and horizontal and vertical mirror image turning and random rotation and translation operations, carrying out centered interception on images subjected to rotation and translation, and discarding labels if targets of the images subjected to processing exceed boundaries.

Preferably, the SSD model described in step S4 is built by adding one fusion layer and four convolutional layers in the VGG-16 network, and the building steps of the training model in step S4 are as follows:

s41, taking the soybean plant sample image as input, and carrying out convolution operation on the image in a convolution layer to obtain a characteristic diagram;

s42, adding an Add4_3 layer to the VGG-16 network, wherein the Add4_3 is formed by fusing (adding) two feature maps of Maxpool3 and Conv4_2, activating by ReLU and normalizing by Batch Normalization (BN), the Add4_3 is used as the input of the Conv4_3 layer, the feature maps of the Conv4_3 layer, the Fc7 layer and the Conv8_ 2-Conv 11_2 layers in the network are convolved by a 3 x 3 convolution kernel, and the confidence coefficient for output classification and the positioning information for output regression are respectively realized;

and S43, combining all the output structures, and obtaining a detection result through non-maximum suppression processing.

By selecting six feature maps of different levels for multi-scale detection and increasing fusion of the feature maps of the lower layer on the basis of keeping detection of the feature maps of the higher layer, the abundant image detail information of the feature maps of the lower layer is fully utilized, the target detection robustness effect of the anti-interference capability is achieved, and the detection and positioning problems under the conditions of deformation, shielding, continuous overlapping and the like are solved.

Preferably, when the confidence for classification is output, each frame generates the confidence of two classes; when the regression positioning information is output, four coordinate values (x, y, w, h) are generated for each frame.

Preferably, the characteristic map in step S41 is calculated as follows:

step 1: dividing a characteristic diagram output by a Conv4_3 layer into 76 × 38 units, wherein each unit uses four default boundary boxes, each default boundary box uses a convolution kernel with the size of 3 × 3 to perform convolution operation, and outputs four elements of a frame, namely, a horizontal coordinate x and a vertical coordinate y at the upper left corner of the output frame, the width w and the height h of the frame output by a frame regression layer, and the confidence degrees that objects in the frame belong to pods and stalks respectively;

step 2: calculating the characteristic diagrams output by the Fc7 layer and the Conv8_ 2-Conv 11_2 layer in sequence according to the same method in the step S411; each layer feature map is divided into 38 × 19, 20 × 10, 1 × 5, 6 × 3 and 1 × 1 units, and the default bounding boxes used by each unit are 6, 4 and 4 respectively.

Preferably, the training error of the pre-trained model in step S4 is less than 15%, and the average value of the test error is less than 20%.

Compared with the prior art, the invention has the beneficial effects that:

(1) the invention can realize the detection and the positioning of the soybean plant stem pod, has the advantages of higher accuracy, good stability, strong anti-interference capability, high universality and the like, has high detection precision on deformed, shielded and continuously overlapped targets, and can be applied to a soybean plant character detection system.

(2) The method has the advantages of a convolutional neural network, reduces the interference of image background and environment brightness, has stronger anti-interference capability on shielding and mutual overlapping, and improves the accuracy of soybean plant stem pod detection.

Drawings

FIG. 1 is a flow chart of a method of soybean plant stem pod identification based on a convolution network of SSD;

fig. 2 is a detailed flowchart of step S4;

FIG. 3 is a sample image of soybean plants identified in the present invention.

Detailed Description

The drawings are for illustrative purposes only and are not to be construed as limiting the patent; for a better understanding of the embodiments, the drawings may be omitted, enlarged or reduced and do not represent actual dimensions; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.

The invention is further described with reference to the following figures and detailed description.

Referring to fig. 1, the present embodiment is a first embodiment of a method for identifying soybean plant stem pods based on SSD convolutional network of the present invention, and comprises the following steps:

s1, fixing a Canon 5D Mark II camera at a position 120cm away from blue background cloth to acquire a single soybean sample image to obtain a soybean plant image library;

s2, traversing all sample images in the image set in the step S1, manually labeling pods and stems of each sample image, labeling the tips of the pods which are not shielded into pods, labeling the exposed parts of the stems into stems, and obtaining an original training set;

and S3, carrying out random image enhancement and data amplification on the labeled training set image in the step S2. Carrying out image enhancement by adopting self-adaptive histogram equalization; randomly adjusting RGB color channels within a certain threshold value, turning horizontal and vertical mirror images, randomly rotating and translating to amplify data, centrally intercepting the rotated and translated images, discarding labels if the target of the processed images exceeds the boundary, and obtaining an enhanced amplification training set;

referring to fig. 2, in step s4 of this embodiment, an SSD convolutional network is constructed, and multi-scale detection is performed by using feature maps of different levels;

s5, conveying the training samples in the steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and meanwhile determining learning parameters in the SSD convolutional neural network;

and S6, transmitting the test set to the trained SSD convolutional neural network for recognition test, and taking a classification result with the confidence level of more than 40% in a recognition result as an output recognition result of the test sample.

In step S1, a Canon 5D Mark II camera is fixed at a position 120cm away from the blue background cloth to acquire a single soybean sample image, so as to obtain a soybean plant image library. Specifically, the plant sample image set stores sample data in the following form:

{image_name，x，y}

wherein, image _ name represents the name of the soybean plant image, x represents the horizontal pixel value of the image, and y represents the vertical pixel value of the image.

In step S2, traversing all sample images in the image library in step S1, performing manual labeling of pods and stems on each sample image, labeling the tips of the pods which are not shielded as pods, and labeling the exposed parts of the stems as stems, thereby obtaining an original image set. Specifically, the plant sample image is used for labeling each real frame to form an image labeling set, and the image labeling set stores labeling data in the following form:

{label，xmin，ymin，xmax，ymax}

wherein label represents the labeled category, xmin represents the abscissa of the labeled minimum pixel point, ymin represents the ordinate of the labeled minimum pixel point, xmax represents the abscissa of the labeled maximum pixel point, and ymax represents the ordinate of the labeled maximum pixel point.

In step S4, the pre-trained model is established as follows:

s41, taking the soybean plant sample image as input, and carrying out convolution operation on the image in the convolution layer to obtain a characteristic diagram;

s42, adding an Add4_3 layer to the VGG-16 network, wherein the Add4_3 is formed by fusing (adding) two feature maps of Maxpool3 and Conv4_2, activating by ReLU and normalizing by Batch Normalization (BN), the Add4_3 is used as the input of the Conv4_3 layer, the feature maps of the Conv4_3 layer, the Fc7 layer and the Conv8_ 2-Conv 11_2 layers in the network are convolved by a 3 x 3 convolution kernel, and the confidence coefficient for output classification and the positioning information for output regression are respectively realized;

s43, combining all output structures, and obtaining a detection result through non-maximum suppression processing; wherein the confidence for output classification is the confidence of the class in each prediction box; the positioning information for output regression is four coordinate values (x, y, w, h) for each prediction frame.

The feature map in step S41 is calculated as follows:

step 1: dividing a characteristic diagram output by a Conv4_3 layer into 76 × 38 units, wherein each unit uses four default boundary boxes, each default boundary box uses a convolution kernel with the size of 3 × 3 to perform convolution operation, and outputs four elements of a frame, namely, a horizontal coordinate x and a vertical coordinate y at the upper left corner of the output frame, the width w and the height h of the frame output by a frame regression layer, and the confidence degrees that objects in the frame belong to pods and stalks respectively;

step 2: calculating the characteristic diagrams output by the Fc7 layer and the Conv8_ 2-Conv 11_2 layer in sequence according to the same method in the step S411; each layer feature map is divided into 38 × 19, 20 × 10, 10 × 5, 6 × 3 and 1 × 1 units, and the default bounding boxes used by each unit are 6, 4 and 4 respectively.

By selecting six feature maps of different levels for multi-scale detection and increasing fusion of the feature maps of the lower layer on the basis of keeping detection of the feature maps of the higher layer, the abundant image detail information of the feature maps of the lower layer is fully utilized, the target detection robustness effect of the anti-interference capability is achieved, and the detection and positioning problems under the conditions of deformation, shielding, continuous overlapping and the like are solved.

The VGG-16 partial network structure added with the residual structure in this embodiment is:

the first layer, using two consecutive 64 convolution filters with size of 3 × 3, with step of 1 and padding of 1, obtains two 600 × 300 × 64 convolution layers (Conv1_1, Conv1_2), after obtaining the output of the convolution layers, uses BN layer (batch normalization) to perform normalization, then uses ReLU function (Rectified Linear Units) as nonlinear activation function to perform activation, and finally uses a maximum pooling layer (maxporoling) with window size of 2 × 2 to perform pooling, and the sampling step of the maximum pooling layer (maxporoling) is 2.

And a second layer, in which 128 convolution filters of 3 × 3 size are continuously used twice, the step is 1, padding (padding) is 1, two convolution layers (Conv2_1, Conv2_2) of 300 × 150 × 128 are obtained, after the output of the convolution layers is obtained, normalization processing is performed by using a BN layer (batch normalization), then activation is performed by using a ReLU function (Rectified linear units) as a nonlinear activation function, and finally pooling is performed by using a maximum pooling layer (maxporoling) of 2 × 2 window size, and the sampling step of the maximum pooling layer (maxporoling) is 2.

And a third layer, in which 256 convolution filters with a size of 3 × 3 are continuously used three times, the step is 1, padding (padding) is 1, three 150 × 75 × 256 convolution layers (Conv3_1, Conv3_2, Conv3_3) are obtained, after the output of the convolution layers is obtained, normalization processing is performed by using a BN layer (batch normalization), then activation is performed by using a Rectified linear units (Rectified linear units) as a nonlinear activation function, and finally pooling is performed by using a maximum pooling layer (maxporoling) with a window size of 2 × 2, and the sampling of the maximum pooling layer (maxporoling) is 2.

And a fourth layer, which uses 512 convolution filters with a quadratic size of 3 × 3, 512 convolution filters with a primary size of 1 × 1, 512 convolution filters with a primary size of 3 × 3, stride of 1, padding of 1 to obtain three 76 × 38 × 512 convolution layers (Conv4_1, Conv4_2, Add4_3, and Conv4_3), obtains outputs of the convolution layers, performs normalization processing using a BN layer (batch normalization), activates using a ReLU function (Rectified linear units) as a nonlinear activation function, and performs pooling using a maximum pooling layer (max pooling) with a window size of 2 × 2, wherein the sampling stride of the maximum pooling layer (max pooling) is 2.

And a fifth layer, using 512 convolution filters with the size of 3 × 3 in succession, the step being 1, and padding (padding) being 1, obtaining three convolution layers (Conv5_1, Conv5_2, Conv5_3) with the size of 38 × 19 × 512, obtaining the output of the convolution layers, normalizing the output by using a BN layer (batch normalization), and then activating the output by using a ReLU function (Rectified linear units) as a nonlinear activation function.

Next, 1024 convolution filters of 3 × 3 size were used for the output of Conv5_3, the step size was 1, padding (padding) was 1, and the Fc6 layer of 38 × 19 × 1024 size was obtained, and further 1024 convolution filters of 1 × 1 size were used for the Fc6 layer, the step size was 1, and padding (padding) was 1, and the Fc7 layer of 38 × 19 × 1024 size was obtained.

Finally, four convolutional layers were added behind the Fc7 layer, 20 × 10 × 512 Conv8, 10 × 5 × 256 Conv9, 6 × 3 × 256 Conv10, and 1 × 1 × 256 Conv11, respectively.

In step S4, the training error of the pre-trained model is less than 15%, and the average value of the test error is less than 20%. The calculation method of the model training error is as follows:

step 1: matching each real frame with a default bounding box corresponding to the maximum jaccard coefficient overlap, and matching the default bounding box with any real frame with the jaccard coefficient overlap larger than 0.5;

step 2: i denotes the default box number, j denotes the real box number, p denotes the category number, 0 is the background, 1 is the pod, 2 is the stem,if the matching is judged to be matched, the maximum jaccard coefficient overlapping with the real frame is larger than the threshold value, the matching value is 1, and if not, the matching value is 0.

And step 3: the total target loss function L (x, c, L, g) is determined by the localization loss L_locAnd confidence loss L_confThe weighted sum yields:

wherein N is the number of default bounding boxes matched with the real box, L_locFor loss of alignment, L_confFor confidence loss, x represents the training sample, c represents the confidence of each class of object, l represents the prediction box, g represents the real box, α represents the weight, α in this embodiment is set to 0.8;

loss of positioning L_locWhere f (x) is a piecewise smoothing function controlled by σ, d represents a default box, w represents the width of a real box or a default bounding box, h represents the height of the real box or the default bounding box, i represents an ith default box, j represents a jth real box, m represents position information of the real box or the default bounding box (where cx represents a center point x-axis coordinate, cy represents a center point y-axis coordinate, w represents the width of the box, h represents the height of the box), and p represents a p-th category:

in the formula,

loss of confidence L_confIs a multi-class softmax loss function, as in equation 8. Wherein,the prediction probability of the ith default frame and the jth real frame corresponding to the category p is represented by the following calculation formula:

the method can accurately detect and position the pods and the stems in the soybean plant image, has stronger anti-interference capability on interference such as shielding, overlapping and the like, and improves the accuracy of pod detection of the soybean plant. Wherein, the identification result of the sample image is shown in fig. 3.

It should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.

Claims (7)

1. A soybean plant stem pod identification method based on a convolution network of an SSD is characterized by comprising the following steps:

s1, fixing a Canon 5D Mark II camera at a position 120cm away from blue background cloth to acquire a single soybean sample image to obtain a soybean plant image library;

s2, traversing all sample images in the image library in the step S1, manually marking pods and stems for each sample image, marking the tips of the pods which are not shielded as pods, marking the exposed parts of the stems as stems, obtaining an original image library, and repeatedly dividing the image library into a training set, a verification set and a test set;

and S3, carrying out random image enhancement and data amplification on the labeled training set image in the step S2. Carrying out image enhancement by adopting self-adaptive histogram equalization; randomly adjusting RGB color channels within a certain threshold value, turning horizontal and vertical mirror images, randomly rotating and translating to amplify data, centrally intercepting the rotated and translated images, discarding labels if the target of the processed images exceeds the boundary, and obtaining an enhanced amplification training set;

s4, constructing an SSD convolutional network, and carrying out multi-scale detection by using feature maps of different layers;

s5, conveying the training samples in the steps S2 and S3 to the SSD convolutional neural network for pre-training and iterative pre-training to obtain a pre-trained model, and meanwhile determining learning parameters in the SSD convolutional neural network;

and S6, transmitting the test set to the trained SSD convolutional neural network for recognition test, and taking a classification result with the confidence level of more than 40% in a recognition result as an output recognition result of the test sample.

2. The method of claim 1, wherein in step S2, each sample image is manually labeled with pods and stalks, wherein the pod tips that are not blocked are labeled with pods and the stalk bare parts are labeled with stalks.

3. The SSD-based convolutional network soybean plant stem pod identification method of claim 1, wherein the labeled training set images of step S3 are subjected to random image enhancement and data amplification. Carrying out image enhancement by adopting self-adaptive histogram equalization; the method comprises the steps of carrying out data amplification by random adjustment of RGB color channels within a certain threshold value and horizontal and vertical mirror image turning and random rotation and translation operations, carrying out centered interception on images subjected to rotation and translation, and discarding labels if targets of the images subjected to processing exceed boundaries.

4. The method for identifying soybean plant stems based on the SSD convolutional network as claimed in claim 1, wherein the SSD model in step S4 is constructed by adding a fusion layer and four convolutional layers in the VGG-16 network, and the training model in step S4 is constructed by the following steps:

s41, taking the soybean plant sample image as input, and carrying out convolution operation on the image in a convolution layer to obtain a characteristic diagram;

s42, adding an Add4_3 layer to the VGG-16 network, wherein the Add4_3 is formed by fusing (adding) two feature maps of Maxpool3 and Conv4_2, activating by ReLU and normalizing by Batch Normalization (BN), the Add4_3 is used as the input of the Conv4_3 layer, the feature maps of the Conv4_3 layer, the Fc7 layer and the Conv8_ 2-Conv 11_2 layers in the network are convolved by a 3 x 3 convolution kernel, and the confidence coefficient for output classification and the positioning information for output regression are respectively realized;

and S43, combining all the output structures, and obtaining a detection result through non-maximum suppression processing.

5. The SSD-based convolutional network soybean plant stem pod identification method of claim 4, wherein upon outputting confidence for classification, each frame generates two categories of confidence; when the regression positioning information is output, four coordinate values (x, y, w, h) are generated for each frame.

6. The SSD-based convolutional network soybean plant stem pod identification method of claim 4, wherein the profile in step S41 is calculated as follows:

step 1: dividing a characteristic diagram output by a Conv4_3 layer into 76 × 38 units, wherein each unit uses four default boundary boxes, each default boundary box uses a convolution kernel with the size of 3 × 3 to perform convolution operation, and outputs four elements of a frame, namely, a horizontal coordinate x and a vertical coordinate y at the upper left corner of the output frame, the width w and the height h of the frame output by a frame regression layer, and the confidence degrees that objects in the frame belong to pods and stalks respectively;

step 2: calculating the characteristic diagrams output by the Fc7 layer and the Conv8_ 2-Conv 11_2 layer in sequence according to the same method in the step S411; each layer feature map is divided into 38 × 19, 20 × 10, 10 × 5, 6 × 3 and 1 × 1 units, and the default bounding boxes used by each unit are 6, 4 and 4 respectively.

7. The SSD-convolutional-network-based soybean plant stem pod identification method of claim 1, wherein the pre-trained model in step S4 has training errors of less than 15% and the average of the test errors is less than 20%.