CN111369540B - Plant leaf disease identification method based on mask convolutional neural network - Google Patents
Plant leaf disease identification method based on mask convolutional neural network Download PDFInfo
- Publication number
- CN111369540B CN111369540B CN202010150980.9A CN202010150980A CN111369540B CN 111369540 B CN111369540 B CN 111369540B CN 202010150980 A CN202010150980 A CN 202010150980A CN 111369540 B CN111369540 B CN 111369540B
- Authority
- CN
- China
- Prior art keywords
- mask
- disease
- target
- network
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 201000010099 disease Diseases 0.000 title claims abstract description 97
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 title claims abstract description 97
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 42
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 45
- 238000012360 testing method Methods 0.000 claims abstract description 31
- 238000001514 detection method Methods 0.000 claims abstract description 29
- 238000012216 screening Methods 0.000 claims abstract description 18
- 230000011218 segmentation Effects 0.000 claims abstract description 16
- 230000002708 enhancing effect Effects 0.000 claims abstract description 4
- 239000013598 vector Substances 0.000 claims description 19
- 238000012545 processing Methods 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 230000009466 transformation Effects 0.000 claims description 4
- 238000013519 translation Methods 0.000 claims description 4
- 238000001228 spectrum Methods 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 2
- 238000002372 labelling Methods 0.000 claims description 2
- 238000000844 transformation Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims 1
- 241000196324 Embryophyta Species 0.000 description 27
- 238000013135 deep learning Methods 0.000 description 4
- 240000008067 Cucumis sativus Species 0.000 description 3
- 235000010799 Cucumis sativus var sativus Nutrition 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000004088 simulation Methods 0.000 description 3
- 230000006378 damage Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000009313 farming Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 241001270131 Agaricus moelleri Species 0.000 description 1
- 241000221785 Erysiphales Species 0.000 description 1
- 206010027146 Melanoderma Diseases 0.000 description 1
- 241000233679 Peronosporaceae Species 0.000 description 1
- 241000700605 Viruses Species 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000001580 bacterial effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 235000013399 edible fruits Nutrition 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 230000005541 medical transmission Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G06T5/73—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
- Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a plant leaf disease identification method based on a mask convolutional neural network, which mainly solves the problem of low accuracy in identifying plant leaf diseases in the prior art. The scheme is as follows: enhancing and expanding the original data set to obtain a training set and a testing set; carrying out semantic segmentation on the training set and the test set to obtain corresponding mask sets; a disease feature screening module is added between a full convolution layer and a mask branch of the model, and a training set and a mask set are input into a network for training to obtain target classification and target detection results; taking a feature map belonging to the disease blade in the target classification result as the input of a mask branch, and obtaining a trained model after multiple iterations; inputting the test set into the model, classifying and detecting the targets of the blades, and dividing the blades belonging to the disease category. The invention improves the accuracy of identifying the leaf diseases on the basis of the traditional method, and can be used for identifying and dividing the leaf diseases of plants in agricultural planting.
Description
Technical Field
The invention belongs to the technical field of image processing, and particularly relates to a plant leaf disease identification method which can be used for cutting and identifying plant leaf disease in agricultural planting.
Background
In modern intelligent agriculture, crop diseases are a great threat to grain safety, and plant diseases cause serious damage to crops by significantly reducing yield. Among them, early blight is a typical disease that may severely reduce yield. Similarly, in humid climates, late blight is another very damaging disease that affects the leaves, stems and fruits of plants. Protection of plants from disease is critical to ensure quality and quantity of the crop. The protection of crops should begin with early disease discovery in order to select the appropriate treatment at the correct time to prevent disease transmission. The disease types for identifying greenhouse plant diseases mainly comprise bacterial spots, early blight, late blight, leaf mold, black spot and other diseases. However, in practice, it is difficult to accurately determine the type of diseases due to the large number of diseases and similar behavior on the leaves.
Currently, disease studies on plant leaves mainly involve their detection and classification using image processing or deep learning methods. In the control of Plant diseases, foreign Plant Village team proposed a method for detecting and classifying diseases through Plant leaves by deep learning in 2016, classifying specific Plant diseases through smart phones mainly in a simple background, recognizing colors, gray scales and segmented pictures under different proportion data sets and different networks mainly through a deep learning method, but the method can only classify Plant disease leaves and cannot segment positions of Plant disease leaves and diseases.
The same year, mads dymann 1 et al, proposed a convolutional neural network to classify plant species in color images. The network was constructed de novo, trained and tested. These images are from six different local datasets, data acquisition is performed in terms of illumination, resolution and soil type at different growth stages, but with lower accuracy and no detection and classification of plant disease.
In 2018 Liu Na et al, image processing technology and artificial neural network technology are applied to realize the detection of cucumber leaf diseases and classification of the degrees of infection, and experimental researches are mainly carried out on cucumber downy mildew, powdery mildew and virus diseases with high morbidity and serious harm, but the number of training samples is small, the number of the identified diseases is small, the test accuracy is low, and the fitting is easy to generate.
2017, he Kaiming proposes a Mask convolutional neural network Mask R-CNN architecture, in which a branch is added to two branches of a fast R-CNN, namely classification and coordinate regression, so as to perform semantic segmentation, image features are extracted mainly by using a residual network res net101/50 or a pyramid network FPN as a main network, foreground and background of a target area are obtained by using an area recommendation network RPN, classification and example segmentation results are obtained for the obtained target area and image features by using a full convolutional layer, and then semantic segmentation results are obtained by using convolutional network identification. The network is mainly used for target detection of COCO data sets, and is not used in the field of plant disease identification at present.
In summary, the current plant disease research mainly includes classification and identification of single plant disease category similar to cucumber, wherein the number of disease category and sample in the contained samples is less, and the identification accuracy is lower. The existing classification method for plant disease leaves through deep learning cannot separate the positions of the plant disease leaves and the diseases, and the identification rate is low.
Disclosure of Invention
The invention aims to solve the defects of the prior art, and provides a plant leaf disease identification method based on a mask convolutional neural network mask-CNN, so as to divide the positions of plant leaf diseases and diseases, and improve the accuracy and efficiency of identification.
In order to achieve the above purpose, the technical scheme of the invention comprises the following steps:
(1) Sequentially enhancing, expanding and semantically segmenting the original data set to obtain a training and testing image set and a mask set;
(2) Optimizing the Mask R-CNN network: namely, a disease feature screening module is added between a full convolution layer after the Mask R-CNN network ROI alignment and Mask branches;
(3) Training the optimized Mask R-CNN network:
(3a) Setting main network parameters:
selecting a backbone network from two residual networks, namely ResNet50 and ResNet 101;
setting the size of the epochs and the number of steps of each epoch training;
setting a receiving threshold T of a disease blade in a disease characteristic screening module 0 The other parameters are default values of Mask R-CNN;
(3b) According to known classification errors L cls Detection error L box And a segmentation error L mask The optimized Mask R-CNN network loss function is determined as follows: l (L) loss ＝L cls +L box +L mask ；
(3c) Inputting the training image set and the training Mask set into the optimized Mask R-CNN network for training to obtain a trained model;
(4) And inputting the test image into the trained model for testing.
Compared with the prior art, the invention has the following advantages:
first, compared with the GoogleNet and VGG methods, the method of the invention improves the identification accuracy of disease leaves when identifying plant leaf diseases.
Second, according to the invention, as Mask images of the target area are generated through Mask branches of the Mask R-CNN network model, plant disease leaves and disease positions thereof can be accurately extracted.
Thirdly, according to the invention, as the disease feature screening module is added in the network structure of the Mask R-CNN, the Mask branch is trained and tested only aiming at unhealthy blades, the burden of the Mask branch is reduced, and the network identification efficiency is improved on the premise of ensuring that the accuracy of the Mask R-CNN network is unchanged.
Fourth, in the present invention, the image transformation is adopted to increase the samples in the data set, and the adaptive contrast enhancement algorithm is adopted to enhance the data set, so as to improve the blurred image in the data set.
Drawings
FIG. 1 is a general flow diagram of an implementation of the present invention;
FIG. 2 is a schematic diagram of the overall structure of the optimized Mask R-CNN network according to the present invention;
FIG. 3 is a block diagram of a training sub-process for an optimized Mask R-CNN network in accordance with the present invention;
FIG. 4 is a training image and training binary mask image acquired in the present invention;
FIG. 5 is a test image of healthy and diseased leaves obtained in the present invention;
FIG. 6 is a resulting image of a healthy leaf identified by the simulation of the present invention;
FIG. 7 is a resulting image of a disease blade identified by the simulation of the present invention.
Detailed Description
Specific embodiments and effects of the present invention are described in further detail below with reference to the accompanying drawings:
the application environment of this example is the farming scene, and the purpose is to detect and discern the vegetation that has the disease in the farming, provides this kind of disease information for the planting personnel is more accurate.
Referring to fig. 1, the implementation steps of this example are as follows:
step 1, image enhanced dataset D 1 。
(1.1) from public item PlaDownloading ntVillage-Dataset to obtain plant disease leaf data set D 0 Pair D using adaptive contrast enhancement algorithm 0 Image enhancement is carried out to improve blurred images in the database, and a database D after image enhancement is obtained 1 ：
(1.1 a) acquiring the Low frequency part m of the image x (i, j) x (i, j) and a high-frequency part h x (i, j) obtaining a low frequency portion of the image by mean filtering:
h x (i,j)＝x(i,j)-m x (i,j)
wherein, (2n+1) 2 Representing a window size with (i, j) as coordinates of a center point of the image;
(1.1 b) multiplying the high frequency part of the image by the gain value G (I, j) of the part to obtain an amplified high frequency part I (I, j):
I(i,j)＝G(i,j)h x (i,j)
wherein G (I, j) can take a constant C greater than 1 to obtain an amplified high-frequency part I c (i, j) is:
I c (i,j)＝Ch x (i,j)
the gain value G (i, j) in this example is taken from the local mean square error sigma x (i, j) inversely proportional variation value
Wherein D is a constant and the local mean square error of the image is:
obtain amplified high-frequency part I σ (i, j) is:
(1.1 c) recombining the high frequency part and the low frequency part to obtain an enhanced image f (i, j):
f(i,j)＝m x (i,j)+I σ (i,j)；
step 2, acquiring a training image set D 3 And test image set D 4 And training mask set D 5 And test mask set D 6 。
(2.1) enhancing the image with the labeling tool labelme of semantic segmentation 1 Respectively plotting the image targets in the image database to generate masks of the targets to obtain a mask set D containing mask information and label information 2 The resulting training binary mask image is shown in fig. 4 b.
(2.2) Using image transformations on enhanced data set D 1 Sum mask set D 2 Sequentially performing translation, rotation and overturn to increase the data volume of the sample and obtain a data set D after expanding the sample 3 Sum mask set D 4 ；
(2.2) expanding the data set D 3 Sum mask set D 4 Dividing into a training image set D according to the proportion of 8:2 5 And test image set D 6 And training mask set D 7 And test mask set D 8 The training set image is shown in fig. 4a, and the test set image is shown in fig. 5a and 5 b;
and 3, continuously optimizing the Mask R-CNN network structure.
The existing Mask R-CNN network comprises a backbone network, an area recommendation generation network RPN, a full convolution layer, mask branches, namely a full convolution layer and a full connection layer, and a disease feature screening module is added between the full convolution layer and the Mask branches to obtain an optimized Mask R-CNN network structure, as shown in fig. 2.
Referring to fig. 2, the Mask R-CNN network structure after optimization in this example is: backbone network- & gt region recommended generation network RPN- & gt full convolution layer- & gt disease feature screening module- & gt full convolution layer- & gt full connection layer.
The disease characteristic screening module is used for judging the confidence degree T of the disease blade output by the full convolution layer 1 Disease leaf joint given by network initialization parametersThreshold T 0 And (3) screening out the characteristic spectrum of the disease leaves in the batch processing, and inputting the screening result into a mask branch, namely a full convolution layer and full connection for processing.
Step 4, training the optimized Mask R-CNN network to obtain a trained network model:
referring to fig. 3, the specific implementation of this step is as follows:
(4.1) setting main network parameters:
selecting a backbone network from two residual networks, namely ResNet50 and ResNet101, wherein ResNet101 is selected as the backbone network in the embodiment;
setting the number of labels to be 11 according to the types of the images in the database, wherein the number of the labels comprises 1 background label and 10 image labels;
setting the iteration times epoch of all samples to be 100, setting the iteration times of each epoch to be 100, setting the learning rate to be 0.001, and setting the weight attenuation coefficient to be 0.0001; setting the number of GPUs as 1, setting the number of images processed by each GPU as 2, and receiving threshold T of disease leaves 0 The other parameters are default values of Mask R-CNN;
(4.2) according to the known classification error L cls Detection error L box And a segmentation error L mask The optimized Mask R-CNN network loss function is determined as follows: l (L) loss ＝L cls +L box +L mask ；
(4.3) training the optimized Mask R-CNN network:
(4.3 a) initializing the network parameters in (3 a) to train the image set D 5 And training mask set D 7 Inputting the optimized Mask R-CNN network;
(4.3 b) extracting feature pattern F of the training image through the training residual network 0 ；
(4.3 c) feature map F 0 Inputting the target region into a region recommendation generation network RPN to obtain a foreground F of the target region 1 And background F 2 ；
(4.3 d) using the ROI alignment method to Align the foreground F of the target region 1 Mapping to feature map F 0 Generates a fixed sizeFeature map F of (1) 3 ：
First, a target region foreground F is calculated 1 Belonging to the characteristic layer:
wherein ,w0 and h0 Respectively representing the width and height of the target area, k 0 The value is 4;
second, in the target region foreground F 1 After finding out the corresponding characteristic layer k, obtaining the step length s corresponding to the characteristic layer;
then, the target area foreground F is calculated 1 Mapping to the width of a feature map
And high->
And obtaining a target region Z on the feature map according to the two parameters:
Z＝w 1 *h 1
then, the target region Z on the feature map is divided into n 2 Obtaining the divided target area Z i ，
Z i ＝w 2 *h 2 ,i＝1,2,…n 2
wherein ,w2 and h2 Represents Z i Width and height, respectively, of the size of
Then, each target zone Z i Dividing the image into four parts, obtaining pixel values of four points by taking the central point position of each part, and taking the maximum value of the four pixel values as each target area Z i To obtain n in the target region Z 2 The pixel values form a characteristic diagram with the size of n multiplied by n;
(4.3 e) feature map F 3 Obtaining a target classification result and a target detection result through the full convolution layer, and calculating a classification error L cls And detecting an error L box ：
Probability p corresponding to target classification result u u Obtaining the classification error: l (L) cls ＝-logp u ；
Let t be i u ＝{t x ,t y ,t w ,t h 4 parameterized coordinates, v, of the target detection result i ＝{v x ,v y ,v w ,v h The detection error L is calculated by the following formula, wherein the detection error L is the target translation scaling parameter box ：
Wherein smooths L1 The smoothed norm loss function is expressed as:
(4.3 f) judging whether the classification result belongs to the disease leaves:
if the target classification does not belong to the disease blade, continuing to judge the target classification of the next blade;
if the classification result belongs to the disease leaves, the confidence degree T of the classification result is obtained 1 And a disease leaf reception threshold T 0 Comparing;
when T is 1 ＞T 0 In the case of feature map F 3 Selecting confidence coefficient as T 1 Feature map F of (1) 4 ；
When T is 1 ＜＝T 0 If so, continuing to judge the target classification of the next blade;
(4.3 g) the feature map F selected in (4.3 d) 4 And training mask set D 7 Inputting the binary mask into a mask branch for training to obtain a binary mask of a target area, namely a segmentation result of a disease blade and a disease position of the disease blade:
first, the feature map F is transformed by deconvolution 3 Amplifying to obtain binary mask regions M of all classes k ；
Then, the binary mask areas of all categories are traversedM k Binary mask region M for each category i Applying Sigmoid activation functions
After the classification operation is carried out, a binary mask of a target area corresponding to the target classification is obtained, wherein the maximum probability y in the classification probability vector H is the maximum probability y;
(4.3 h) calculating the segmentation error L of the binary mask of the target region obtained in (4.3 e) mask ；
Where y is the predictive probability of the binary mask for the target region,
a true tag that is a binary mask of the target region;
(4.3 i) calculating the loss value L of the network loss ＝L cls +L box +L mask Back-propagating updating network weights after each iteration using the loss values;
(4.3 g) determining whether the number of iterations of all samples is greater than 100 times set:
if the iteration times of all the samples are more than 100, stopping the network training to obtain a trained network model;
and (3) repeating (4.3 c) to (4.3 g) until the iteration number of all samples is greater than 100 when the iteration number of all samples is less than or equal to 100.
And 5, obtaining a recognition result of the plant leaf diseases.
(5.1) the test image set D obtained in (1.2) 6 The healthy leaves in (a) are as shown in fig. 5a and the diseased leaves are as shown in fig. 5 b) are input into the trained network model, feature vectors are extracted, and feature vector A of each real target classification in n real leaf categories is calculated i Cosine similarity cos of feature vector B classified with prediction target i (θ) to obtain a classified probability vector P:
P＝{cos 1 (θ)，cos 2 (θ)，…，cos i (θ)，…cos n (θ)}，i＝1,2…,n
wherein, cosine similarity cos i The calculation formula of (θ) is as follows:
wherein ,||Ai The I represents the second norm of the feature vector of each real target classification, and the B represents the second norm of the feature vector of the predicted target classification;
(5.3) selecting the maximum probability value q in the probability vector P, and taking the category corresponding to q as a target classification result;
(5.2) judging whether the category corresponding to q belongs to the disease leaves:
if the class corresponding to q is healthy leaf, directly outputting the target detection result and the target classification result;
if the category corresponding to q belongs to the disease leaves, q is matched with a disease leaf receiving threshold T 0 Comparison is performed:
if q > T 0 Outputting the target detection, target classification and the segmentation result of the disease position;
if q < = T 0 And directly outputting the target detection result and the target classification result.
The effect of the invention can be further illustrated by the following simulation experiments:
experimental conditions
The software platform for the experimental training is as follows: google colab; the hardware platform is as follows: tesla P4 GPU; development environments are keras and Tensorflow;
the software platform for the experimental test is as follows: windows10; the hardware platform is as follows: a CPU;
the test image is selected from the blade image with the size (256,256,3) shown in fig. 5, and the test set is selected from the test set D obtained in the step 2 3 。
Second, experimental details
Experiment 1. Comparing the single test images with Mask R-CNN network model and the method of the present invention.
Fig. 5 is a test image, wherein fig. 5a is a healthy leaf and fig. 5b is a diseased leaf. The detection result of the Mask R-CNN network model is shown in FIG. 6, wherein FIG. 6a is the detection result of healthy leaves, and FIG. 6b is the detection result of diseased leaves. The detection result of the method is shown in fig. 7, wherein fig. 7a is the detection result of healthy leaves, and fig. 7b is the detection result of disease leaves. From the detection results of the two methods, when healthy leaves are detected, compared with a mask-CNN network, the method reduces redundant operation of dividing the disease areas of the leaves.
Experiment 2. Comparative experiments were performed with the inventive method and the google net and VGG networks, respectively.
The method and the existing GoogleNet and VGG network models are used for carrying out the test on the image set D obtained in the step 1 6 1000 tests were performed, and the recognition accuracy results obtained are shown in table 1,
table 1 recognition accuracy results
| Method | Average accuracy rate | Time/s |
| VGG | 0.8846 | 3.17 |
| GoogleNet | 0.9040 | 3.32 |
| Optimized Mask R-CNN | 0.9257 | 3.59 |
As can be seen from Table 1, the method of the present invention has improved recognition accuracy compared with the conventional GoogleNet and VGG network models.
The above description is only a specific example of the invention and does not constitute any limitation of the invention, and it will be apparent to those skilled in the art that modifications and variations in form and detail may be made without departing from the principles, construction of the invention, but these modifications and variations based on the inventive concept remain within the scope of the appended claims.
Claims (9)
1. The plant leaf disease identification method based on the mask convolutional neural network is characterized by comprising the following steps of:
(1) Sequentially enhancing, expanding and semantically segmenting the original data set to obtain a training and testing image set and a mask set;
(2) Optimizing the Mask R-CNN network: namely, a disease feature screening module is added between a full convolution layer after the Mask R-CNN network ROI alignment and Mask branches;
(3) Training the optimized Mask R-CNN network:
(3a) Setting main network parameters:
selecting a backbone network from two residual networks, namely ResNet50 and ResNet 101;
setting the size of the epochs and the number of steps of each epoch training;
setting a receiving threshold T of a disease blade in a disease characteristic screening module 0 The other parameters are default values of Mask R-CNN;
(3b) According to known classification errors L cls Detection error L box And a segmentation error L mask The optimized Mask R-CNN network loss function is determined as follows: l (L) loss ＝L cls +L box +L mask ；
(3c) Inputting the training image set and the training Mask set into the optimized Mask R-CNN network for training to obtain a trained model, wherein the training model is realized as follows:
(3c1) Initializing the network parameters in (3 a), training the image set D 5 And training mask set D 7 Inputting the optimized Mask R-CNN network;
(3c2) Extraction of D through training residual network 5 The characteristics of (3) are obtained to obtain a characteristic spectrum F 0 ；
(3c3) Feature map F 0 Inputting the target region into a region recommendation generation network RPN to obtain a foreground F of the target region 1 And background F 2 ；
(3c4) Foreground F of target region using ROI alignment method 1 Mapping to feature map F 0 Generates a feature map F of a fixed size 3 ；
(3c5) Map F of the characteristics 3 Obtaining a target classification result and a target detection result through the full convolution layer, and calculating a classification error L cls And detecting an error L box ；
(3c6) If the classification result is a disease leaf, confidence coefficient T of the classification result is obtained 1 And a disease leaf reception threshold T 0 In comparison, when T 1 ＞T 0 When the disease characteristic screening module is used, the disease characteristic screening module is used for screening the disease characteristic image F 3 Selecting confidence coefficient as T 1 Feature map F of (1) 4 ；
(3c7) The feature map F selected in (3 c 6) 4 And training mask set D 7 Inputting the images into a mask branch for training to obtain a binary mask of a target area, namely a segmentation result of the disease blade and the disease position thereof, and calculating a segmentation error L mask ；
(3c8) Calculating a loss value L of a network loss ＝L cls +L box +L mask The network weight is updated by back propagation of the loss value after each iteration, and when the trained epoch is greater than the initialized epoch, the network training is stopped, and a trained network model is obtained;
(4) And inputting the test image into the trained model for testing.
2. The method of claim 1, wherein (1) the original data set is sequentially enhanced, expanded and semantically segmented to obtain a training and testing image set and a mask set, which are implemented as follows:
(1a) Downloading from public Plant Village-Dataset a Plant disease leaf Dataset D 0 Pair D using adaptive contrast enhancement algorithm 0 Image enhancement is carried out to improve blurred images in the database, and a data set D after image enhancement is obtained 1 ；
(1b) Labeling tool for enhanced data set D using semantic segmentation 1 Respectively plotting the image targets in the image database to generate masks of the targets to obtain a mask set D containing mask information and label information 2 ；
(1c) Enhanced data set D using image transformations 1 Sum mask set D 2 Sequentially performing translation, rotation and overturn to increase the data volume of the sample and obtain a data set D after expanding the sample 3 Sum mask set D 4 ；
(1d) The expanded data set D is added according to the proportion of 8:2 3 Sum mask set D 4 Divided into training image sets D 5 And test image set D 4 Training mask set D 7 And test mask set D 8 。
3. The method of claim 2, wherein the image enhancement in (1 a) is performed using an adaptive contrast enhancement algorithm, implemented as follows:
(1a1) Dividing the image into high frequency parts h x (i, j) and a low frequency part m x (i, j), wherein (i, j) refers to a pixel point of the image;
(1a2) Multiplying the high frequency part in the image by the gain value G (I, j) of the part to obtain an amplified high frequency part I (I, j):
I(i,j)＝G(i,j)h x (i,j)
(1a3) After recombining the high frequency part and the low frequency part, an enhanced image f (i, j) is obtained:
f(i,j)＝m x (i,j)+I(i,j)。
4. the method of claim 1, wherein the optimized Mask R-CNN network obtained in (2) has the following structure:
residual network ResNet101/50 or pyramid network FPN- & gt region recommendation generation network RPN- & gt full convolution layer- & gt disease feature screening module- & gt mask branch, namely full convolution layer and full connection layer;
the disease feature screening module is used for judging the confidence coefficient T of the disease blade output by the full convolution layer 1 With a given disease leaf acceptance threshold T 0 And (3) screening out the characteristic spectrum of the disease leaves in the batch processing, and inputting the screening result into a mask branch for processing.
5. The method according to claim 1, wherein (3 c 4) the ROI alignment method is used to Align the foreground F of the target region 1 Mapping to feature map F 0 Is realized as follows:
(3 c4 a) calculating a feature layer to which the target area belongs:
wherein ,w0 and h0 Respectively representing the width and height of the target area, k 0 The value is 4;
(3 c4 b) after finding the corresponding characteristic layer k in the target area, obtaining the step length s corresponding to the characteristic layer;
(3 c4 c) calculating the width w of the target region mapped to the feature map 1 And height h 1 Obtaining a target region Z on the feature map:
Z＝w 1 *h 1
wherein ,
(3 c4 d) dividing the target region Z on the feature map into n 2 Obtaining the divided target area Z i ，
Z i ＝w 2 *h 2 ,i＝1,2,…n 2
wherein ,w2 and h2 Represents Z i Width and height, respectively, of the size of
(3 c4 e) each target zone Z i Dividing the image into four parts, obtaining pixel values of four points by taking the central point position of each part, and taking the maximum value of the four pixel values as each target area Z i Finally, n is obtained in the target zone Z 2 And the pixel values form a characteristic diagram with the size of n multiplied by n.
6. The method of claim 1, wherein (3 c 5) the classification error L is calculated cls And detecting an error L loc The implementation is as follows:
(3 c5 a) probability p corresponding to the target classification result u u Obtaining classification error L cls ：
L cls ＝-logp u
(3 c5 b) let t be i u ＝{t x ,t y ,t w ,t h 4 parameterized coordinates, v, of the target detection result i ＝{v x ,v y ,v w ,v h The detection error L is calculated by the following formula, wherein the detection error L is the target translation scaling parameter box ：
Wherein smooths L1 The smoothed norm loss function is expressed as:
7. the method of claim 1, wherein (3 c 7) obtaining a binary mask through a mask branch is implemented as follows:
(3 c7 a) mapping the feature map F by deconvolution transformation 3 Amplifying to obtain binary mask regions M of all classes k ；
(3 c7 b) traversing the binary mask areas M of all classes k Binary mask region M for each category i Applying Sigmoid activation functions
After the classification operation is carried out, a binary mask of a target area corresponding to the target classification is obtained, wherein the maximum probability y in the classification probability vector H is the maximum probability y;
(3 c7 c) calculating the segmentation error L of the mask branch mask ：
wherein ,
the true label of the binary mask for the target region.
8. The method of claim 1, wherein (4) inputting the test image into the trained model for testing is accomplished by:
(4a) The test image set D obtained in (1 a) is collected 6 Inputting the probability vector P into a trained network model, extracting a feature vector, obtaining a classified probability vector P by calculating the similarity of the feature vector, selecting the maximum probability value q in the P, and taking the class corresponding to the q as a target classification result;
(4b) Judging whether the category corresponding to q belongs to the disease leaves:
if the class corresponding to q is healthy leaf, directly outputting the target detection result and the target classification result;
if the category corresponding to q belongs to the disease leaves, q is matched with a disease leaf receiving threshold T 0 Comparison is performed:
if q > T 0 Outputting the target detection, target classification and the segmentation result of the disease positions;
if q < = T 0 And directly outputting the target detection result and the target classification result.
9. The method of claim 8, wherein (4 a) calculating the similarity of the feature vectors to obtain the classified probability vectors is performed by calculating cosine similarity cos (θ) of the feature vector a of each real object classification and the feature vector B of the predicted object classification, as follows:
wherein A represents the two norms of the feature vector of the real target classification, and B represents the two norms of the feature vector of the predicted target classification.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010150980.9A CN111369540B (en) | 2020-03-06 | 2020-03-06 | Plant leaf disease identification method based on mask convolutional neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010150980.9A CN111369540B (en) | 2020-03-06 | 2020-03-06 | Plant leaf disease identification method based on mask convolutional neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111369540A CN111369540A (en) | 2020-07-03 |
| CN111369540B true CN111369540B (en) | 2023-06-02 |
Family
ID=71210311
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010150980.9A Active CN111369540B (en) | 2020-03-06 | 2020-03-06 | Plant leaf disease identification method based on mask convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111369540B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112115888B (en) * | 2020-09-22 | 2022-06-03 | 四川大学 | Plant disease diagnosis system based on disease spot correlation |
| CN112598031A (en) * | 2020-12-08 | 2021-04-02 | 北京农业信息技术研究中心 | Vegetable disease detection method and system |
| CN112634147B (en) * | 2020-12-09 | 2024-03-29 | 上海健康医学院 | PET image noise reduction method, system, device and medium for self-supervision learning |
| CN112560644B (en) * | 2020-12-11 | 2021-09-28 | 四川大学 | Crop disease and insect pest automatic identification method suitable for field |
| CN112699941B (en) * | 2020-12-31 | 2023-02-14 | 浙江科技学院 | Plant disease severity image classification method, device, equipment and storage medium |
| CN112884022B (en) * | 2021-01-29 | 2021-11-12 | 浙江师范大学 | Unsupervised depth characterization learning method and system based on image translation |
| CN113052799A (en) * | 2021-03-09 | 2021-06-29 | 重庆大学 | Osteosarcoma and osteochondroma prediction method based on Mask RCNN network |
| CN113112498B (en) * | 2021-05-06 | 2024-01-19 | 东北农业大学 | Grape leaf spot identification method based on fine-grained countermeasure generation network |
| CN113239788A (en) * | 2021-05-11 | 2021-08-10 | 嘉兴学院 | Mask R-CNN-based wireless communication modulation mode identification method |
| CN113191334B (en) * | 2021-05-31 | 2022-07-01 | 广西师范大学 | Plant canopy dense leaf counting method based on improved CenterNet |
| CN113762190B (en) * | 2021-09-15 | 2024-03-29 | 中科微至科技股份有限公司 | Method and device for detecting package stacking based on neural network |
| CN114239756B (en) * | 2022-02-25 | 2022-05-17 | 科大天工智能装备技术（天津）有限公司 | Insect pest detection method and system |
| CN114742204A (en) * | 2022-04-08 | 2022-07-12 | 黑龙江惠达科技发展有限公司 | Method and device for detecting straw coverage rate |
| CN114943988B (en) * | 2022-06-16 | 2024-04-02 | 浙大城市学院 | Planar target monitoring method based on instance segmentation and deep convolution neural network |
| CN117011718B (en) * | 2023-10-08 | 2024-02-02 | 之江实验室 | Plant leaf fine granularity identification method and system based on multiple loss fusion |
| CN117372790B (en) * | 2023-12-08 | 2024-03-08 | 浙江托普云农科技股份有限公司 | Plant leaf shape classification method, system and device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163207A (en) * | 2019-05-20 | 2019-08-23 | 福建船政交通职业学院 | One kind is based on Mask-RCNN ship target localization method and storage equipment |
| CN110717903A (en) * | 2019-09-30 | 2020-01-21 | 天津大学 | Method for detecting crop diseases by using computer vision technology |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10282589B2 (en) * | 2017-08-29 | 2019-05-07 | Konica Minolta Laboratory U.S.A., Inc. | Method and system for detection and classification of cells using convolutional neural networks |
-
2020
- 2020-03-06 CN CN202010150980.9A patent/CN111369540B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110163207A (en) * | 2019-05-20 | 2019-08-23 | 福建船政交通职业学院 | One kind is based on Mask-RCNN ship target localization method and storage equipment |
| CN110717903A (en) * | 2019-09-30 | 2020-01-21 | 天津大学 | Method for detecting crop diseases by using computer vision technology |
Non-Patent Citations (1)
| Title |
|---|
| 基于迁移学习的番茄叶片病害图像分类;王艳玲等;《中国农业大学学报》(第06期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111369540A (en) | 2020-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111369540B (en) | Plant leaf disease identification method based on mask convolutional neural network | |
| Lv et al. | Maize leaf disease identification based on feature enhancement and DMS-robust alexnet | |
| Dias et al. | Multispecies fruit flower detection using a refined semantic segmentation network | |
| Jaisakthi et al. | Grape leaf disease identification using machine learning techniques | |
| CN110222215B (en) | Crop pest detection method based on F-SSD-IV3 | |
| CN111860330A (en) | Apple leaf disease identification method based on multi-feature fusion and convolutional neural network | |
| Revathi et al. | Homogenous segmentation based edge detection techniques for proficient identification of the cotton leaf spot diseases | |
| Zhang et al. | Robust image segmentation method for cotton leaf under natural conditions based on immune algorithm and PCNN algorithm | |
| CN114693616A (en) | Rice disease detection method, equipment and medium based on improved target detection model and convolutional neural network | |
| Loresco et al. | Segmentation of lettuce plants using super pixels and thresholding methods in smart farm hydroponics setup | |
| Jia et al. | Dough-stage maize (Zea mays L.) ear recognition based on multiscale hierarchical features and multifeature fusion | |
| CN113569772A (en) | Remote sensing image farmland instance mask extraction method, system, equipment and storage medium | |
| Reddy et al. | Mulberry leaf disease detection using yolo | |
| CN113077452A (en) | Apple tree pest and disease detection method based on DNN network and spot detection algorithm | |
| Guo et al. | Identification of maize and wheat seedlings and weeds based on deep learning | |
| Chiu et al. | Semantic segmentation of lotus leaves in UAV aerial images via U-Net and deepLab-based networks | |
| CN113723833B (en) | Method, system, terminal equipment and storage medium for evaluating quality of forestation actual results | |
| Rony et al. | BottleNet18: Deep Learning-Based Bottle Gourd Leaf Disease Classification | |
| Gao et al. | Classification Method of Rape Root Swelling Disease Based on Convolution Neural Network | |
| Sangari et al. | Analyzing the optimal performance of pest image segmentation using non linear objective assessments | |
| CN115601634A (en) | Image blade identification method and device based on hierarchical attention mechanism | |
| Farahani et al. | Identification of grape leaf diseases using proposed enhanced VGG16 | |
| Chodey et al. | Pest Detection in Crop using Video and Image Processing | |
| Raghavendra | An Efficient Approach for Coffee Leaf Disease Classification and Severity Prediction. | |
| Firdaus et al. | Initial Design For Utilizing Machine Learning In Identifying Diseases In Palm Oil Plant |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |