CN112183635A - Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network - Google Patents
Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network Download PDFInfo
- Publication number
- CN112183635A CN112183635A CN202011047680.4A CN202011047680A CN112183635A CN 112183635 A CN112183635 A CN 112183635A CN 202011047680 A CN202011047680 A CN 202011047680A CN 112183635 A CN112183635 A CN 112183635A
- Authority
- CN
- China
- Prior art keywords
- disease
- segmentation
- scale
- image
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 76
- 230000003902 lesion Effects 0.000 title claims abstract description 69
- 238000000034 method Methods 0.000 title claims abstract description 55
- 201000010099 disease Diseases 0.000 claims abstract description 120
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 claims abstract description 120
- 230000004913 activation Effects 0.000 claims abstract description 37
- 238000000605 extraction Methods 0.000 claims abstract description 33
- 206010039509 Scab Diseases 0.000 claims abstract description 21
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 claims abstract description 17
- 238000005070 sampling Methods 0.000 claims abstract description 13
- 230000000694 effects Effects 0.000 claims abstract description 7
- 238000012549 training Methods 0.000 claims description 60
- 238000010586 diagram Methods 0.000 claims description 23
- 238000012360 testing method Methods 0.000 claims description 21
- 239000013598 vector Substances 0.000 claims description 20
- 238000013507 mapping Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 6
- 210000002569 neuron Anatomy 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 230000004927 fusion Effects 0.000 claims description 3
- 239000000126 substance Substances 0.000 claims description 2
- 238000002372 labelling Methods 0.000 abstract description 18
- 230000010354 integration Effects 0.000 abstract description 2
- 241000196324 Embryophyta Species 0.000 description 23
- 230000006870 function Effects 0.000 description 14
- 241000227653 Lycopersicon Species 0.000 description 5
- 235000007688 Lycopersicon esculentum Nutrition 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 5
- 201000004569 Blindness Diseases 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000001788 irregular Effects 0.000 description 2
- 210000004205 output neuron Anatomy 0.000 description 2
- 208000035143 Bacterial infection Diseases 0.000 description 1
- 241001454293 Tetranychus urticae Species 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 208000022362 bacterial infectious disease Diseases 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/061—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using biological neurons, e.g. biological neurons connected to an integrated circuit
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Neurology (AREA)
- Microelectronics & Electronic Packaging (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method for realizing segmentation and identification of plant leaf lesions by a multi-scale deconvolution network. Firstly, a multi-scale residual block is utilized to construct a multi-scale feature extraction module, and multi-scale disease features are extracted; then, a classification and bridging module is introduced to obtain an activation map of a specific class, the activation map contains key information of the scab of the specific class, and the activation map is subjected to up-sampling to realize the segmentation of the scab; and finally, designing a deconvolution module, extracting the real positions of the network concerned scabs by combining a small amount of scab labeling guide features, and further optimizing the recognition and segmentation effects. The method provided by the invention can be suitable for the condition of identifying and dividing the plant leaf diseases with insufficient pixel-level labeled samples, and realizes the integration of identification and division. The model has strong robustness in disease images with insufficient light and noise interference.
Description
Technical Field
The invention belongs to the field of plant disease detection, and particularly relates to a method for realizing segmentation and identification of plant leaf disease spots by a multi-scale deconvolution network.
Background
Crop diseases are one of the major causes of reduced agricultural product yield. The disease spot characteristics of crops are timely and effectively analyzed, the disease type and degree of the crops can be rapidly judged, and corresponding disease control guidance suggestions are provided, so that the economic loss is reduced. Currently, there are two main types of methods for classifying plant diseases. One relies mainly on artificial design feature extraction, using a machine learning approach to classify features. The method generally needs to divide the diseased spots or diseased leaves, has large workload, designs a characteristic extraction method aiming at different disease combinations each time, and is difficult to distinguish similar diseases. The second method is to use a deep convolutional neural network to directly extract disease features to classify disease data, and mainly has the following three difficulties: on one hand, tomato leaf lesions are different in size and irregular in shape, and dynamic changes of the size and the shape of the lesions are difficult to adapt by using a convolution kernel with a fixed size; on the other hand, for plant lesion segmentation, a semantic segmentation network is mainly used at present to divide a lesion image into a background and a lesion, but training of the semantic segmentation network needs a large amount of pixel-level labeling information, which is time-consuming and labor-consuming; finally, the convolution kernel extraction features have blindness and uncertainty, and if the model is over-fitted, the model can respond to a non-lesion area to influence the identification result of the model.
In view of the above situation, a multi-scale feature extraction method, a weak supervised semantic segmentation method based on image class labeling, and an interpretability method of a convolutional neural network are increasingly emphasized. For plant leaf diseases, early disease spots are tiny, and detailed information is lost; as the disease continues, the lesion becomes larger in outline and gradually spreads into an irregular shape. The dynamic change of the size and the shape of the lesion spots cannot be represented by adopting the receptive field with a fixed size, and the problem can be solved by adopting a multi-scale feature extraction method. The method can integrate multiple receptive fields on the same characteristic layer to freely extract global and local characteristics of diseases, excavate deeper disease image abstract characteristics, and improve the expression capability of the characteristics on the size and shape of lesion spots. For plant leaf disease images, the acquisition of lesion marking images is difficult. The problem can be solved by weak supervised semantic segmentation based on image category labeling, an activation graph of a specific category can be obtained by using an image category labeling sample, the activation graph has key characteristics of a target image, good segmentation generalization performance can be realized by up-sampling the activation graph, and the dependence of a model on a pixel level labeling sample is reduced. For the deep convolutional neural network, the recognition effect of the model is influenced by paying attention to the non-lesion part, and the interpretability of the convolutional neural network can solve the problem. Usually, deconvolution is utilized to visualize the lesion site, guide the recognition model to pay attention to the real disease features, and enhance the certainty of the feature extraction of the convolution kernel.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a method for realizing the segmentation and the identification of plant leaf disease spots by a multi-scale deconvolution network, based on disease category marking and a small amount of disease spot marking, a multi-scale disease feature is extracted by a multi-scale feature extraction module, a classification and bridging module is introduced to obtain an activation map of a specific category, the deconvolution network is used for realizing the disease spot segmentation, the network is guided to extract the real position of the disease spot by the aid of the small amount of disease spot marking, the identification and the segmentation effects are further optimized, the integration of the identification and the segmentation is realized, the model has stronger robustness in a disease image with insufficient light and noise interference, and the method can be suitable for the identification and the segmentation of the plant leaf disease with the insufficient number of disease spot marking samples.
The technical solution for realizing the purpose of the invention is as follows:
a method for realizing segmentation and identification of plant leaf lesions by a multi-scale deconvolution network is characterized by comprising the following steps:
step 1: constructing a multi-scale feature extraction module by using the multi-scale residual block to obtain a high-level semantic feature map of the disease image;
step 2: constructing a classification and bridging module: firstly, a high-level semantic feature map is sent to a full-connection layer for calculation, the obtained neurons contain key features for judging image categories, the classification loss of disease images is calculated by using a cross entropy loss function, a SoftMax classifier is trained to obtain correct category probability distribution, disease category prediction is output, and network parameters are updated; then, carrying out linear transformation on the neurons to obtain an activation map of a specific disease category, wherein the activation map comprises a specific category lesion area;
and step 3: performing up-sampling on the activation image of the specific disease category to obtain a predicted lesion segmentation image;
and 4, step 4: selecting a plurality of images for pixel level marking for each disease category, calculating the two-category cross entropy loss of each pixel point, jointly optimizing the category loss and the segmentation loss by using a model loss function, and updating network parameters;
and 5: after each epoch training is finished, recording the training accuracy and the average loss; predicting all test samples by using the trained model, and recording the test accuracy and average loss; and selecting the weight with the minimum damage to the disease identification on the test set as a final plant leaf lesion segmentation and identification model.
Further, the method for realizing the segmentation and identification of the plant leaf disease spots by the multi-scale deconvolution network comprises the following steps of:
taking the feature extraction part of ResNet-50 as a basic skeleton of a multi-scale feature extraction module, and replacing a residual block in ResNet-50 with a multi-scale residual block to construct the multi-scale feature extraction module;
wherein, the structure of the multi-scale residual block is as follows: the input characteristic diagram is passed through 1X 1 convolution layer to obtain characteristic diagram x, and said characteristic diagram x is divided into s portions according to channel number to obtain characteristic diagram xi(i ═ 1,2,3.. s), where the value of s is set to 4; x is the number of1Directly outputting, and corresponding the rest feature subgraphs to convolution operation k of 3 × 3i，xiAnd pass ki-1Adding the calculated feature maps, and inputting the result to kiIn, yiIs a characteristic subgraph xiCorresponding to the output, the calculation of the high-level semantic feature map is written as:
wherein each convolution operation k of 3 x 3iIt is possible to receive the characteristic subgraph x directlyi(ii) a Each pass kiThe operation and the reception fields of the characteristic subgraphs have increased possibility, the similar residual type multi-convolution combined structure enables the same characteristic layer to be capable of freely combining the reception fields with different sizes, and finally the 1 multiplied by 1 convolution layer is used for combining the output of the characteristic subgraphs into the same tensor to be continuously transmitted downwards.
Further, the method for realizing the segmentation and identification of the plant leaf disease spots by the multi-scale deconvolution network comprises the following steps of 2, specifically:
step 2-1: high-level semantic feature graph y obtained by multi-scale feature extraction module by adopting full connection layeriMapping to a class vector
Class vector using softmax classifier
Mapping to [0,1]And calculating the classification loss of the disease image by adopting a cross entropy loss function as follows:
wherein the content of the first and second substances,
representing the probability value that the ith training sample is predicted to be the jth class,
representing the real disease category marked by the ith training sample, wherein c is the total number of the disease categories, and N is the total number of the training samples;
step 2-2: vector the category
Mapping the high-dimensional feature vector through reverse full-connection linear transformation, and remolding into a feature diagram form F ═ H, W, C, wherein H and W are the height and width of the feature diagram, and C is the number of feature diagram channels; in order to prevent the loss of the characteristics, the characteristic vectors and the characteristic maps before and after reverse full connection are correspondingly added, and the characteristic fusion is carried out through the vector addition to obtain the activation map of the specific disease category.
Further, the method for realizing the segmentation and identification of the plant leaf disease spots by the multi-scale deconvolution network comprises the following steps of (3) up-sampling an activation map of a specific disease category:
the method comprises the steps of performing deconvolution by combining a nearest neighbor interpolation method with an up-sampling rate of 2 and convolution operation, performing vector splicing on feature maps with corresponding sizes in a multi-scale feature extraction module by adopting skip connection operation, gradually recovering an activation map of a specific disease category to the size of the resolution of an original image by adopting an up-sampling mode, mapping the activation map to a position between (0 and 1) by adopting a Sigmoid function, and setting pixels with a threshold value of 0.5 and a gray value of more than 0.5 as lesion pixels to obtain a predicted lesion segmentation image.
Further, in the method for segmenting and identifying the plant leaf lesion spots of the multi-scale deconvolution network, the step 4 of calculating the two-classification cross entropy loss and model loss function specifically comprises the following steps:
step 4-1: calculating the cross entropy value of the predicted scab segmentation image and the scab pixel level annotation image, and evaluating the training effect of the segmentation model, wherein the two-classification cross entropy loss is as follows:
wherein N is the total number of pixels contained in the image,
the method comprises the steps that a predicted value of a jth pixel in a binary image output by an ith training sample through a deconvolution module is used, wherein j is 0 to represent a background pixel, and j is 1 to represent a lesion pixel;
the real value of the jth pixel in the ith training sample is taken as the real value of the jth pixel in the ith training sample;
step 4-2: during training, a model loss function is used for jointly optimizing classification loss and segmentation loss, after a cycle of training of a pixel-level marked image of a lesion is completed, a part of disease category marking samples are trained, and then training is performed once by using a training set of pixel-level marking, so that alternate iterative training is performed, and an epoch training is completed until all training samples of disease category marking are trained, wherein the model loss function is as follows:
L＝k*lcls+(1-k)*lseg
wherein lclsClassifying the loss for the disease class output by the first classifier,/segFor lesion segmentation loss output by the second classifier, when a sample with lesion labels is trained, the model focuses more on the position information of lesion pixels; when training a sample without lesion marking, the model only focuses on the image type information.
Compared with the prior art, the invention adopting the technical scheme has the following technical effects:
1. according to the method for realizing the segmentation and identification of the plant leaf disease spots by the multi-scale deconvolution network, deeper multi-scale disease features are excavated by the multi-scale feature extraction module, the dynamic change of the size and the shape of the disease spots is adapted, the accuracy is improved, and the shapes and the boundaries of the disease spots can be relatively completely segmented;
2. the method for realizing the segmentation and the identification of the plant leaf disease spots by the multi-scale deconvolution network utilizes a large number of disease category labeling samples to train a classification and bridging module to generate an activation map of a specific disease category, and the activation map is up-sampled to realize the segmentation of the disease spots. The experimental result shows that the activation graphs of the same disease category sample obtained by the method have similar activation modes, the activation graphs are subjected to up-sampling, and the segmentation model can realize good segmentation generalization performance under the condition of not using a large number of scab pixel level labeling samples for training;
3. according to the method for realizing the segmentation and identification of the plant leaf disease spots by the multi-scale deconvolution network, a small amount of pixel-level labeling information is utilized to guide the feature extraction network to pay attention to the correct positions of the disease spots, the problems of blindness and uncertainty of convolution kernel extraction features are solved, and the classification performance of the model is improved. Experimental results show that the identification accuracy of the method is high on the basis of adding different interference test sets.
Drawings
FIG. 1 is a flow chart of a method for realizing segmentation and identification of plant leaf lesions by using a multi-scale deconvolution network according to the present invention;
FIG. 2 is a schematic diagram of a network structure of a multi-scale deconvolution network implementing a method for segmenting and identifying plant leaf lesions according to the present invention;
FIG. 3 is a schematic structural diagram of a multi-scale residual error module for implementing a method for segmenting and identifying plant leaf lesions by using a multi-scale deconvolution network according to the present invention;
FIG. 4 is a schematic diagram of lesion segmentation in embodiment 1 of the present invention, in which (a) is an original image, (b) is an artificially labeled lesion image, (c) and (d) are schematic diagrams of the FCN-8s and U-Net models, (e) and (f) are schematic diagrams of the U-ResNet-50 and U-Res2Net-50 models, respectively, and (g) is a schematic diagram of the segmentation obtained by the method of the present invention.
Fig. 5 is a disease image of a certain category (first row) and an activation map of a certain category (second row) calculated by the classification and bridging module in embodiment 1 of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention, and are not to be construed as limiting the present invention.
A method for realizing segmentation and identification of plant leaf disease spots by a multi-scale deconvolution network is divided into four stages, namely, multi-scale disease features are obtained; outputting disease category prediction and obtaining an activation map of a specific category; upsampling the activation map; and (5) dividing the lesion spots.
1. First, a convolutional neural network model is constructed, and as shown in fig. 2, training model network parameters are initialized.
2. Starting to train the model, and applying the training strategy shown in fig. 1 to the image class labeling sample and the lesion-free labeling sample, specifically comprising the following steps:
step 1: and constructing a multi-scale feature extraction module by using the multi-scale residual block to obtain a high-level semantic feature map of the disease image.
And taking the feature extraction part of ResNet-50 as a basic skeleton of the multi-scale feature extraction module, and replacing the residual block in ResNet-50 with a multi-scale residual block to construct the multi-scale feature extraction module. Wherein, the constitution of the multi-scale residual block is as follows: after 1 × 1 convolution, averagely dividing the obtained feature graph x into s parts according to the number of channels to obtain a feature sub graph xi(i ═ 1,2,3.. s), where the value of s is set to 4. Except for x1In addition, the remaining feature maps correspond to a convolution operation k of 3 × 3i(i＝2,3,4)，yi(i ═ 1,2,3,4) is feature map xiAnd outputting correspondingly. The calculation of the feature map can be written as:
wherein each convolution operation k of 3 x 3iIt is possible to receive the characteristic subgraph x directlyi(ii) a Each pass kiOperation, feature subgraphHas an increased probability of having a receptive field. The kind residual difference type multi-convolution combination structure enables the same characteristic layer to freely combine the receptive fields with different sizes, and finally the 1 x 1 convolution layer is used for combining the output of the characteristic subgraph into the same tensor to be continuously transmitted downwards.
The disease characteristics are extracted by using a multi-scale residual error module, so that receptive fields with different sizes can be integrated on the same characteristic layer, and the extraction of the size and shape characteristics of the dynamically changed lesion spots is adapted.
Step 2: and constructing a classification and bridging module. Firstly, a high-level semantic feature map is sent to a full-connection layer for calculation, the obtained neurons contain key features for judging image categories, the classification loss of disease images is calculated by using a cross entropy loss function, a SoftMax classifier is trained to obtain correct category probability distribution, disease category prediction is output, and network parameters are updated; and then carrying out linear transformation on the neurons to obtain an activation map of a specific disease category, wherein the activation map comprises key features of the disease spots of the specific category.
Step 2-1: designing full-connection layers with the number of output neurons being 512 and 10 respectively according to the disease types, and using the full-connection layers to obtain the high-level semantic feature graph y obtained by the multi-scale feature extraction moduleiMapping to a class vector
Map it to [0,1 ] using softmax classifier]Meanwhile, calculating the classification loss of the disease image by using a cross entropy loss function, specifically:
wherein
Representing a probability value of predicting the ith training sample into the jth category;
representing the real disease category marked by the ith training sample; c isThe total number of disease categories is set to be 10; and N is the total number of training samples.
Step 2-2: vector the category
And remapping the high-dimensional feature vector through the reverse fully-connected linear transformation with the output neuron numbers of 512 and 25088, and reshaping the feature map into a feature map form F ═ H, W and C, wherein H and W are the height and width of the feature map, and C is the feature map channel number. In order to prevent the loss of the characteristics, the characteristic vectors and the characteristic graphs before and after reverse full connection are correspondingly added by using vector addition operation to complete characteristic fusion, and an activation graph of a specific disease category is obtained.
And step 3: and (4) performing up-sampling on the activation image of the specific disease category to obtain a predicted lesion segmentation image. The method comprises the following specific steps:
the method comprises the steps of combining a nearest neighbor interpolation method with an up-sampling rate of 2 and convolution operation to achieve deconvolution, carrying out vector splicing on feature graphs with corresponding sizes in a multi-scale feature extraction module by adopting skip connection operation, gradually recovering an activation graph of a specific disease category to the size of input image resolution through up-sampling, mapping the features between (0 and 1) by adopting a Sigmoid function, setting a threshold value to be 0.5, and setting pixels with gray values larger than 0.5 as lesion pixels to obtain a predicted lesion segmentation image.
And 4, step 4: selecting 5 diseases for each disease category, marking the disease spots at a pixel level, and calculating two-category cross entropy loss for each pixel point; and jointly optimizing classification loss and segmentation loss by using a model loss function, and updating network parameters.
Step 4-1: by calculating the cross entropy value of the predicted scab segmentation image and the pixel level annotation image, the training effect of the segmentation model can be evaluated, and the two-classification cross entropy loss is as follows:
wherein N is the total number of pixels contained in the image,
the method comprises the steps that a predicted value of a jth pixel in a binary image output by an ith training sample through a deconvolution module is used, wherein j is 0 to represent a background pixel, and j is 1 to represent a lesion pixel;
the true value of the jth pixel in the ith training sample.
Step 4-2: and (3) jointly optimizing classification loss and segmentation loss by using a model loss function during training. After 45 scab pixel-level labeled images are trained for one round, training a part of disease category labeled samples, then training once by using a training set of pixel-level labeling, and repeating the training alternately until all training samples of disease category labeling are trained, thus completing an epoch training. The model loss function is:
L＝k*lcls+(1-k)*lseg
wherein lclsClassifying the loss for the disease class output by the first classifier,/segLesion segmentation loss, which is the output of the second classifier. When a sample marked with scabs is trained, the model focuses more on the position information of the scab pixels, and the value of k is 0.2; when training a sample without lesion marking, the model only focuses on the image type information, and the value of k is 1.
3. After each epoch training is completed, the training accuracy and average loss are recorded. And (5) carrying out one-time prediction on all test samples by using the model trained in the current round, and recording the test accuracy and the average loss. And selecting the weight with the minimum damage to the disease identification on the test set as a final model.
Example 1
In this example, a common data set of leaf diseases of plantavivlage plants is used to test the method of the present invention, and the results are as follows:
selecting 18160 images of 10 types of tomato leaf diseases in the data set, wherein the images comprise 9 tomato disease leaf images and 1 healthy leaf image, such as bacterial disease, early blight, late blight, leaf mold, spot blight, two-spotted spider mite disease, wheel spot disease, mosaic disease and yellow leaf curl disease. The data set was divided by 3:7 into a training set containing 5453 tomato leaf images and a test set containing 12707 tomato leaf images. The input image size is set to 224 × 224 pixels. Except for healthy leaves, the other 9 types of diseases are subjected to pixel level labeling by selecting 30 diseases, wherein 5 diseases are used for training a model, and 25 diseases are used for testing the segmentation performance.
5 models need to be trained for evaluating the segmentation performance of the model of the method:
(1) training a multi-scale deconvolution network model by using 5453 image class labeling samples and 45 samples with lesion marks, wherein the method is the method disclosed by the invention;
(2) replacing a classification and bridging module of the multi-scale deconvolution network with a 3 x 3 convolution kernel with the same number of input and output channels to obtain a model, called U-Res2Net-50 for short, and training the U-Res2Net-50 by using 45 samples with lesion marks;
(3) replacing the multi-scale residual block of the U-Res2Net-50 with a Res-Net residual block, obtaining a model which is called as U-ResNet-50 for short, and training the U-Res2Net-50 by using 45 samples with lesion marks;
(4) 45 samples marked with lesions are used to train the FCN-8s semantic segmentation model.
(5) The U-Net semantic segmentation model was trained using 45 plaque-labeled samples.
Testing is carried out on 225 test sets with lesion pixel level marking information, and 4 common indexes of semantic segmentation are adopted: the difference between the actual segmentation result and the manually labeled data is measured by Pixel Accuracy (PA), Mean Pixel Accuracy (MPA), Mean Intersection Over Unit (MIOU), and Weighted Intersection Over unit (FWIOU), and the obtained segmentation result is as follows.
The lesion segmentation results are explained below:
FIG. 4 shows the divided lesion images, wherein (a) is the original image, (b) is the artificially labeled lesion image, (c) and (d) are the divided schematic diagrams of the FCN-8s and U-Net models, (e) and (f) are the divided schematic diagrams of the U-ResNet-50 and U-Res2Net-50 models, respectively, and (g) is the divided schematic diagram obtained by the method of the present invention. It can be seen that the scab segmentation graph obtained by the method has higher integrity, relatively smooth edges and fewer mistaken segmentation areas.
The segmentation precision of U-Net is obviously higher than FCN-8s, which shows that the detailed information of lesion segmentation can be supplemented by using skip connection to fuse low-level semantic features. The segmentation precision of the U-ResNet-50 is higher than that of the U-Net, because the U-ResNet-50 has a deeper network layer than the U-Net model, and can extract higher-level semantic features to avoid wrong segmentation of the similarity region. Compared with U-ResNet-50, U-Res2Net-50 can segment the shape of the lesion more completely by introducing the multi-scale residual block, the predicted shape of the lesion is closer to the shape of the artificial mark, and the edge of the lesion is clearer and more accurate. The main reason is that the multi-scale residual block fuses global and local information by using reception fields with different sizes, and extraction of space and detail information by a network is effectively enhanced. The segmentation performance of the multi-scale deconvolution model is obviously superior to that of U-Res2Net-50, so that the scab can be completely segmented, redundant non-scab pixels in the U-Res2Net-50 are removed, the scab pixels and background pixels are effectively distinguished, the introduction of a classification and bridging module and a small amount of scab pixel level labeling sample supervision is illustrated, the feature learning of a network on target class pixels and the sensitivity of the network on scab segmentation can be enhanced, the cost of scab labeling can be reduced, and the network is guided to extract the real position of the scab.
And respectively carrying out brightness reduction and noise addition on the test data set to simulate the interference which may occur in the actual shooting scene. To evaluate the classification performance of the multi-scale deconvolution model, 3 models need to be trained: the original image and the test set added with the interference are tested, and the classification accuracy is obtained as follows:
the following explains the disease identification result:
on the original picture test set, the classification accuracy of the method is higher than that of the other two methods, and the identification accuracy reaches 99.35%. On the basis of an interference test set with the brightness reduced by 30%, the identification accuracy of the method is improved by nearly 3% compared with Res2Net-50, and the fact that a small amount of lesion marking information can guide the correct position of a feature extraction network concerned disease based on a multi-scale deconvolution model is proved, and the identification performance of the model is improved. Compared with ResNet-50, Res2Net-50 uses a multi-scale residual error module to extract the disease image characteristics, the identification accuracy is improved by about 1 percentage point on an interference test set with 30% reduced brightness, and the multi-scale residual error module is proved to be capable of enhancing the characteristic extraction performance of the model. The identification accuracy of the method on different interference test sets is higher than that of other two models, and the method has stronger robustness on illumination and noise.
As shown in fig. 5, the disease images of a certain category and the activation maps of a specific category calculated by the classification and bridging module have similar activation patterns although the appearances are very different, which indicates that the classification and bridging module can effectively capture the key features of the diseases of the specific category and send the key features into the segmentation network, and good segmentation generalization performance can be achieved even with a small amount of training data for lesion labeling.
The foregoing is directed to embodiments of the present invention and, more particularly, to a method and apparatus for controlling a power converter in a power converter, including a power converter, a power.
Claims (5)
1. A method for realizing segmentation and identification of plant leaf lesions by a multi-scale deconvolution network is characterized by comprising the following steps:
step 1: constructing a multi-scale feature extraction module by using the multi-scale residual block to obtain a high-level semantic feature map of the disease image;
step 2: constructing a classification and bridging module: firstly, a high-level semantic feature map is sent to a full-connection layer for calculation, the obtained neurons contain key features for judging image categories, the classification loss of disease images is calculated by using a cross entropy loss function, a SoftMax classifier is trained to obtain correct category probability distribution, disease category prediction is output, and network parameters are updated; then, carrying out linear transformation on the neurons to obtain an activation map of a specific disease category, wherein the activation map comprises a specific category lesion area;
and step 3: performing up-sampling on the activation image of the specific disease category to obtain a predicted lesion segmentation image;
and 4, step 4: selecting a plurality of images for pixel level marking for each disease category, calculating the two-category cross entropy loss of each pixel point, jointly optimizing the category loss and the segmentation loss by using a model loss function, and updating network parameters;
and 5: after each epoch training is finished, recording the training accuracy and the average loss; predicting all test samples by using the trained model, and recording the test accuracy and average loss; and selecting the weight with the minimum damage to the disease identification on the test set as a final plant leaf lesion segmentation and identification model.
2. The method for realizing segmentation and identification of plant leaf lesions by using the multi-scale deconvolution network according to claim 1, wherein the step of obtaining the high-level semantic feature map of the disease image in the step 1 specifically comprises the following steps:
taking the feature extraction part of ResNet-50 as a basic skeleton of a multi-scale feature extraction module, and replacing a residual block in ResNet-50 with a multi-scale residual block to construct the multi-scale feature extraction module;
wherein, the structure of the multi-scale residual block is as follows: the input characteristic diagram is passed through 1X 1 convolution layer to obtain characteristic diagram x, and said characteristic diagram x is divided into s portions according to channel number to obtain characteristic diagram xi(i ═ 1,2,3.. s), where the value of s is set to 4; x is the number of1Directly outputting, and corresponding the rest feature subgraphs to convolution operation k of 3 × 3i，xiAnd pass ki-1After the feature maps obtained by the operation are added,is input to kiIn, yiIs a characteristic subgraph xiCorresponding to the output, the calculation of the high-level semantic feature map is written as:
wherein each convolution operation k of 3 x 3iIt is possible to receive the characteristic subgraph x directlyi(ii) a Each pass kiThe operation and the reception fields of the characteristic subgraphs have increased possibility, the similar residual type multi-convolution combined structure enables the same characteristic layer to be capable of freely combining the reception fields with different sizes, and finally the 1 multiplied by 1 convolution layer is used for combining the output of the characteristic subgraphs into the same tensor to be continuously transmitted downwards.
3. The method for realizing plant leaf disease spot segmentation and identification by using the multi-scale deconvolution network according to claim 1 or 2, wherein the step of obtaining the activation map of the specific disease category in the step 2 specifically comprises the following steps:
step 2-1: high-level semantic feature graph y obtained by multi-scale feature extraction module by adopting full connection layeriMapping to a class vector
Class vector using softmax classifier
Mapping to [0,1]And calculating the classification loss of the disease image by adopting a cross entropy loss function as follows:
wherein the content of the first and second substances,
representing the probability value that the ith training sample is predicted to be the jth class,
representing the real disease category marked by the ith training sample, wherein c is the total number of the disease categories, and N is the total number of the training samples;
step 2-2: vector the category
Mapping the high-dimensional feature vector through reverse full-connection linear transformation, and remolding into a feature diagram form F ═ H, W, C, wherein H and W are the height and width of the feature diagram, and C is the number of feature diagram channels; in order to prevent the loss of the characteristics, the characteristic vectors and the characteristic maps before and after reverse full connection are correspondingly added, and the characteristic fusion is carried out through the vector addition to obtain the activation map of the specific disease category.
4. The method for realizing plant leaf lesion segmentation and identification by using the multi-scale deconvolution network according to claim 1, wherein the step of upsampling the activation map of the specific disease category in the step 3 specifically comprises the steps of:
the method comprises the steps of performing deconvolution by combining a nearest neighbor interpolation method with an up-sampling rate of 2 and convolution operation, performing vector splicing on feature maps with corresponding sizes in a multi-scale feature extraction module by adopting skip connection operation, gradually recovering an activation map of a specific disease category to the size of the resolution of an original image by adopting an up-sampling mode, mapping the activation map to a position between (0 and 1) by adopting a Sigmoid function, and setting pixels with a threshold value of 0.5 and a gray value of more than 0.5 as lesion pixels to obtain a predicted lesion segmentation image.
5. The method for segmenting and identifying plant leaf lesions of the multi-scale deconvolution network according to claim 1, wherein the step 4 of calculating the two-class cross entropy loss and model loss function specifically comprises:
step 4-1: calculating the cross entropy value of the predicted scab segmentation image and the scab pixel level annotation image, and evaluating the training effect of the segmentation model, wherein the two-classification cross entropy loss is as follows:
wherein N is the total number of pixels contained in the image,
the method comprises the steps that a predicted value of a jth pixel in a binary image output by an ith training sample through a deconvolution module is used, wherein j is 0 to represent a background pixel, and j is 1 to represent a lesion pixel;
the real value of the jth pixel in the ith training sample is taken as the real value of the jth pixel in the ith training sample;
step 4-2: during training, a model loss function is used for jointly optimizing classification loss and segmentation loss, after a cycle of training of a pixel-level marked image of a lesion is completed, a part of disease category marking samples are trained, and then training is performed once by using a training set of pixel-level marking, so that alternate iterative training is performed, and an epoch training is completed until all training samples of disease category marking are trained, wherein the model loss function is as follows:
L＝k*lcls+(1-k)*lseg
wherein lclsClassifying the loss for the disease class output by the first classifier,/segFor lesion segmentation loss output by the second classifier, when a sample with lesion labels is trained, the model focuses more on the position information of lesion pixels; when training a sample without lesion marking, the model only focuses on the image type information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011047680.4A CN112183635A (en) | 2020-09-29 | 2020-09-29 | Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011047680.4A CN112183635A (en) | 2020-09-29 | 2020-09-29 | Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112183635A true CN112183635A (en) | 2021-01-05 |
Family
ID=73946355
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011047680.4A Pending CN112183635A (en) | 2020-09-29 | 2020-09-29 | Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112183635A (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112750111A (en) * | 2021-01-14 | 2021-05-04 | 浙江工业大学 | Method for identifying and segmenting diseases in tooth panoramic picture |
| CN112883873A (en) * | 2021-02-22 | 2021-06-01 | 暨南大学 | Leaf disease identification method and device, electronic device and storage medium |
| CN112906706A (en) * | 2021-03-31 | 2021-06-04 | 西南科技大学 | Improved image semantic segmentation method based on coder-decoder |
| CN113033549A (en) * | 2021-03-09 | 2021-06-25 | 北京百度网讯科技有限公司 | Training method and device for positioning diagram acquisition model |
| CN113112498A (en) * | 2021-05-06 | 2021-07-13 | 东北农业大学 | Grape leaf scab identification method based on fine-grained countermeasure generation network |
| CN113205051A (en) * | 2021-05-10 | 2021-08-03 | 中国科学院空天信息创新研究院 | Oil storage tank extraction method based on high spatial resolution remote sensing image |
| CN113269191A (en) * | 2021-04-19 | 2021-08-17 | 内蒙古智诚物联股份有限公司 | Crop leaf disease identification method and device and storage medium |
| CN113344009A (en) * | 2021-06-03 | 2021-09-03 | 大连大学 | Light and small network self-adaptive tomato disease feature extraction method |
| CN113808143A (en) * | 2021-09-06 | 2021-12-17 | 沈阳东软智能医疗科技研究院有限公司 | Image segmentation method and device, readable storage medium and electronic equipment |
| CN114022872A (en) * | 2021-09-24 | 2022-02-08 | 中国海洋大学 | Multi-crop leaf disease identification method based on dynamic neural network |
| CN114758132A (en) * | 2022-04-29 | 2022-07-15 | 重庆邮电大学 | Fruit tree pest and disease identification method and system based on convolutional neural network |
| CN115116054A (en) * | 2022-07-13 | 2022-09-27 | 江苏科技大学 | Insect pest identification method based on multi-scale lightweight network |
| CN115294555A (en) * | 2022-09-27 | 2022-11-04 | 江苏景瑞农业科技发展有限公司 | Plant disease intelligent diagnosis method and system based on neural network |
| CN114758132B (en) * | 2022-04-29 | 2024-06-07 | 重庆邮电大学 | Fruit tree disease and pest identification method and system based on convolutional neural network |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108986124A (en) * | 2018-06-20 | 2018-12-11 | 天津大学 | In conjunction with Analysis On Multi-scale Features convolutional neural networks retinal vascular images dividing method |
| US20190015059A1 (en) * | 2017-07-17 | 2019-01-17 | Siemens Healthcare Gmbh | Semantic segmentation for cancer detection in digital breast tomosynthesis |
| CN109359681A (en) * | 2018-10-11 | 2019-02-19 | 西京学院 | A kind of field crop pest and disease disasters recognition methods based on the full convolutional neural networks of improvement |
| CN111598892A (en) * | 2020-04-16 | 2020-08-28 | 浙江工业大学 | Cell image segmentation method based on Res2-uneXt network structure |
-
2020
- 2020-09-29 CN CN202011047680.4A patent/CN112183635A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190015059A1 (en) * | 2017-07-17 | 2019-01-17 | Siemens Healthcare Gmbh | Semantic segmentation for cancer detection in digital breast tomosynthesis |
| CN108986124A (en) * | 2018-06-20 | 2018-12-11 | 天津大学 | In conjunction with Analysis On Multi-scale Features convolutional neural networks retinal vascular images dividing method |
| CN109359681A (en) * | 2018-10-11 | 2019-02-19 | 西京学院 | A kind of field crop pest and disease disasters recognition methods based on the full convolutional neural networks of improvement |
| CN111598892A (en) * | 2020-04-16 | 2020-08-28 | 浙江工业大学 | Cell image segmentation method based on Res2-uneXt network structure |
Non-Patent Citations (2)
| Title |
|---|
| RAIHAN KABIR等: "Discriminant Feature Extraction using Disease Segmentation for Automatic Leaf Disease Diagnosis", 《ICCA 2020: PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON COMPUTING ADVANCEMENTS》, pages 1 - 7 * |
| 任守纲 等: "反卷积引导的番茄叶部病害识别及病斑分割模型", 农业工程学报, vol. 36, no. 12, pages 186 - 195 * |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112750111A (en) * | 2021-01-14 | 2021-05-04 | 浙江工业大学 | Method for identifying and segmenting diseases in tooth panoramic picture |
| CN112750111B (en) * | 2021-01-14 | 2024-02-06 | 浙江工业大学 | Disease identification and segmentation method in tooth full-view film |
| CN112883873A (en) * | 2021-02-22 | 2021-06-01 | 暨南大学 | Leaf disease identification method and device, electronic device and storage medium |
| CN112883873B (en) * | 2021-02-22 | 2023-08-01 | 暨南大学 | Leaf disease identification method, device, electronic device and storage medium |
| CN113033549B (en) * | 2021-03-09 | 2022-09-20 | 北京百度网讯科技有限公司 | Training method and device for positioning diagram acquisition model |
| CN113033549A (en) * | 2021-03-09 | 2021-06-25 | 北京百度网讯科技有限公司 | Training method and device for positioning diagram acquisition model |
| CN112906706A (en) * | 2021-03-31 | 2021-06-04 | 西南科技大学 | Improved image semantic segmentation method based on coder-decoder |
| CN113269191A (en) * | 2021-04-19 | 2021-08-17 | 内蒙古智诚物联股份有限公司 | Crop leaf disease identification method and device and storage medium |
| CN113112498A (en) * | 2021-05-06 | 2021-07-13 | 东北农业大学 | Grape leaf scab identification method based on fine-grained countermeasure generation network |
| CN113112498B (en) * | 2021-05-06 | 2024-01-19 | 东北农业大学 | Grape leaf spot identification method based on fine-grained countermeasure generation network |
| CN113205051A (en) * | 2021-05-10 | 2021-08-03 | 中国科学院空天信息创新研究院 | Oil storage tank extraction method based on high spatial resolution remote sensing image |
| CN113344009B (en) * | 2021-06-03 | 2023-08-01 | 大连大学 | Light and small network self-adaptive tomato disease feature extraction method |
| CN113344009A (en) * | 2021-06-03 | 2021-09-03 | 大连大学 | Light and small network self-adaptive tomato disease feature extraction method |
| CN113808143A (en) * | 2021-09-06 | 2021-12-17 | 沈阳东软智能医疗科技研究院有限公司 | Image segmentation method and device, readable storage medium and electronic equipment |
| CN113808143B (en) * | 2021-09-06 | 2024-05-17 | 沈阳东软智能医疗科技研究院有限公司 | Image segmentation method and device, readable storage medium and electronic equipment |
| CN114022872A (en) * | 2021-09-24 | 2022-02-08 | 中国海洋大学 | Multi-crop leaf disease identification method based on dynamic neural network |
| CN114022872B (en) * | 2021-09-24 | 2024-05-10 | 中国海洋大学 | Dynamic neural network-based method for identifying leaf diseases of various crops |
| CN114758132A (en) * | 2022-04-29 | 2022-07-15 | 重庆邮电大学 | Fruit tree pest and disease identification method and system based on convolutional neural network |
| CN114758132B (en) * | 2022-04-29 | 2024-06-07 | 重庆邮电大学 | Fruit tree disease and pest identification method and system based on convolutional neural network |
| CN115116054A (en) * | 2022-07-13 | 2022-09-27 | 江苏科技大学 | Insect pest identification method based on multi-scale lightweight network |
| CN115116054B (en) * | 2022-07-13 | 2024-05-24 | 江苏科技大学 | Multi-scale lightweight network-based pest and disease damage identification method |
| CN115294555A (en) * | 2022-09-27 | 2022-11-04 | 江苏景瑞农业科技发展有限公司 | Plant disease intelligent diagnosis method and system based on neural network |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112183635A (en) | Method for realizing segmentation and identification of plant leaf lesions by multi-scale deconvolution network | |
| CN111914907B (en) | Hyperspectral image classification method based on deep learning space-spectrum combined network | |
| CN108492297B (en) | MRI brain tumor positioning and intratumoral segmentation method based on deep cascade convolution network | |
| CN110136103A (en) | Medical image means of interpretation, device, computer equipment and storage medium | |
| CN110647875B (en) | Method for segmenting and identifying model structure of blood cells and blood cell identification method | |
| Anandhakrishnan et al. | Deep Convolutional Neural Networks for image based tomato leaf disease detection | |
| CN110163077A (en) | A kind of lane recognition method based on full convolutional neural networks | |
| CN112639997A (en) | Disease diagnosis system and method using neural network for segmentation | |
| CN114998220B (en) | Tongue image detection and positioning method based on improved Tiny-YOLO v4 natural environment | |
| CN108596240B (en) | Image semantic segmentation method based on discriminant feature network | |
| CN110807485B (en) | Method for fusing two-classification semantic segmentation maps into multi-classification semantic map based on high-resolution remote sensing image | |
| CN112464983A (en) | Small sample learning method for apple tree leaf disease image classification | |
| CN111833353B (en) | Hyperspectral target detection method based on image segmentation | |
| CN114913498A (en) | Parallel multi-scale feature aggregation lane line detection method based on key point estimation | |
| CN114821014A (en) | Multi-mode and counterstudy-based multi-task target detection and identification method and device | |
| CN110288041A (en) | Chinese herbal medicine classification model construction method and system based on deep learning | |
| Yildiz et al. | Hybrid image improving and CNN (HIICNN) stacking ensemble method for traffic sign recognition | |
| CN114155249A (en) | Three-dimensional cell image example segmentation method based on depth vector field regression | |
| CN113657414A (en) | Object identification method | |
| CN116311387B (en) | Cross-modal pedestrian re-identification method based on feature intersection | |
| Patil et al. | Sensitive crop leaf disease prediction based on computer vision techniques with handcrafted features | |
| CN112488996A (en) | Inhomogeneous three-dimensional esophageal cancer energy spectrum CT (computed tomography) weak supervision automatic labeling method and system | |
| CN116883650A (en) | Image-level weak supervision semantic segmentation method based on attention and local stitching | |
| CN116189130A (en) | Lane line segmentation method and device based on image annotation model | |
| INTHIYAZ et al. | YOLO (YOU ONLY LOOK ONCE) Making Object detection work in Medical Imaging on Convolution detection System. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |