CN111611972A - Recognition method of crop leaf species based on multi-view and multi-task ensemble learning - Google Patents

Abstract

The invention relates to a method for identifying crop leaf species based on multi-view and multi-task integrated learning. The method selects a leaf image as an original data set, and performs feature extraction to obtain data sets under several views; using a CNN model as a basic learner, Perform separate ensemble learning on the datasets under several views and the original datasets; then fix the parameters of all basic learners, remove the last layer of the fully connected classifier in the basic learners, and then splicing the outputs of all models Then, a new classifier is added, and joint feature selection is performed on several views, so that the accuracy of the validation set reaches the expectation, and the model under multiple views is obtained; and then the multi-task learning is used to identify the leaf species. The invention strengthens the accuracy and generalization ability of the model, and solves the problem of insufficient generalization ability caused by insufficient training data of the traditional deep learning model and simple stacking of the model.

Description

Recognition method of crop leaf species based on multi-view and multi-task ensemble learning

technical field

The invention belongs to the field of artificial intelligence, and provides an improved method for improving the effect of identifying crop leaves and their existing diseases on the basis of a traditional deep learning model.

Background technique

The current food security problem is becoming more and more serious. Many factors are threatening food security, among which plant diseases pose a serious threat to food security on a global scale. In the past, the identification of crop diseases was mostly manual, but there are many shortcomings in manual identification. With the rise of precision agriculture, the use of information technology to assist agricultural production provides new ideas for the identification of crop diseases. Image processing technology is one of them. It has various advantages for crop disease identification that traditional methods do not have, that is, real-time performance. It is strong, fast, and has a low misjudgment rate, and it can even provide the necessary methods to prevent and control the spread of diseases in time.

At present, the difficulty of identifying crop diseases through images mainly lies in image segmentation, feature extraction and classification recognition.

The main methods to solve these difficulties include threshold segmentation method, edge detection method, mathematical morphology method, support vector machine method and fuzzy clustering method. Although these methods have achieved good classification results, these methods employ traditional machine learning methods to identify diseases by combining low-level visual features with multiple algorithms. This makes them also have some limitations.

Among them, the threshold segmentation method is characterized by simplicity and high execution efficiency, but the selection of the threshold, the color, texture and other characteristics of crop disease and insect pest areas are often quite different from non-disease areas. The segmentation efficiency of the edge detection method depends on the edge detection operator, and the robustness is poor. The disadvantage of mathematical morphology is that the object formed by the union, intersection and difference of various geometric primitives is different from the human sense of shape. The fuzzy clustering method has a slow convergence speed and must first determine the number of classifications and other limitations. The performance of the support vector machine method depends too much on the kernel function and the training speed of the samples. In addition, the above methods extract feature time points too single, and most feature extraction is performed when the symptoms of crop diseases and insect pests are very obvious, which seriously affects the real-time performance, and cannot achieve early identification and early control. Moreover, it is sensitive to noise and initialization data, which in turn leads to the impact of segmentation accuracy, which is particularly prominent in the segmentation of crop disease images with complex growth environments. For example, when the image background is complex or the leaves are powdery, the recognition work will be difficult. At the same time, they mostly rely on hand-crafted features and cannot resolve the semantic gap.

Compared to them, CNN as a deep learning model can automatically discover increasingly higher-level features from data, and has achieved remarkable success in many different fields. Especially in the field of image recognition, CNN has a stable performance when the learning data is sufficient. For general large-scale image classification problems, convolutional neural networks can be used to construct hierarchical classifiers, and can also be used to extract discriminative features of images in fine-grained recognition for other classifiers. study. For the latter, feature extraction can artificially input different parts of the image into the convolutional neural network, or it can be extracted by the convolutional neural network itself through unsupervised learning. These data all show that CNN has achieved great success in the field of image recognition. Therefore, in recent years, many researchers have used CNN methods for plant disease diagnosis.

At present, although the image recognition method based on deep learning has better accuracy than many traditional algorithms, a large number of crop recognition models still face the problem of generalization ability of the model. There are two main reasons: First, the scale of the data set limits. The manual acquisition and manual labeling of crop disease pictures is time-consuming and labor-intensive, resulting in less data for model training. The traditional solution is to use data augmentation to expand the data set, but the degree of improving the generalization ability of the model is very limited. Second, a large number of crop recognition models simply use a simple stack of convolutional neural networks and fully-connected layers, and are only implemented under the idea of a single view. However, different views of an object describe different characteristics of the object. This The defect leads to the poor generalization ability of the model itself.

SUMMARY OF THE INVENTION

In order to solve the problems existing in the prior art, the present invention provides a method for identifying crop leaf species based on multi-view and multi-task integrated learning with high model accuracy and strong generalization ability.

In order to achieve the above purpose, the technical solution proposed by the present invention is: a method for identifying crop leaf species based on multi-view and multi-task integrated learning,

First, the leaf image is selected as the original data set, and feature extraction is performed on the original data set to obtain several data sets under a single view;

Then, using the CNN model as the base learner, separate ensemble learning is performed on several single-view datasets and original datasets;

After the individual ensemble learning is completed, the parameters of all base learners are fixed, and the last layer of the fully connected classifier in all base learners is removed, and then the outputs of all CNN models are spliced together, and a new fully connected classifier is added. Joint feature selection is performed on several views, so that the accuracy of the validation set reaches the expected value, and models under multiple views are obtained to complete multi-view ensemble learning;

Then, multi-task learning is used to share the feature representations learned by different tasks to identify leaf species.

The further design of the above-mentioned technical scheme is: the selection method of the basic learner is: select a deep learning image recognition model to form a model family, then number each model in the model family, and then randomly select a certain number of these models. The model is used as all base learners of the dataset under a single view; for the original dataset, all models in the model family are used as base learners.

The deep learning image recognition models include GoogleNet, VGG, Resnet and so on.

The loss function in the multi-view ensemble learning is:

Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

The invention adopts CNN integrated learning, and uses a variety of mature CNN models as basic learners. Based on this, the multi-view learning is used to train the model for the five views of the edge, grayscale, texture, texture and original picture of the crop leaf. Finally, multi-task learning is used to share the feature representations learned by different tasks. The accuracy and generalization ability of the model are strengthened, and the problem of insufficient generalization ability caused by the lack of training data of traditional deep learning models and the simple stacking depth of the model is generally solved.

This patent adopts multi-task learning, which realizes the hard sharing of hidden layer parameters by sharing the hidden layer among all tasks while retaining the output layer of a specific task, so that multiple tasks can share the learned features of different tasks in parallel training. representation, reducing the risk of overfitting.

Description of drawings

Fig. 1 is the model design drawing of the present invention;

Fig. 2 is the model training flow chart of the present invention;

Figure 3 is a structural diagram of the VGG model;

Fig. 4 is the integrated learning model of the picture after the grayscale extraction;

Figure 5 is a picture extracted by texture;

Figure 6 is a multi-view learning structure diagram;

Figure 7 is a multi-task learning classifier.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

The model design of the crop leaf species identification method based on multi-view and multi-task ensemble learning in this embodiment is shown in Figure 1. Step 1: Take the leaf image as the original data set, and perform feature extraction on the original data set to obtain several views. The data set; using the CNN model as the base learner, separate ensemble learning is performed on the data set under several views and the original data set;

The original dataset images can be obtained under different views through a specific convolution kernel, such as grayscale, texture, edge, texture, etc. (Figure 5 shows a new image extracted by texture). These convolution kernels need to be designed. If we want to get the image texture, we can set the parameters of the convolution kernel to (-1, 0, 1; -2, 0, 2; -1, 0, 1). The parameters of the convolution kernel can be designed according to the requirements of specific tasks. After the original dataset is extracted, several datasets under different views can be obtained.

These pictures of different views are trained separately in different views along with the original pictures. The view definition is shown in formula (1);

Among them, the total number of views is 1+|view|, v ₀ represents the view that has not passed the feature extraction, that is, the original picture, and v _i (1≤i≤|view|) means the i (1≤i≤|view) |) views extracted by feature extraction methods.

The model training process in this embodiment is shown in Figure 2, and the process of ensemble learning is as follows:

First, we define a model family, which is defined as formula (2), where the total number of models is ||.

Here we will use some mature deep learning image recognition models, such as GoogleNet, VGG, Resnet, etc. to form a model family. For example, the VGG deep convolutional neural network, which explores the relationship between the depth of the convolutional neural network and its performance, is successfully constructed by repeatedly stacking 3*3 small convolution kernels and 2*2 max pooling layers Convolutional neural network with 16-19 layers deep. So far, VGGNet is still used to extract image features. The network structure of VGGNet is shown in Figure 3. VGGNet contains many levels of networks, ranging from 11 to 19 layers in depth. The more commonly used ones are VGGNet-16 and VGGNet-19. VGGNet divides the network into 5 segments, each segment connects multiple 3*3 convolutional networks together, each segment of convolution is followed by a maximum pooling layer, and the last is 3 fully connected layers and a softmax layer.

Then, each model in the model family is numbered, and a certain number of models are randomly selected among these models to serve as all base learners under a view. For example, we select five models to form a model family, numbered 1, 2, 3, 4, and 5 respectively. For example, in the grayscale view, we select three models from the model family as all the base learners under the grayscale image, assuming that 1, 2, and 5 are randomly selected, then the 1st, 2nd, and 5th models will be used as grayscale. All models of degree images. In the original view, we will use all models without random selection.

For a single model in a family of models, a new fully connected classifier needs to be designed according to the class and size of our own dataset. For example, under the keras framework, we can set up multiple fully connected layers according to the scale of the specific data set. The activation function of the first several fully connected layers can be set to "relu", and these layers are used for feature learning. ; The last layer is set to "softmax", the role of this layer is to calculate the probability that the image belongs to each category; then we need to freeze part of the convolution base of a single model. Generally in deep learning, one or three convolutional layers and a pooling layer are used as a group, and then several groups constitute a convolutional base. The freezing operation here is usually freezing all convolutional and pooling layers except the last group. Under the keras framework, we can freeze the convolutional layer (pooling layer) by setting the trainable parameter of the convolutional layer (pooling layer) to False. Next, train the model according to the specific data set (images corresponding to different views), so that the accuracy of the validation set can achieve the desired effect. Then unfreeze the last few groups of the convolution base, usually the last third and second groups can be unfrozen. We can unfreeze the convolutional layer (pooling layer) by setting the trainable parameter of the convolutional layer (pooling layer) to True. Finally, jointly train the model so that the accuracy of the validation set achieves the desired effect. We freeze the entire base learner (single model) after training.

Under a single view, we remove the last layer of the fully connected classifier in all base learners, and then splicing the outputs of all models (after removing the last layer, the output is the feature, not the category), And add a new classifier, and then perform ensemble learning on multiple base learners, so that the accuracy of the validation set can achieve the desired effect, so that the ensemble learning model under a single view can be obtained, as shown in Figure 4.

Step 2. In a single view, after the effect of ensemble learning is as expected, we will fix the parameters of all basic learners in a single view. Then remove the last layer of the fully connected classifier in all single views, then splicing the outputs of all models, adding a new classifier, and then performing joint feature selection on multiple views to make the validation set accuracy rate To achieve the desired effect, the model under multiple views can be obtained, and the integrated learning of multiple views can be completed. Part of the structure is shown in Figure 6.

The loss functions involved in multi-view ensemble learning are as follows:

其中

由于图片的维度不确定，所以第j个样本x_ij的维度中的height与width不是确定的数(之后会对其进行预处理，使得基学习器的所有输入样本维度都相同，但允许不同的基学习器的输入样本有所差异。)，而标签的维度都相同，都为类别总数K。We define the dataset under the i (0≤i≤|view|) view as

in

Since the dimension of the picture is uncertain, the height and width in the dimension of the jth sample x _ij are not definite numbers (it will be preprocessed later, so that all input samples of the basic learner have the same dimension, but allow different The input samples of the base learner are different.), and the dimensions of the labels are the same, which are the total number of categories K.

Suppose we are in the _i -th view vi, use the t-th base learner m _t , use softmax for multi-classification, and the loss function uses multi-class cross-entropy, we can get the j-th sample under the view v _i in the base learning The probability of belonging to the kth class under the device m _t is formula (3)

So the loss function for the jth sample is

Then we can get the loss function under the t-th base learner m _t under the i-th view v _i as

我们在每个视图的每个模型下使用正则化约束

则我们可以得到多视图下的损失函数为Since we use all base learners under view v ₀ , and the model learned by the ensemble under the i (1≤i≤|view|)th view is randomly selected, the total number of random selections is p(p<|model |). We can get that the model selected under the view v ₀ is m _i ,m ₂ ,…,m _|modej| , and the model selected under the i-th (1≤i≤|view|) view is

We use regularization constraints under each model per view

Then we can get the loss function under multi-view as

The overall loss function is:

Step 3: Use multi-task learning to share the feature representations learned by different tasks to identify leaf species.

The core idea of multi-task learning is that multiple tasks are trained in parallel and share the feature representations learned by different tasks, which can make full use of the features learned by the model and save resources. If three tasks are selected, they are the type of crops, the type of disease and the severity of the disease. In our network, we remove the last layer of the original fully-connected classifier, and then add the classifier shown in Figure 7 after the penultimate layer, where the first task is to classify the types of crops, In keras, the activation function of the last layer can be set to "softmax", and the second task is used for disease classification. In keras, the activation function of the last layer can be set to "softmax", and the third task is used for prediction The severity of the disease. In keras, the activation function of the last layer can be set to "mse". Then, the model parameters after multi-view learning are fixed, and then the training is performed to make the accuracy of the validation set achieve the desired effect, so that multi-task learning can be realized. Multi-task learning is an inductive transfer method that exploits domain-specific information implicit in training signals for multiple related tasks. In the process of backward propagation, multi-task learning allows the features in the shared hidden layer that are dedicated to a certain task to be used by other tasks; multi-task learning will learn features that are applicable to several different tasks, and such features can be used in a single task. Learning in the network is often not easy to learn.

The technical solutions of the present invention are not limited to the above-mentioned embodiments, and all technical solutions obtained by adopting equivalent replacement methods fall within the protection scope of the present invention.

Claims (4)

1. A crop leaf type identification method based on multi-view and multi-task ensemble learning is characterized in that:

selecting a leaf image as an original data set, and performing feature extraction on the original data set to obtain a plurality of data sets under a single view;

the CNN model is used as a base learner, and independent integrated learning is respectively carried out on a data set and an original data set under a plurality of single views;

after the independent integrated learning is finished, parameters of all the base learners are fixed, the last layer of all the fully connected classifiers in all the base learners is removed, then the outputs of all the CNN models are spliced, new fully connected classifiers are added, and the combined feature selection is carried out on a plurality of views, so that the accuracy of a verification set reaches an expected value, the models under a plurality of views are obtained, and the multi-view integrated learning is finished;

and (3) utilizing multi-task learning to share the learned feature representations of different tasks and identify the blade types.

2. The crop leaf type identification method based on multi-view and multi-task ensemble learning as claimed in claim 1, wherein the selection method of the base learner is as follows: selecting a deep learning image recognition model to form a model family, numbering each model in the model family, and randomly selecting a certain number of models from the models to serve as all base learners of a data set under a single view; for the raw data set, all models in the model family are used as basis learners.

3. The method for identifying the crop leaf types based on the multi-view and multi-task ensemble learning as claimed in claim 2, wherein: the deep learning image recognition model includes GoogleNet, VGG, Resnet, and the like.

4. The method for identifying the crop leaf types based on the multi-view and multi-task ensemble learning as claimed in claim 2, wherein the loss function in the multi-view ensemble learning is as follows:

Images