CN110263863B - Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2 - Google Patents

Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2 Download PDF

Info

Publication number
CN110263863B
CN110263863B CN201910547744.8A CN201910547744A CN110263863B CN 110263863 B CN110263863 B CN 110263863B CN 201910547744 A CN201910547744 A CN 201910547744A CN 110263863 B CN110263863 B CN 110263863B
Authority
CN
China
Prior art keywords
network
bilinear
image
feature extraction
fine
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910547744.8A
Other languages
Chinese (zh)
Other versions
CN110263863A (en
Inventor
袁培森
申成吉
任守纲
顾兴健
车建华
徐焕良
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Agricultural University
Original Assignee
Nanjing Agricultural University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Agricultural University filed Critical Nanjing Agricultural University
Priority to CN201910547744.8A priority Critical patent/CN110263863B/en
Publication of CN110263863A publication Critical patent/CN110263863A/en
Application granted granted Critical
Publication of CN110263863B publication Critical patent/CN110263863B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Molecular Biology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)

Abstract

本发明公开了一种基于迁移学习与双线性InceptionResNetV2的细粒度菌类表型识别方法，其主要步骤为：(1)建立基于迁移学习与双线性的细粒度菌类表型识别模型；(2)基于识别模型进行迁移学习与训练；(3)将图像输入识别模型后进行预处理；(4)对预处理后的图像数据进行特征提取。本发明将两个对称InceptionResNetV2特征提取网络提取到的特征结合起来，得到更细粒度的特征，使识别效果更好；并且使用基于模型的迁移学习训练方法，将在ImageNet数据集上预训练好的特征提取网络参数权重迁移到菌类细粒度表型数据集上，能够在较短的训练时间内，达到更好的收敛性能，使识别结果更好。

Figure 201910547744

The invention discloses a fine-grained fungi phenotype identification method based on migration learning and bilinear InceptionResNetV2, the main steps of which are: (1) establishing a fine-grained fungi phenotype identification model based on migration learning and bilinear; (2) Perform transfer learning and training based on the recognition model; (3) Preprocess the image after inputting the image to the recognition model; (4) Perform feature extraction on the preprocessed image data. The present invention combines the features extracted by two symmetrical InceptionResNetV2 feature extraction networks to obtain more fine-grained features, so that the recognition effect is better; The parameter weights of the feature extraction network are transferred to the fine-grained phenotype dataset of fungi, which can achieve better convergence performance in a shorter training time and make the recognition results better.

Figure 201910547744

Description

Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2
Technical Field
The invention belongs to the fields of computers, artificial intelligence and image processing, and particularly relates to a fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2.
Background
Fine-grained Image Recognition (Fine-grained Image Recognition) is currently applied to the fields of vehicle type Recognition, bird Recognition and the like, but no product specially used for fungus phenotype Recognition exists at present due to the fact that the number of types of fungi is large, the similarity of different subclasses is high, and the Recognition difficulty is high.
Although some fine-grained image recognition technologies exist in the market at present, the fine-grained phenotype recognition cannot be performed on the fungi. Specifically, the following problems need to be solved:
(1) how to use a model-based transfer learning method to transfer the pre-trained model weight on the ImageNet data set to a fungus fine-grained phenotype recognition model, reduce the required data size and training time, and obtain better initial performance and convergence performance.
(2) How to combine the image features extracted by the two feature extraction networks by using bilinear fusion operation to obtain the features with finer granularity for image recognition.
(3) How to use the InceptionResNet V2 feature extraction network with stronger feature extraction capability to extract the features of the image and obtain better features to perform bilinear fusion operation.
Disclosure of Invention
The invention aims to overcome the defects in the prior art, and provides a fine-grained fungus phenotype identification method based on transfer learning and bilinear inclusion ResNet V2, which can be used for training a model according to a fine-grained fungus phenotype data set and identifying different types of fine-grained fungus phenotype images.
In order to achieve the purpose, the invention adopts the technical scheme that: a fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2 comprises the following steps:
step 1, establishing a fine-grained fungus phenotype identification model based on transfer learning and bilinearity;
step 2, performing transfer learning and training based on the recognition model;
step 3, inputting the image into the recognition model and then preprocessing the image;
step 4, extracting the characteristics of the preprocessed image data; the method comprises the steps of extracting feature vectors in an image by adopting an IncepotionResNet V2 feature extraction network with a symmetrical structure, then carrying out bilinear convergence operation on the extracted feature vectors and a self-generated transpose thereof to obtain bilinear feature matrices of each position of the image, converting the bilinear feature matrices into bilinear feature vectors, and finally carrying out multi-classification on the bilinear feature vectors through a full-connection layer and a softmax layer to obtain each class probability.
Further, the preprocessing in step 3 includes centralization, normalization, scaling, random cropping, and random horizontal flipping.
Further, after inputting image data of any size into the network identification model, the average value of the whole data set is firstly subtracted and divided by the standard deviation of the whole data set for centering and normalization, then the image is scaled to 448 pixels on the short side, a square image area of 448 x 448 is cut out from the image by using a random cutting mode, and finally the image is randomly flipped.
Further, in the step 4, an inclusion resnetv2 network in an inclusion series network model is used for feature extraction, and a residual block is added to the inclusion resnetv2 feature extraction network.
Furthermore, the first 7 layers of the inclusion rennet v2 network are composed of three convolution layers, one maximum pooling layer, two convolution layers, and one maximum pooling layer, and then 10 times of the residual inclusion module with three branches, a simpler inclusion module, 20 times of the residual inclusion module with two branches, a 4-branch inclusion module, and 10 times of the residual inclusion module with two branches, and then one convolution layer to obtain an output result.
Further, the bilinear model B is composed of quadruplets, as shown in formula (1),
B＝(fA,fB,P,C) (4)
wherein f isAAnd fBIs a feature function, P is a pooling function of the model, and C is a classification function of the fungi;
the output features are combined from the features at each location using the outer product of the matrix, as shown in equation (2),
bilinear(L,I,fA,fB)＝fA(L,I)TfB(L,I) (5)
wherein L represents a position and a scale, and I represents a picture; if the dimensions of the extracted features of the two feature functions are (K, M) and (K, N), respectively, the dimensions become (M, N) after bilinear fusion operation, if the features of each position are integrated by using summation pooling, as shown in formula (3),
Figure BDA0002104496560000021
wherein Φ (I) represents a global picture feature representation;
finally, the bilinear eigenvector x ═ phi (I) is transformed by the square root of the sign
Figure BDA0002104496560000022
And increasing L2 regularization
Figure BDA0002104496560000023
And then inputting the result into a classifier to obtain a final classification result.
Further, the training process is divided into two steps:
(1) firstly, fixing InceptionResNet V2 characteristics, extracting pre-training parameters loaded by a network and obtained on an ImageNet data set, and only allowing the parameters of the random initialization of the final full-connection layer to be trained;
(2) after the network is converged, the parameters of the InceptionResNet V2 feature extraction network are resolvable and fine-tuned by using a smaller learning rate.
Further, the overall training process is as follows:
(1) constructing a fine-grained fungus phenotype identification model based on transfer learning and bilinearity, wherein the fine-grained fungus phenotype identification model contains InceptionResNetV2 as a feature extraction network;
(2) initializing an InceptionResNet 2 feature extraction network by using an ImageNet pre-training model, and initializing parameters of a full connection layer by using a Glorot normal initializer;
(3) fixing the parameters of the InceptionResNet V2 feature extraction network, so that the parameter values of the part cannot be updated through back propagation in the subsequent training process;
(4) obtaining training samples after image preprocessing from an input pipeline, wherein the batch size is 8, and the image size is 448 x 448;
(5) inputting the batch training samples obtained in the step (4) into a network model, performing feature extraction and bilinear fusion operation and a full connection layer, and finally calculating the probability of each category through softmax;
(6) calculating a loss value of the network model by using a class cross entropy loss function;
(7) by calculating the gradient value, an SGD optimizer is used, the initial learning rate is set to be 1.0, the learning rate attenuation is 1e-8, the Momentum is set to be 0.9, the error is reversely propagated back to the whole network, and the parameters of the full connection layer are updated;
(8) judging whether the specified iteration times are reached to 100 or the 10 early-stop conditions that the iteration change of the verification loss value is not more than 0.001 are met, if so, determining that the network is converged, and entering the step (9), otherwise, entering the step (4) again;
(9) changing the learning rate of the SGD optimizer to 0.001;
(10) the fixation of the InceptionResNet V2 feature extraction network pre-training parameters is released, so that the network can update the parameter values of the part through back propagation;
(11) obtaining training samples after image preprocessing from an input pipeline, wherein the batch size is 8, and the image size is 448 x 448;
(12) inputting the batch training samples obtained in the step (11) into a network model, performing feature extraction, bilinear fusion operation and a full connection layer, and finally calculating the probability of each category through softmax;
(13) calculating a loss value of the network model by using a class cross entropy loss function;
(14) by calculating the gradient value, an SGD optimizer is used, the initial learning rate is set to be 0.001, the learning rate attenuation is 1e-8, the Momentum is set to be 0.9, the error is reversely propagated back to the whole network, and the parameters of each layer of the network are updated;
(15) judging whether the specified iteration times are 70 or the 10 early-stop conditions that the iteration change of the verification loss value is not more than 0.001 are met, if so, determining that the network is converged, and entering the step (16), otherwise, entering the step (11) again;
(16) and calculating the accuracy, precision, recall rate and F1 value of the network model through the test set.
The invention has the beneficial effects that: (1) and by using bilinear convergence, combining the features extracted by the two symmetrical InceprionResNet 2 feature extraction networks to obtain the features with finer granularity, so that the recognition effect is better. (2) By using the model-based transfer learning training method, the pre-trained feature extraction network parameter weights on the ImageNet data set are transferred to the fungus fine-grained phenotype data set, so that better convergence performance can be achieved in shorter training time, and the recognition result is better.
The method is compared with the results of the accuracy, the precision, the recall rate and the F1 value of fungus fine-grained phenotype data sets by using the symmetric VGG16 model and the symmetric VGG19 model respectively, and is shown in the table 1.
TABLE 1 results
Figure BDA0002104496560000041
It can be seen from the table that the model for identifying the fine-grained fungus phenotype based on the transfer learning and bilinear using the symmetric inclusion resnetv2 network provided by the invention has the best effect, and achieves the accuracy of 0.90, the accuracy of 0.91, the recall rate of 0.90 and the F1 value of 0.90, and each index is about 2-6% higher than that of other methods.
Drawings
FIG. 1 is a framework of a fine-grained fungus phenotype recognition model of bilinear IncepotionResNet V2.
Fig. 2 is a pre-processing flow diagram.
Fig. 3 is an inclusion module network structure.
Fig. 4 is an inclpetionresnetv 2 overall network structure.
Fig. 5 is a schematic diagram of migration learning.
Fig. 6 is a training flow diagram.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments.
Network model
The invention selects the InceptionResNet V2 network as the feature extraction network in the Bilinear CNN network, and hopefully, the effect of the whole network model is improved by means of stronger feature extraction capability brought by a deeper network.
The fine-grained fungus phenotype identification model based on the transfer learning and the bilinearity is obtained, the whole network structure is shown in fig. 1, after an image is input into the network model, the image is firstly subjected to the preprocessing processes of centralization, normalization, random cutting and random horizontal turning, then the feature vector extracted from the image is obtained through an Inception ResNet V2 feature extraction network with a symmetrical structure, then the extracted feature vector and the self-generated transposition of the extracted feature vector are subjected to bilinear converging operation to obtain a bilinear feature matrix of each position of the image, the bilinear feature matrix is converted into a bilinear feature vector, and finally the bilinear feature vector is subjected to multi-classification through a full-connection layer and a softmax layer to obtain the probability of each category. The names of the fungus categories related in the present invention are: amanita varia, Xerocomus subtutosus, Conocbe antibodies, Cortinarius rubellus, Helvella crispa, Cuphophllus flavipes, Hygrocbe reidii, Inocbe antibodies, Lyophylom fumosum, Russulotictoides, Tricholoma fulvum, Tricholoma scioides, Lycodon utroform, Rhodocollybia butyracea f.
1. Image input and pre-processing
After image data with any size is input into a network model, the average value of the whole data set is firstly subtracted and divided by the standard deviation of the whole data set for centralization and normalization, and the aim is to enable the data to be scaled to be close to the 0 value without changing the distribution of the data, reduce the difference of different samples in the process of calculating the gradient and accelerate the convergence of the network.
The image is then scaled to 448 pixels on the short side and a random cropping is used to crop 448 x 448 square image areas from the image, and finally the image is randomly horizontally flipped. Preprocessing means such as random clipping and random horizontal turning are all used for increasing the diversity of data sets and enabling the network model to have better generalization performance. Due to the characteristics of the fungus data set, the growth direction of the fungus is from bottom to top, so that only horizontal overturning is adopted instead of vertical overturning.
The image color is represented using three channels of RGB, so the pre-processed image data size is 448 x 3, which is then fed into the feature extraction network for processing, the overall pre-processing being shown in fig. 2.
2. Feature extraction network
A feature extraction network based on the migration learning and bilinear fine-grained fungus phenotype identification model is constructed using an InceptionResNet V2 network. As shown in fig. 3, by using a structure (bottleeck Layer) in which a plurality of convolution kernels are processed and recombined in parallel at 4 branches, the width of the network is increased, and the receptivity of the network to different sizes and scales is increased, so that the problems of too many deep neural network parameters, too large computational complexity, gradient diffusion and the like are solved. The reasonable dimensionality decomposition is carried out by using the splitting convolution operation, under the condition that the detail characteristics are not lost in a large amount, the dimensionality decomposition operation can save a plurality of parameter quantities, the calculation consumption is reduced, the convergence speed of the network is accelerated, meanwhile, the depth of the network is further deepened, and the nonlinearity of the network is improved.
The InceptionResNet V2 feature extraction network has the general structure shown in FIG. 4, and by referring to the residual error network of Microsoft, the design of a residual error block is added, so that parameters can be transmitted by skipping layers through shortcuts in some networks. The first 7 layers of the InceptionResNet V2 network are composed of three convolutional layers, one maximum pooling layer, two convolutional layers and one maximum pooling layer, and then 10 times of residual error inclusion modules with three branches are repeated, and the output result is obtained through one convolutional layer, through a simpler Inception module, through 20 times of residual error inclusion modules with two branches, through one 4-branch Inception module, and finally through 10 times of residual error inclusion modules with two branches.
The parameters of the InceptionResNet V2 feature extraction network main layers are shown in Table 1, where only the top 7 convolutional and maximum pooling layers are listed, as well as the merge, convolutional, residual, and last convolutional layers of each residual Inclusion module, followed by the Batch Normalization and ReLU layers. The image starts from the dimension size of 448 x 3 entering the input layer, the depth of the image is increased through continuous convolution, the maximum pooling layer is halved by the dimension of the image, the residual increment module maintains the dimension size of the image unchanged, the length and the width of the image are reduced and the depth is increased every time the residual increment module passes, the dimension size at the final output is 12 x 1536, and the total parameter number is 54336736.
Table 1 inclusion resnetv2 feature extraction network main layer parameters
Figure BDA0002104496560000061
Figure BDA0002104496560000071
3. Bilinear fusion and classification
Bilinear means that for a function f (x, y), when one of the parameters, e.g., x, is fixed, the function f (x, y) is linear to the other parameter y. In the present invention, the bilinear model B is composed of quadruplets, as shown in equation (1),
B＝(fA,fB,P,C) (7)
wherein f isAAnd fBIs a feature function, P is a pooling function of the model, and C is a classification function of the fungus.
The feature function f, i.e., the feature extraction network in the present invention, is to map an input picture and position to features of size c × D, where D is depth. The features output in the present invention are combined from the features at each position using the outer product of the matrix, as shown in equation (2),
bilinear(L,I,fA,fB)＝fA(L,I)TfB(L,I) (8)
where L denotes the position and scale and I denotes the picture. If the dimensions of the extracted features of the two feature functions are (K, M) and (K, N) respectively, the dimensions become (M, N) after the bilinear fusion operation. If summing pooling is used to integrate the characteristics of the various locations, as shown in equation (3),
Figure BDA0002104496560000072
where Φ (I) represents a global picture feature representation.
Finally, the bilinear eigenvector x ═ phi (I) is transformed by the square root of the sign
Figure BDA0002104496560000073
And increasing L2 regularization
Figure BDA0002104496560000074
And then inputting the result into a classifier to obtain a final classification result.
In the invention, the length and width of the feature extracted by the IncepistionResNet V2 feature extraction network are both 12, and the depth is 1536. Performing bilinear fusion on the feature vectors first requires that the three-dimensional feature vector reshape is a two-dimensional feature vector to obtain a 144 × 1536 feature vector. Then, the eigenvectors are transposed to obtain eigenvectors with dimensions 1536 × 144, and the original eigenvectors and the transposed eigenvectors are used for matrix outer product, namely bilinear fusion operation, to obtain bilinear eigenvectors with dimensions 1536 × 1536. The bilinear eigenvectors are flattened into one-dimensional bilinear eigenvectors of size 2359296, plus a signed square root transform and an L2 regularization layer, followed by multi-classification by softmax using a fully-connected layer with a parameter number of 33030158.
Second, migration learning
In the invention, model-based transfer learning is used, an ImageNet data set with about 1419 thousands of pictures is used as a source domain, the ImageNet data set comprises a plurality of categories, wherein the categories of plants, mushrooms and the like similar to the fungus target task of the invention exist, and model weights pre-trained on the ImageNet data set are transferred onto the fungus data set of the invention, as shown in FIG. 5, so that the required data volume is reduced, higher initial performance, higher training speed and better convergence performance can be obtained.
The pre-training model is obtained from a Keras pre-training model library and then loaded into a fine-grained fungus phenotype identification model based on transfer learning and bilinearity, and the training process is divided into two steps:
(1) the inclusion resnetv2 feature is first fixed to extract the pre-training parameters loaded by the network that are obtained on the ImageNet dataset, allowing only the last full-link layer randomly initialized parameters to be trained.
(2) After the network is converged, the parameters of the InceptionResNet V2 feature extraction network are resolvable and fine-tuned by using a smaller learning rate.
The reason for fixing the InceptionseResNet V2 network pre-training parameters in the first step is that the added full-connection layer is initialized randomly, a large loss value and a large gradient are generated at the beginning, the pre-trained parameters are easy to break, and therefore the whole model needs to be fine-tuned by using a small learning rate after the full-connection layer converges.
The pre-training model optimizer for transfer learning of the invention uses a Stochastic Gradient Descent (SGD) algorithm as an optimizer,
the general training process is shown in fig. 6, and includes the following specific steps:
(1) constructing a fine-grained fungus phenotype identification model based on transfer learning and bilinearity, wherein the fine-grained fungus phenotype identification model contains InceptionResNetV2 as a feature extraction network;
(2) initializing an InceptionResNet 2 feature extraction network by using an ImageNet pre-training model, and initializing parameters of a full connection layer by using a Glorot normal initializer;
(3) fixing the parameters of the InceptionResNet V2 feature extraction network, so that the parameter values of the part cannot be updated through back propagation in the subsequent training process;
(4) obtaining training samples after image preprocessing from an input pipeline, wherein the batch size is 8, and the image size is 448 x 448;
(5) inputting the batch training samples obtained in the step (4) into a network model, performing feature extraction and bilinear fusion operation and a full connection layer, and finally calculating the probability of each category through softmax;
(6) calculating a loss value of the network model by using a class cross entropy loss function;
(7) by calculating the gradient value, an SGD optimizer is used, the initial learning rate is set to be 1.0, the learning rate attenuation is 1e-8, the Momentum is set to be 0.9, the error is reversely propagated back to the whole network, and the parameters of the full connection layer are updated;
(8) judging whether the specified iteration times are reached to 100 or the 10 early-stop conditions that the iteration change of the verification loss value is not more than 0.001 are met, if so, determining that the network is converged, and entering the step (9), otherwise, entering the step (4) again;
(9) changing the learning rate of the SGD optimizer to 0.001;
(10) the fixation of the InceptionResNet V2 feature extraction network pre-training parameters is released, so that the network can update the parameter values of the part through back propagation;
(11) obtaining training samples after image preprocessing from an input pipeline, wherein the batch size is 8, and the image size is 448 x 448;
(12) inputting the batch training samples obtained in the step (11) into a network model, performing feature extraction, bilinear fusion operation and a full connection layer, and finally calculating the probability of each category through softmax;
(13) calculating a loss value of the network model by using a class cross entropy loss function;
(14) by calculating the gradient value, an SGD optimizer is used, the initial learning rate is set to be 0.001, the learning rate attenuation is 1e-8, the Momentum is set to be 0.9, the error is reversely propagated back to the whole network, and the parameters of each layer of the network are updated;
(15) judging whether the specified iteration times are 70 or the 10 early-stop conditions that the iteration change of the verification loss value is not more than 0.001 are met, if so, determining that the network is converged, and entering the step (16), otherwise, entering the step (11) again;
(16) and calculating the accuracy, precision, recall rate and F1 value of the network model through the test set.
The foregoing illustrates and describes the principles, general features, and advantages of the present invention. It should be understood by those skilled in the art that the above embodiments do not limit the scope of the present invention in any way, and all technical solutions obtained by using equivalent substitution methods fall within the scope of the present invention.
The parts not involved in the present invention are the same as or can be implemented using the prior art.

Claims (4)

1.一种基于迁移学习与双线性InceptionResNetV2的细粒度菌类表型识别方法，其特征在于包括以下步骤：1. a fine-grained fungal phenotype identification method based on migration learning and bilinear InceptionResNetV2, is characterized in that comprising the following steps: 步骤1、建立基于迁移学习与双线性的细粒度菌类表型识别模型；Step 1. Establish a fine-grained fungal phenotype recognition model based on transfer learning and bilinearity; 步骤2、基于识别模型进行迁移学习与训练；Step 2. Perform transfer learning and training based on the recognition model; 步骤3、将图像输入识别模型后进行预处理；Step 3. Preprocess the image after inputting the image into the recognition model; 步骤4、对预处理后的图像数据进行特征提取；采用对称结构的InceptionResNetV2特征提取网络提取图像中的特征向量，然后对提取出的特征向量和其自生的转置进行双线性汇合操作得到图片各个位置的双线性特征矩阵，并将双线性特征矩阵转化为双线性特征向量，最后通过全连接层后接softmax层对双线性特征向量进行多分类得到各类别概率；Step 4. Perform feature extraction on the preprocessed image data; use the symmetrical structure InceptionResNetV2 feature extraction network to extract the feature vector in the image, and then perform a bilinear confluence operation on the extracted feature vector and its self-generated transpose to obtain a picture The bilinear feature matrix of each position is converted into a bilinear feature vector, and finally the bilinear feature vector is multi-classified by the fully connected layer followed by the softmax layer to obtain the probability of each category; 所述步骤4中采用Inception系列网络模型中的InceptionResNetV2网络进行特征提取，且在InceptionResNetV2特征提取网络中加入了残差块；所述InceptionResNetV2网络的前7层由三层卷积层、一层最大池化层、两层卷积层、一层最大池化层组成，之后重复10次具有三个分支的残差Inception模块，又通过一个较简单的Inception模块，再经过20次具有两个分支的残差Inception模块，又通过一个4个分支的Inception模块，最后经过10次具有两个分支的残差Inception模块，再通过一个卷积层得到输出结果；In the step 4, the InceptionResNetV2 network in the Inception series network model is used for feature extraction, and a residual block is added to the InceptionResNetV2 feature extraction network; the first 7 layers of the InceptionResNetV2 network are composed of three layers of convolution layers and one layer of maximum pooling. It consists of a layer, a two-layer convolution layer, and a maximum pooling layer, and then repeats the residual Inception module with three branches 10 times, and then passes through a simpler Inception module, and then passes through 20 residuals with two branches. The difference Inception module passes through a 4-branch Inception module, and finally passes through 10 residual Inception modules with two branches, and then passes through a convolutional layer to get the output result; 所述训练过程分为两个步骤：The training process is divided into two steps: (1)首先固定InceptionResNetV2特征提取网络加载的在ImageNet数据集上得到的预训练参数，只允许训练最后的全连接层随机初始化的参数；(1) First, fix the pre-training parameters obtained on the ImageNet dataset loaded by the InceptionResNetV2 feature extraction network, and only allow the training of the parameters randomly initialized by the final fully connected layer; (2)待网络收敛后，再解固InceptionResNetV2特征提取网络的参数，使用较小的学习率进行微调；(2) After the network converges, unfix the parameters of the InceptionResNetV2 feature extraction network, and use a smaller learning rate for fine-tuning; 总的训练流程如下：The overall training process is as follows: (1)构建基于迁移学习与双线性的细粒度菌类表型识别模型，其中包含InceptionResNetV2作为特征提取网络；(1) Build a fine-grained fungal phenotype recognition model based on transfer learning and bilinearity, including InceptionResNetV2 as a feature extraction network; (2)使用ImageNet预训练模型初始化InceptionResNetV2特征提取网络，使用Glorot正常初始化器对全连接层参数进行初始化；(2) Use the ImageNet pre-training model to initialize the InceptionResNetV2 feature extraction network, and use the Glorot normal initializer to initialize the fully connected layer parameters; (3)将InceptionResNetV2特征提取网络的参数固定，使之后的训练过程无法通过反向传播更新此部分的参数值；(3) The parameters of the InceptionResNetV2 feature extraction network are fixed, so that the subsequent training process cannot update the parameter values of this part through backpropagation; (4)从输入管道中获取图像预处理之后的训练样本，batch大小为8，图像大小为448*448；(4) Obtain the training samples after image preprocessing from the input pipeline, the batch size is 8, and the image size is 448*448; (5)将(4)获取到的批次训练样本输入网络模型，经过特征提取与双线性汇合操作以及全连接层，最后通过softmax计算各类别的概率；(5) Input the batch training samples obtained in (4) into the network model, go through feature extraction, bilinear confluence operation and full connection layer, and finally calculate the probability of each category through softmax; (6)使用类别交叉熵损失函数计算网络模型的损失值；(6) Use the category cross entropy loss function to calculate the loss value of the network model; (7)通过计算梯度值，使用SGD优化器，设置初始学习速率为1.0，学习率衰减为1e-8，动量Momentum设置为0.9，将误差反向传播回整个网络，更新全连接层的参数；(7) By calculating the gradient value, using the SGD optimizer, set the initial learning rate to 1.0, the learning rate decay to 1e-8, and the momentum Momentum to 0.9, backpropagate the error back to the entire network, and update the parameters of the fully connected layer; (8)判断是否达到指定迭代次数100或满足验证损失值10个迭代变化不超过0.001的早停法条件，若是，认为网络已经收敛，则进入步骤(9)，若否则重新进入步骤(4)；(8) Judging whether the specified number of iterations is 100 or the condition of the early stopping method that the change of the verification loss value in 10 iterations does not exceed 0.001 is met. If so, the network is considered to have converged, and then go to step (9), otherwise, re-enter step (4) ; (9)改变SGD优化器的学习速率至0.001；(9) Change the learning rate of the SGD optimizer to 0.001; (10)解除对InceptionResNetV2特征提取网络预训练参数的固定，使网络可以通过反向传播更新此部分的参数值；(10) Unfix the pre-training parameters of the InceptionResNetV2 feature extraction network, so that the network can update the parameter values of this part through backpropagation; (11)从输入管道中获取图像预处理之后的训练样本，batch大小为8，图像大小为448*448；(11) Obtain the training samples after image preprocessing from the input pipeline, the batch size is 8, and the image size is 448*448; (12)将(11)获取到的批次训练样本输入网络模型，经过特征提取与双线性汇合操作以及全连接层，最后通过softmax计算各类别的概率；(12) Input the batch training samples obtained in (11) into the network model, go through feature extraction, bilinear convergence operation and fully connected layer, and finally calculate the probability of each category through softmax; (13)使用类别交叉熵损失函数计算网络模型的损失值；(13) Use the category cross entropy loss function to calculate the loss value of the network model; (14)通过计算梯度值，使用SGD优化器，设置初始学习速率为0.001，学习率衰减为1e-8，动量Momentum设置为0.9，将误差反向传播回整个网络，更新网络每一层的参数；(14) By calculating the gradient value, using the SGD optimizer, set the initial learning rate to 0.001, the learning rate decay to 1e-8, and the momentum Momentum to 0.9, backpropagate the error back to the entire network, and update the parameters of each layer of the network ; (15)判断是否达到指定迭代次数70或满足验证损失值10个迭代变化不超过0.001的早停法条件，若是，认为网络已经收敛，则进入步骤(16)，若否则重新进入步骤(11)；(15) Judging whether the specified number of iterations is 70 or the condition of the early stopping method that the change of the verification loss value in 10 iterations does not exceed 0.001 is satisfied. If so, the network is considered to have converged, and then go to step (16), otherwise, re-enter step (11) ; (16)通过测试集计算网络模型的准确率、精确率、召回率、F1值。(16) Calculate the accuracy, precision, recall, and F1 value of the network model through the test set. 2.根据权利要求1所述的基于迁移学习与双线性InceptionResNetV2的细粒度菌类表型识别方法，其特征在于，所述步骤3中的预处理包括中心化、归一化、缩放、随机裁剪和随机水平翻转。2. the fine-grained fungi phenotype identification method based on transfer learning and bilinear InceptionResNetV2 according to claim 1, is characterized in that, the preprocessing in described step 3 comprises centralization, normalization, scaling, randomization Crop and randomly flip horizontally. 3.根据权利要求2所述的基于迁移学习与双线性InceptionResNetV2的细粒度菌类表型识别方法，其特征在于，任意大小的图像数据输入网络识别模型后，首先减去整个数据集的平均值并除以整个数据集的标准差进行中心化和归一化处理，之后图像被缩放至短边为448个像素，并使用随机裁剪的方式从图像中裁剪出448*448的正方形图像区域，最后随机对图像进行水平翻转。3. The fine-grained fungi phenotype identification method based on transfer learning and bilinear InceptionResNetV2 according to claim 2, is characterized in that, after the image data of any size is input into the network identification model, first subtract the average of the entire data set. value and divided by the standard deviation of the entire dataset for centering and normalization, after which the image is scaled to 448 pixels on the short side, and a 448*448 square image area is cropped from the image using random cropping, Finally, the image is randomly flipped horizontally. 4.根据权利要求1所述的基于迁移学习与双线性InceptionResNetV2的细粒度菌类表型识别方法，其特征在于，双线性模型B由四元组组成，如公式(1)所示，4. the fine-grained fungi phenotype identification method based on transfer learning and bilinear InceptionResNetV2 according to claim 1, is characterized in that, bilinear model B is made up of quaternary group, as shown in formula (1), B＝(fA,fB,P,C) (1)B=(f A ,f B ,P,C) (1) 其中fA和fB是特征函数，P是模型的池化函数，C是菌类的分类函数；where f A and f B are the feature functions, P is the pooling function of the model, and C is the classification function of fungi; 输出的特征由每个位置上的特征使用矩阵的外积组合而来，如公式(2)所示，The output features are combined from the features at each position using the outer product of the matrix, as shown in formula (2), bilinear(L,I,fA,fB)＝fA(L,I)TfB(L,I) (2)bilinear(L,I,f A ,f B )=f A (L,I) T f B (L,I) (2) 其中L表示位置和尺度，I表示图片；如果两个特征函数的提取出特征的维度分别为(K,M)和(K,N)，则经过bilinear双线性汇合操作后，维度变成(M,N)，若使用求和池化来综合各个位置的特征，则如公式(3)所示，Among them, L represents the position and scale, and I represents the image; if the dimensions of the extracted features of the two feature functions are (K, M) and (K, N), respectively, after the bilinear bilinear convergence operation, the dimension becomes ( M, N), if the summation pooling is used to synthesize the features of each position, as shown in formula (3),
Figure FDA0003177861050000031
Figure FDA0003177861050000031
 其中Φ(I)表示全局的图片特征表示；where Φ(I) represents the global image feature representation; 最后将双线性特征向量x＝Φ(I)经过符号平方根变换
Figure FDA0003177861050000032
并增加L2正则化
Figure FDA0003177861050000033
再输入分类器得到最后的分类结果。Finally, the bilinear eigenvector x=Φ(I) is transformed by the symbolic square root
Figure FDA0003177861050000032
and add L2 regularization
Figure FDA0003177861050000033
Then enter the classifier to get the final classification result.
CN201910547744.8A 2019-06-24 2019-06-24 Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2 Active CN110263863B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910547744.8A CN110263863B (en) 2019-06-24 2019-06-24 Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910547744.8A CN110263863B (en) 2019-06-24 2019-06-24 Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2

Publications (2)

Publication Number Publication Date
CN110263863A CN110263863A (en) 2019-09-20
CN110263863B true CN110263863B (en) 2021-09-10

Family

ID=67920752

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910547744.8A Active CN110263863B (en) 2019-06-24 2019-06-24 Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2

Country Status (1)

Country Link
CN (1) CN110263863B (en)

Families Citing this family (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110781921A (en) * 2019-09-25 2020-02-11 浙江农林大学 Depth residual error network and transfer learning-based muscarinic image identification method and device
CN111177446B (en) * 2019-12-12 2023-04-25 苏州科技大学 A Method for Footprint Image Retrieval
CN111401122B (en) * 2019-12-27 2023-09-26 航天信息股份有限公司 Knowledge classification-based complex target asymptotic identification method and device
CN111462068B (en) * 2020-03-30 2023-03-21 华南理工大学 Bolt and nut detection method based on transfer learning
CN111613287B (en) * 2020-03-31 2023-08-04 武汉金域医学检验所有限公司 Report coding model generation method, system and equipment based on Glow network
CN111666984B (en) * 2020-05-20 2023-08-25 海南电网有限责任公司电力科学研究院 Overvoltage intelligent identification method based on transfer learning
CN111709326B (en) * 2020-05-29 2023-05-12 安徽省科亿信息科技有限公司 Cascade type particle crop quality analysis method and system based on deep learning
CN111860601B (en) * 2020-06-22 2023-10-17 北京林业大学 Method and device for predicting type of large fungi
CN111914802B (en) * 2020-08-17 2023-02-07 中国电波传播研究所（中国电子科技集团公司第二十二研究所） Ionosphere return scattering propagation pattern identification method based on transfer learning
CN112052904A (en) * 2020-09-09 2020-12-08 陕西理工大学 An identification method of pests and diseases based on transfer learning and convolutional neural network
CN112364899A (en) * 2020-10-27 2021-02-12 西安科技大学 Abrasive grain ferrographic image intelligent identification method based on virtual image and transfer learning
CN112633370B (en) * 2020-12-22 2022-01-14 中国医学科学院北京协和医院 Detection method, device, equipment and medium for filamentous fungus morphology
CN112614124B (en) * 2020-12-29 2025-02-07 腾讯音乐娱乐科技（深圳）有限公司 Video clarity assessment model training method, device and storage medium
CN113128593A (en) * 2021-04-20 2021-07-16 南京林业大学 Plant fine-grained identification method based on bilinear convolutional neural network
CN113642518B (en) * 2021-08-31 2023-08-22 山东省计算中心（国家超级计算济南中心） Discrimination method of cell membrane coloring integrity in HER2 pathological images based on migration learning
CN114638313A (en) * 2022-03-24 2022-06-17 晶展科技(莆田)有限公司 A method, device, medium and equipment for identifying and classifying mahogany furniture
CN114722928B (en) * 2022-03-29 2024-04-16 河海大学 A method for cyanobacteria image recognition based on deep learning
CN118486054B (en) * 2024-07-12 2024-10-18 四川互慧软件有限公司 Snake species identification method, snake species identification neural network training method and medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509976A (en) * 2018-02-12 2018-09-07 北京佳格天地科技有限公司 The identification device and method of animal
CN109086836A (en) * 2018-09-03 2018-12-25 淮阴工学院 A kind of automatic screening device of cancer of the esophagus pathological image and its discriminating method based on convolutional neural networks
CN109389245A (en) * 2018-09-06 2019-02-26 浙江鸿程计算机系统有限公司 A kind of multifactor fusion school district school age population prediction technique based on deep neural network
CN109508655A (en) * 2018-10-28 2019-03-22 北京化工大学 SAR target recognition method based on incomplete training set of Siamese network
CN109508650A (en) * 2018-10-23 2019-03-22 浙江农林大学 A kind of wood recognition method based on transfer learning
CN109635712A (en) * 2018-12-07 2019-04-16 杭州电子科技大学 Spontaneous micro- expression type method of discrimination based on homogeneous network

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10043101B2 (en) * 2014-11-07 2018-08-07 Adobe Systems Incorporated Local feature representation for image recognition

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108509976A (en) * 2018-02-12 2018-09-07 北京佳格天地科技有限公司 The identification device and method of animal
CN109086836A (en) * 2018-09-03 2018-12-25 淮阴工学院 A kind of automatic screening device of cancer of the esophagus pathological image and its discriminating method based on convolutional neural networks
CN109389245A (en) * 2018-09-06 2019-02-26 浙江鸿程计算机系统有限公司 A kind of multifactor fusion school district school age population prediction technique based on deep neural network
CN109508650A (en) * 2018-10-23 2019-03-22 浙江农林大学 A kind of wood recognition method based on transfer learning
CN109508655A (en) * 2018-10-28 2019-03-22 北京化工大学 SAR target recognition method based on incomplete training set of Siamese network
CN109635712A (en) * 2018-12-07 2019-04-16 杭州电子科技大学 Spontaneous micro- expression type method of discrimination based on homogeneous network

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
"基于深度学习的细粒度图像识别研究";陈鹤森;《中国优秀硕士学位论文全文数据库 信息科技辑》;20181115;第1.3.2,3.4.2.2，4.1-4.4节 *

Also Published As

Publication number Publication date
CN110263863A (en) 2019-09-20

Similar Documents

Publication Publication Date Title
CN110263863B (en) Fine-grained fungus phenotype identification method based on transfer learning and bilinear InceptionResNet V2
CN110020682B (en) Attention mechanism relation comparison network model method based on small sample learning
CN110210551B (en) A Visual Object Tracking Method Based on Adaptive Subject Sensitivity
CN109584337B (en) Image generation method for generating countermeasure network based on condition capsule
CN106845529B (en) Image feature identification method based on multi-view convolution neural network
CN108985317B (en) An Image Classification Method Based on Separable Convolution and Attention Mechanism
CN108717568B (en) A Method of Image Feature Extraction and Training Based on 3D Convolutional Neural Network
Juefei-Xu et al. Local binary convolutional neural networks
Lin et al. Hyperspectral image denoising via matrix factorization and deep prior regularization
US11216913B2 (en) Convolutional neural network processor, image processing method and electronic device
US20190087726A1 (en) Hypercomplex deep learning methods, architectures, and apparatus for multimodal small, medium, and large-scale data representation, analysis, and applications
Lee et al. Meta-learning sparse implicit neural representations
CN110929602A (en) Foundation cloud picture cloud shape identification method based on convolutional neural network
CN112686331B (en) Forged image recognition model training method and forged image recognition method
CN111695467A (en) Spatial spectrum full convolution hyperspectral image classification method based on superpixel sample expansion
CN113379655B (en) Image synthesis method for generating antagonistic network based on dynamic self-attention
CN112613479B (en) Expression recognition method based on lightweight streaming network and attention mechanism
CN112329818B (en) Hyperspectral image non-supervision classification method based on graph convolution network embedded characterization
CN106529458A (en) Deep neural network space spectrum classification method for high-spectral image
CN106408009B (en) Neighborhood weighted average hyperspectral image classification method based on depth confidence network
CN114549891A (en) Foundation cloud picture cloud identification method based on contrast self-supervision learning
CN109886160A (en) A face recognition method under unrestricted conditions
CN112634136A (en) Image super-resolution method and system based on image characteristic quick splicing
CN114694219B (en) Facial expression recognition method and device for improving capsule network
CN117315324A (en) Lightweight class detection method and system for Mars rugged topography

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant