CN111652308B - Flower identification method based on ultra-lightweight full convolutional neural network - Google Patents
Flower identification method based on ultra-lightweight full convolutional neural network Download PDFInfo
- Publication number
- CN111652308B CN111652308B CN202010476319.7A CN202010476319A CN111652308B CN 111652308 B CN111652308 B CN 111652308B CN 202010476319 A CN202010476319 A CN 202010476319A CN 111652308 B CN111652308 B CN 111652308B
- Authority
- CN
- China
- Prior art keywords
- flower
- neural network
- ultra
- data set
- convolutional neural
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 63
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 32
- 230000011218 segmentation Effects 0.000 claims abstract description 30
- 238000013528 artificial neural network Methods 0.000 claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 25
- 230000003416 augmentation Effects 0.000 claims abstract description 13
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 13
- 239000011159 matrix material Substances 0.000 claims abstract description 11
- 238000012216 screening Methods 0.000 claims abstract description 11
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000003062 neural network model Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000004891 communication Methods 0.000 claims description 3
- 238000004519 manufacturing process Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 7
- 238000004364 calculation method Methods 0.000 abstract description 7
- 238000013526 transfer learning Methods 0.000 abstract description 5
- 238000004140 cleaning Methods 0.000 abstract description 4
- 230000000717 retained effect Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 9
- 238000005516 engineering process Methods 0.000 description 7
- 238000011161 development Methods 0.000 description 6
- 230000018109 developmental process Effects 0.000 description 6
- 238000000605 extraction Methods 0.000 description 6
- 238000012706 support-vector machine Methods 0.000 description 5
- 230000004927 fusion Effects 0.000 description 4
- 230000001965 increasing effect Effects 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000003709 image segmentation Methods 0.000 description 3
- 241000894007 species Species 0.000 description 3
- 238000003860 storage Methods 0.000 description 3
- 241000196324 Embryophyta Species 0.000 description 2
- 241000220317 Rosa Species 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 235000007516 Chrysanthemum Nutrition 0.000 description 1
- 244000189548 Chrysanthemum x morifolium Species 0.000 description 1
- 240000007377 Petunia x hybrida Species 0.000 description 1
- 244000028344 Primula vulgaris Species 0.000 description 1
- 235000016311 Primula vulgaris Nutrition 0.000 description 1
- 101150041570 TOP1 gene Proteins 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000006835 compression Effects 0.000 description 1
- 238000007906 compression Methods 0.000 description 1
- 238000013434 data augmentation Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 230000002950 deficient Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 238000010187 selection method Methods 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/28—Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/457—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by analysing connectivity, e.g. edge linking, connected component analysis or slices
Abstract
The flower identification method based on the ultra-light-weight full convolution neural network is characterized in that the collected data set is subjected to cleaning and segmentation processing, the color image in the data set is binarized by adopting an OTSU algorithm, the maximum connected region is marked by a maximum connected region method, the position information of the connected region is extracted, a matrix mask is generated, and the matrix mask is overlapped with an original picture to obtain a segmentation map of flowers; training a heavy-weight flower classification neural network by adopting a transfer learning method, screening flower segmentation graphs, and constructing a new data set from the retained flower segmentation graphs; and constructing an ultra-lightweight convolutional neural network suitable for a weak calculation platform, inputting a newly constructed data set into the convolutional neural network for training, and realizing flower identification and classification. Compared with other existing methods, the ultra-lightweight neural network and the flower data set augmentation method provided by the invention have the advantage that the speed and the precision are improved considerably.
Description
Technical Field
The invention relates to the technical field of intelligent flower identification, in particular to a flower identification method based on an ultra-lightweight full convolution neural network.
Background
With the great improvement of the living standard of people in China, the flower consumer market in China becomes increasingly prosperous, and especially after 2000 years, the flower industry in China is subjected to rapid development, and the production scale is in the first place in the world. The diversity of flowers and plants is required to have great commercial value due to the excellent properties of the flowers and plants. In order to obtain the maximum utilization efficiency, quick and accurate flower variety classification has great fundamental significance for the development and utilization of the resources.
In the intelligent recognition field, people design different intelligent flower recognition systems according to the shape, color and other characteristics of flowers, but the traditional method extracts low-layer or middle-layer characteristics which do not have good generalization capability and semantic meaning, so that people begin to automatically recognize the flowers by adopting advanced pattern recognition, image processing and deep learning technologies, design various different deep convolutional neural network models, and greatly improve the accuracy of flower classification, such as:
literature: 384-91, a novel deep convolutional neural network is proposed by Prasad M, jwala Lakshmamma B, chandana A H, et al, A efficient classification of flower images with convolutional neural networks [ J ]. International Journal of Engineering & Technology, and 384-91, features are extracted by using four convolution kernels with different sizes, and after two layers of fully connected layers, the final classification is performed by applying a Softmax regression function, and a stochastic pooling Technology combining the advantages of maximum pooling and average pooling is adopted.
Literature: wu Di, hou Lingyan, liu Xiulei et al, improved flower image classification of deep neural network [ J ]. University of Henan (Nature science edition), 2019, adaptively modifying the InceptionV3 network according to the task of flower classification, combining the Tanh-ReLu function with soft saturation characteristic at negative value with the InceptionV3 network, and training the neural network with superior classification effect by using the transfer learning technology.
Literature:M,Budak U,Guo Y,et al.Efficient deep features selections and classification for flower species recognition[J]the method comprises applying a mixed model of AlexNet and VGG16 to the species classification of flowers, using a Support Vector Machine (SVM) with Radial Basis Function (RBF) kernel as a classifier, and applying the features extracted by the mixed model to the classifier for classification.
There have been many literature in recent years to continually improve these approaches by modifying activation functions, feature fusion, etc., such as: literature: yuan Peisen, li Wei, ren Shougang, et al, chrysanthemum flower and variety identification based on convolutional neural networks [ J ]. Agricultural engineering journal, 2018,34 (5);
literature: guo Zi, comfort Liu Changyan, et al, reLU function-based convolutional neural network flower identification algorithm [ J ]. Computer technology and development 2018,28 (5): 154-7;
literature: yin Gong, fu Xiang, have received the flower image classification [ J ]. Chinese image graphics journal, 2019,24 (5): 762-72 with selective convolution feature fusion.
Chinese patent application number: 201910548293. X' also provides a flower identification method based on a CNN feature fusion frame, which combines a plurality of effective features of a picture with CNN with better picture identification performance, trains an identification model based on each feature, combines the fusion frame formed by combining each simple flower identification model, and improves the accuracy of flower identification by utilizing different features of flowers.
In order to pursue better classification effect, the network model structure becomes more complex and the network layer number is deeper, but the accuracy of related tasks is improved, which brings about a plurality of problems. The first is: storage of model weights. The deep neural network contains a large number of weight parameters, the incredible parameter amount can be increased by deepening the network to improve the accuracy, and considerable memory space is needed for saving the model file. The second is: the predicted time problem is applied. An increase in the number of parameters also results in an increase in the computational effort of the network. In prediction, the input data needs to spend a great deal of operation resources through forward propagation, which will cause a great increase in latency of the application.
There are also related patents that propose a flower identification method based on a mobile terminal, such as a flower identification method and a device thereof: 201810538860.9A flower identification method under an android system is provided, a flower database is constructed by collecting images of flowers, feature extraction is carried out through a deep convolutional neural network, flowers are distinguished by Euclidean distance of the features, and in order to adapt to the computing capability of mobile terminal equipment, flower distinguishing is carried out by the Euclidean distance, so that the accuracy is slightly insufficient;
chinese patent: "a flower identification method on intelligent terminal" "" application number: 201410582707.8 by extracting local shape features, feature coding, feature multi-layer clustering, global shape feature extraction and global color feature extraction from the pictures, extracting information related to flower categories from the pictures by a feature fusion method, training by using a linear support vector machine to obtain a support vector machine model, extracting features rapidly by feature extraction and using a KD tree structure, and classifying by using the support vector machine model.
Because the difference between flowers in nature is large, the large-scale labeled data set is deficient, but under the weak calculation platform, the heavy-weight neural network model is not supported, and the limited data set is not suitable for a large model, the flower classification has a plurality of difficulties at present: 1) The weak calculation force platform cannot support the heavy-weight neural network model, and the ultra-light neural network has the problem of low accuracy; 2) Flowers at different angles can lose part of information, and particularly in an ultra-lightweight model, the characteristic extraction capability is poor, so that the classification error is serious; 3) The flower data set has complex sources and uneven quality, and has difficulty in extracting the characteristics.
Disclosure of Invention
In order to solve the technical problems, the invention provides a flower identification method based on an ultra-lightweight full convolution neural network, which provides an automatic flower picture data set augmentation method aiming at the defects of a data set; on the other hand, aiming at the increasing demand of the development of weak computing power platforms, in particular mobile terminal application, the ultra-lightweight full convolution neural network suitable for the edge computing equipment is designed to classify flower images. The ultra-lightweight neural network and the augmentation method of the flower data set can be effectively operated, and the ultra-lightweight neural network and the augmentation method of the flower data set can be well verified through experiments. In addition, compared with other existing methods, the method has the advantage that the speed and the precision are improved considerably.
The technical scheme adopted by the invention is as follows:
the flower identification method based on the ultra-light-weight full convolution neural network comprises the steps of firstly, designing and implementing a crawler program of a flower image to obtain a rough data set, then performing cleaning and segmentation treatment on the collected data set, binarizing the color image in the data set by adopting an OTSU algorithm, marking the largest 3 connected areas by a maximum connected area method, extracting the position information of the connected areas, generating a matrix mask, and overlapping the matrix mask back to an original picture to obtain a segmentation map of the flower;
then training a heavy-weight flower classification neural network by adopting a transfer learning method, screening the flower segmentation map, and constructing a new data set from the retained flower segmentation map;
and finally, constructing an ultra-lightweight convolutional neural network suitable for a weak calculation platform, inputting the newly constructed data set into the convolutional neural network for training, realizing flower identification and classification, and finally applying the convolutional neural network to edge computing equipment.
The flower identification method based on the ultra-lightweight full convolutional neural network comprises the following steps:
step 1: providing an automatic flower data effective augmentation method, obtaining a screened flower segmentation map, and forming a new data set Larger_dataset;
step 2: constructing an ultra-light convolution neural network, and training an ultra-light convolution neural network model by utilizing the new data set obtained in the step 1;
step 3: and identifying the flowers by using the trained ultra-lightweight convolutional neural network model.
In the step 1, the automatic flower data effective augmentation method comprises the following steps:
step 1.1: finally converting the color picture into a binary picture:
firstly graying a color picture, and then using Gaussian filtering to reduce noise; then determining a threshold value of a foreground background of the gray-scale image by using an OTSU algorithm, setting a pixel value larger than the threshold value to 255, and setting a pixel value smaller than the threshold value to 0, thereby finishing binarization of the color picture;
step 1.2: the maximum 3 connected regions are marked by using the maximum connected region method. And one connected region is a pixel set formed by adjacent pixels with the same value, all the connected regions are calibrated by scanning all pixel points, and the largest 3 connected regions are obtained according to the area attribute of the region. The minimum threshold of the area needs to be set according to the actual requirement.
Step 1.3: extracting the position information of 3 communication domains to be selected, setting the pixel point value in the region as 1, setting the other regions as 0, and generating 3 mask matrixes consistent with the original image size.
Step 1.4: superposing the mask matrix and the original picture to obtain final 3 candidate segmentation graphs;
step 1.5: training a heavy-level flower image classification neural network, screening candidate segmentation graphs, automatically discarding the mistakenly segmented flower graphs by comparing the set accuracy, processing all the pictures collected by the network, and obtaining a new data graph set together with a common data set to finish the manufacture of a large_dataset data set.
In the step 2, 23 layers of ultra-lightweight convolutional neural networks are constructed, the input size is 224 x 224, and a feature map with the output size of 14 x 14 is output after 4 times of maximum pooling;
the depth convolution of 3*3 is responsible for filtering, the point-by-point convolution of 1*1 is responsible for channel conversion, the number of two convolution kernels is doubled in the deepening process along with the network layer number, the size is continuously reduced, and high-level features are abstracted from an input image.
In the step 2, the ultra-lightweight convolutional neural network model outputs 2 image position constraint items, namely the central point coordinate (x, y) value of the flower, which is normalized relative to the length and width of the whole image, besides outputting N kinds of information at the output layer;
the loss function of the network is:
wherein N is the number of samples; m is the number of categories; y is ic Indicating a variable 0 or1, which is 1 if the class is the same as the class of sample i, or 0 otherwise; p is p ic The prediction probability of the observation sample i belonging to the category c; epsilon is the loss value of the center point coordinates.
The expression term of the center point coordinate in the loss function is formula (1), wherein l obj Defined as 0 if there is a flower having a value of 1 and if there is no flower; (x, y),respectively representing actual center point coordinates and predicted center point coordinates;
calculating a loss value epsilon of the coordinates of the central point of the flower:
the flower identification method based on the ultra-lightweight full convolutional neural network has the following technical effects:
1: according to the method, firstly, an effective flower data set augmentation method is designed aiming at the conditions that the current flower data set has complex sources, uneven quality and insufficient types and affects the performance of a model, flower image segmentation is carried out through a maximum connected area method, a heavyweight network screens a segmentation map to strengthen the flower data set, and through experimental verification, the detection precision of a neural network model can be greatly improved by training the enhanced data set.
2: the invention constructs an ultra-light weight network for classifying and detecting flowers in real time on a weak computing platform such as a mobile terminal, wherein the model adopts a structure of alternating repetition of 1*1 convolution and 3*3 convolution, and additionally outputs 2 image position constraint items, namely center point coordinates of the flowers, besides outputting N kinds of information on an output layer, so that a trained filter is more focused on the self region of the flowers, and the extracting capability of the flower characteristics is enhanced.
3: the ultra-light network model can effectively reduce the quantity of parameters and the calculated amount on the premise of ensuring the accuracy of the model, and can be widely applied to various mobile devices with limited calculation capability due to the advantages of rapidness and high efficiency.
4: the ultra-lightweight neural network and the augmentation method of the flower data set can be effectively operated, and the ultra-lightweight neural network and the augmentation method of the flower data set can be well verified through experiments. In addition, compared with other existing methods, the method has the advantage that the speed and the precision are improved considerably.
Drawings
Fig. 1 is a specific flowchart for capturing flower images.
FIG. 2 (a) is a non-floral picture taken by a crawler program;
FIG. 2 (b) is a difficult-to-discern picture of a target acquired by a crawler;
fig. 2 (c) is a false target picture (not rose) taken by the crawler.
Fig. 3 is a diagram showing the existence of a plurality of floral bodies.
Fig. 4 is a flowchart for segmentation and identification of flowers.
FIG. 5 (a) is a diagram before binarization of a picture;
fig. 5 (b) is a binarized result diagram of the picture.
Fig. 6 is a mask matrix diagram.
FIG. 7 (a) is a graph of the preliminary segmentation result;
FIG. 7 (b) is a second diagram of the preliminary segmentation result;
fig. 7 (c) is a preliminary segmentation result diagram three.
Fig. 8 is a graph of flower picture quantity distribution of two data sets.
FIG. 9 is a block diagram of an ultra lightweight network model.
Fig. 10 is a training loss trend graph for two networks on an enhanced data set.
FIG. 11 is a graph of classification accuracy versus Oxford102 dataset for various methods.
Detailed Description
The mobilized digital life becomes a mainstream life style increasingly, and the defect of the heavy-weight neural network makes the mobilized digital life difficult to apply to weak computing platforms such as AI edge computing equipment. The simplified network and method can be applied to weak computing power platforms such as mobile terminals in real time, but the accuracy rate of the network and method often cannot meet the requirements. In order to solve this contradiction, researchers usually use a model compression method, that is, pruning parameters or transforming data types of the trained models, so that the storage of network weight parameters becomes more compact, thereby solving the memory problem and the prediction speed problem. Compared with the method for directly processing the original model weight, the lightweight model has the advantages of constructing a more efficient convolution network calculation mode, reducing network parameters and simultaneously achieving good network performance. Meanwhile, for a lightweight model, the quality of a data set is also a key factor for determining the performance of the model, and the data sets with various types, strong representativeness and enough number can greatly improve the performance of the model.
The flower identification method based on the ultra-light-weight full convolution neural network comprises the steps of firstly, cleaning and dividing a collected data set, binarizing a color image in the data set by adopting an OTSU algorithm, marking the largest 3 connected areas by a maximum connected area method, extracting position information of the connected areas, generating a matrix mask, and overlapping the matrix mask back to an original picture to obtain a flower division map; then training a heavy-weight flower classification neural network by adopting a transfer learning method, screening the flower segmentation map, and constructing a new data set from the retained flower segmentation map; and finally, constructing an ultra-lightweight convolutional neural network suitable for a weak calculation platform, inputting the newly constructed data set into the convolutional neural network for training, realizing flower identification and classification, and finally applying the convolutional neural network to edge computing equipment.
The flower identification method based on the ultra-lightweight full convolutional neural network comprises the following steps:
step 1: providing an automatic flower data effective augmentation method, obtaining a screened flower segmentation map through a flow shown in fig. 4, and forming a new data set Larger_dataset;
step 2: constructing an ultra-light convolution neural network, and training an ultra-light convolution neural network model by utilizing the new data set obtained in the step 1;
step 3: and identifying the flowers by using the trained ultra-lightweight convolutional neural network model.
The details of each step are as follows:
step 1: the development and development of the deep learning technology are based on data, a high-quality target task data set is beneficial to accurately extracting effective characteristics through an algorithm, a solid foundation is laid for obtaining a network model with excellent performance, and the Oxfor102 data set commonly used today only comprises 8189 pictures of 102 flowers, and the quality and the number of the data set are insufficient to support an ultra-lightweight neural network model under a weak computing power platform.
In order to acquire a better data set, the Google search engine downloads 52753 pictures of 102 flowers corresponding to the types on the Oxford102 data set, and then carries out cleaning treatment, firstly, a user makes rules for capturing the pictures, inputs the rules into a crawler program, the program starts to request a query page from the Google search engine, and for a returned webpage, the program analyzes all picture links appearing on the webpage, and at the same time, acquires the next search page to prepare for capturing in the next round. The analyzed download links should also be filtered according to the given specification of the user, the link addresses meeting the requirements are saved, and finally the files are downloaded to the local disk according to the addresses. The specific grabbing flow is shown in fig. 1.
Since the crawler downloads the picture only according to the given keyword, many dirty data such as rose picture dirty data acquired by the crawler program shown in fig. 2 (a), 2 (b) and 2 (c) will occur. And the network pictures have complex sources and uneven quality, so that further image segmentation processing is needed. In addition to the above dirty data, there is a reality that a plurality of flower subjects exist simultaneously in a few pictures. As shown in fig. 3.
Aiming at the problems, the invention provides an automatic flower data effective augmentation method based on a heavyweight neural network model. Firstly, extracting a maximum connected region candidate domain in a picture, then, using a pre-trained neural network model to conduct predictive screening of the flower map, obtaining a final segmentation map, and integrating the final segmentation map into a new data set.
The data augmentation of flowers is divided into five steps, and the flow is shown in figure 4:
the first step: the method is to convert the color picture into a binary picture, and comprises the following main steps: firstly graying a color picture, and then using Gaussian filtering to reduce noise; then, an OTSU algorithm is used for determining a threshold value of foreground and background of the gray-scale image, a pixel value larger than the threshold value is set to 255, and a pixel value smaller than the threshold value is set to 0, so that binarization of the color picture is completed. A flower map with complex background is selected as an example, and the binarized comparison map is shown in fig. 5 (a) and 5 (b).
Among them, the OTSU algorithm is described in Otsu N.A threshold selection method from gray-level history [ J ]. IEEE transactions on systems, man, and cybernetics,1979,9 (1): 62-6. The OTSU algorithm is also known as the "oxford algorithm" or "maximum inter-class variance method". The method is considered as an optimal algorithm for selecting the threshold value in image segmentation, is simple to calculate and is not influenced by the brightness and the contrast of the image, so that the method is widely applied to digital image processing. The present invention also uses this more reliable binary segmentation algorithm.
And a second step of: the maximum 3 connected regions are marked by using the maximum connected region method. The invention marks all the connected areas by scanning all the pixel points. And acquiring the maximum 3 connected domains according to the area attribute of the region.
And a third step of: extracting the position information of 3 communication domains to be selected, setting the pixel point value in the region as 1, setting the other regions as 0, and generating 3 mask matrixes consistent with the original image size. The mask matrix diagram in this embodiment is shown in fig. 6.
Fourth step: and superposing the mask matrix and the original picture to obtain final 3 candidate segmentation graphs. As shown in fig. 7 (a), 7 (b) and 7 (c). As can be seen from fig. 7 (a) to 7 (c), the segmentation result fig. 7 (c) is not a flower picture and should be discarded.
Fifth step: training a heavyweight flower image classification neural network and screening candidate segmentation graphs. The heavy-weight level network has higher requirement on the computing capacity of the equipment, but has correspondingly better performance, and the heavy-weight level neural network used for screening the segmentation map can be selected from models with better performance at present, such as VGG, resNet152, dark net53 and the like, and the dark net53 network model is selected in the invention. And inputting the candidate segmentation map into the trained heavyweight level neural network for prediction screening. By comparing the set accuracies, the misclassified flower map is automatically discarded. After processing all 52753 pictures collected by the network, the processed pictures are combined with the Oxford102 to obtain a new data atlas which is 20554 pieces in total, and the creation of the large_dataset data set is completed.
The method for automatically screening out the qualified flower pictures by combining the pre-trained neural network avoids the defect that a large amount of picture data is large in manual screening workload, and greatly reduces the workload.
The final gather sort-out results in a large enhancement of the large dataset compared to the Oxford102 dataset. For the 102 flowers contained, the data amount of substantially each flower was expanded by 2 to 4 times. Wherein, the minimum ping primrose pictures are increased from 40 to 100, and the maximum petunia is increased from 258 to 644. The picture distribution of the two data sets is shown in fig. 8.
Step 2: aiming at a weak computing power platform with poor computing capability, the ultra-light weight network model is researched and constructed, the large_dataset data set obtained in the step 1 is input into the ultra-light weight network model for training, and finally the ultra-light weight network model is applied to the classification and identification of flowers on edge computing equipment, so that the performance of the ultra-light weight network model is well represented. The designed ultra-lightweight network model has 23 layers in total, the network structure is shown in fig. 9, the input size is 224 x 224, the characteristic diagram with the output size of 14 x 14 after 4 times of maximum pooling is carried out, and the specific network structure is shown in fig. 9.
The structure of alternating repetition of 1*1 convolution and 3*3 convolution in the model is referred to as a depth resolvable convolution structure in a MobileNet network, and the proposed depth resolvable convolution structure is mentioned in the paper of the MobileNet network structure, compared with a standard convolution, the parameter quantity and the calculated quantity can be effectively reduced on the premise of ensuring the accuracy of the model. The depth convolution of 3*3 is responsible for filtering, the point-by-point convolution of 1*1 is responsible for channel conversion, the number of two convolution kernels is doubled in the deepening process along with the network layer number, the size is continuously reduced, and high-level features are abstracted from an input image. The ultra-light network model can be widely applied to various mobile devices with limited computing capacity due to the advantages of rapidness and high efficiency.
As shown in fig. 9, the present model outputs 2 additional image position constraint items, that is, the center point coordinate (x, y) value of the flower, normalized with respect to the entire image length and width, in addition to class_number category information at the output layer. The use of this constraint allows the trained filter to focus more on the flower's own region, enhancing the extraction of flower features.
The loss function of the network is:
wherein N is the number of samples; m is the number of categories; y is ic An indicator variable (0 or 1), which is 1 if the class is the same as the class of sample i, or 0 otherwise; p is p ic The prediction probability of the observation sample i belonging to the category c; epsilon is the loss value of the center point coordinates.
The expression term of the center point coordinate in the loss function is formula (1), wherein l obj Defined as 0 if there is a flower having a value of 1 and if there is no flower; (x, y),respectively representing the actual center point coordinate and the predicted center point coordinate value.
Calculating a loss value epsilon of the coordinates of the central point of the flower:
on the large_dataset, the ultra-lightweight network model was trained, and the training parameters are shown in table 1.
Table 1 training parameter set point table
Training loss graphs obtained after training are shown in fig. 10, and it can be seen from fig. 10 that the average loss rate decreases to the lowest point when iterating to 25000 times.
Meanwhile, the invention also carries out the same training on the Oxford102 data set before processing to compare the detection results after training of the two data sets. After training of the two data sets, the obtained Top1 accuracy details are shown in Table 2.
Table 2 classification accuracy table for networks trained on two data sets
The comparison finds that better classification accuracy can be obtained by using the enhanced data set, and the improvement on the ultra-lightweight network model is obvious.
The training completed network is compared on an Oxford102 flower data set, and the result shows that compared with other existing models, the ultra-lightweight network model trained after data enhancement by the method of the invention has good accuracy and detection speed performance in flower classification, as shown in figure 11. Based on the characteristics of limited computing capacity and high storage cost of mobile terminal equipment, flower classification application using the ultra-lightweight network model as a basic backbone network has a very high practical prospect.
References to other models available in fig. 11:
[8] yang Guoliang, wang Zhiyuan, zhang Yu. A modified fine image classification of deep convolutional neural networks [ J ]. University of Jiangxi university (Nature science edition), 2017,41 (05): 476-83.
[9]M,Budak U,Guo Y,et al.Efficient deep features selections and classification for flower species recognition[J].Measurement,2019,137(7-13)。
[10]Ge W,Yu Y.Borrowing treasures from the wealthy:Deep transfer learning through selective joint fine-tuning；proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition,F,2017[C].
[11]Hiary H,Saadeh H,Saadeh M,et al.Flower classification using deep convolutional neural networks[J].IET Computer Vision,2018,12(6):855-62.
[12]Prasad M,Jwala Lakshmamma B,Chandana A H,et al.An efficient classification of flower images with convolutional neural networks[J].International Journal of Engineering&Technology,2018,7(11):384-91。
Claims (2)
1. The flower identification method based on the ultra-light weight full convolution neural network is characterized by comprising the following steps of:
step 1: providing an automatic flower data effective augmentation method to obtain a screened flower segmentation map to form a new data set;
in the step 1, the automatic flower data effective augmentation method comprises the following steps:
step 1.1: finally converting the color picture into a binary picture:
firstly graying a color picture, and then using Gaussian filtering to reduce noise; then determining a threshold value of a foreground background of the gray-scale image by using an OTSU algorithm, setting a pixel value larger than the threshold value to 255, and setting a pixel value smaller than the threshold value to 0, thereby finishing binarization of the color picture;
step 1.2: marking the largest 3 connected areas by adopting a maximum connected area method; one connected region is a pixel set formed by adjacent pixels with the same value, all the connected regions are calibrated by scanning all pixel points, and the largest 3 connected regions are obtained according to the area attribute of the region;
step 1.3: extracting the position information of 3 communication domains to be selected, setting the pixel point value in the region as 1, setting the other regions as 0, and generating 3 mask matrixes consistent with the original image size;
step 1.4: superposing the mask matrix and the original picture to obtain final 3 candidate segmentation graphs;
step 1.5: training a heavy-level flower image classification neural network, screening candidate segmentation graphs, automatically discarding the mistakenly segmented flower graphs by comparing the set accuracy, processing all the pictures collected by the network, and obtaining a new data graph set together with a common data set to finish the manufacture of a large_dataset data set;
step 2: constructing an ultra-light convolution neural network, and training an ultra-light convolution neural network model by utilizing the new data set obtained in the step 1;
in the step 2, the ultra-lightweight convolutional neural network model outputs 2 image position constraint items, namely the central point coordinate (x, y) value of the flower, which is normalized relative to the length and width of the whole image, besides outputting N kinds of information at the output layer;
the loss function of the network is:
wherein N is the number of samples; m is the number of categories; y is ic Indicating a variable 0 or1, which is 1 if the class is the same as the class of sample i, or 0 otherwise; p is p ic The prediction probability of the observation sample i belonging to the category c; epsilon is the loss value of the center point coordinate;
the expression term of the center point coordinate in the loss function is formula (1), wherein l obj Defined as 0 if there is a flower having a value of 1 and if there is no flower; (x, y),respectively representing actual center point coordinates and predicted center point coordinates;
calculating a loss value epsilon of the coordinates of the central point of the flower:
step 3: and identifying the flowers by using the trained ultra-lightweight convolutional neural network model.
2. The flower identification method based on the ultra-lightweight full convolutional neural network according to claim 1, wherein the flower identification method comprises the following steps: in the step 2, 23 layers of ultra-lightweight convolutional neural networks are constructed, the input size is 224 x 224, and a feature map with the output size of 14 x 14 is output after 4 times of maximum pooling; the depth convolution of 3*3 is responsible for filtering, the point-by-point convolution of 1*1 is responsible for channel conversion, the number of two convolution kernels is doubled in the deepening process along with the network layer number, the size is continuously reduced, and high-level features are abstracted from an input image.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010403907 | 2020-05-13 | ||
| CN2020104039078 | 2020-05-13 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111652308A CN111652308A (en) | 2020-09-11 |
| CN111652308B true CN111652308B (en) | 2024-02-23 |
Family
ID=72346888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010476319.7A Active CN111652308B (en) | 2020-05-13 | 2020-05-29 | Flower identification method based on ultra-lightweight full convolutional neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111652308B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3171297A1 (en) * | 2015-11-18 | 2017-05-24 | CentraleSupélec | Joint boundary detection image segmentation and object recognition using deep learning |
| CN107480649A (en) * | 2017-08-24 | 2017-12-15 | 浙江工业大学 | A kind of fingerprint pore extracting method based on full convolutional neural networks |
| CN108416440A (en) * | 2018-03-20 | 2018-08-17 | 上海未来伙伴机器人有限公司 | A kind of training method of neural network, object identification method and device |
| CN109325484A (en) * | 2018-07-30 | 2019-02-12 | 北京信息科技大学 | Flowers image classification method based on background priori conspicuousness |
| KR101979654B1 (en) * | 2018-01-15 | 2019-05-17 | 주식회사 비엔인더스트리 | License plate recognition apparatus and the method thereof |
| KR20190098671A (en) * | 2018-02-14 | 2019-08-22 | 삼성전자주식회사 | High speed processing method of neural network and apparatus using thereof |
| CN110188705A (en) * | 2019-06-02 | 2019-08-30 | 东北石油大学 | A kind of remote road traffic sign detection recognition methods suitable for onboard system |
| CN110853009A (en) * | 2019-11-11 | 2020-02-28 | 北京端点医药研究开发有限公司 | Retina pathology image analysis system based on machine learning |
| CN110929603A (en) * | 2019-11-09 | 2020-03-27 | 北京工业大学 | Weather image identification method based on lightweight convolutional neural network |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| ATE514144T1 (en) * | 2001-10-16 | 2011-07-15 | Univ Chicago | COMPUTER-ASSISTED DETECTION OF THREE-DIMENSIONAL LESIONS |
| US9563825B2 (en) * | 2014-11-20 | 2017-02-07 | Adobe Systems Incorporated | Convolutional neural network using a binarized convolution layer |
| US11222263B2 (en) * | 2016-07-28 | 2022-01-11 | Samsung Electronics Co., Ltd. | Neural network method and apparatus |
| US10878576B2 (en) * | 2018-02-14 | 2020-12-29 | Elekta, Inc. | Atlas-based segmentation using deep-learning |
-
2020
- 2020-05-29 CN CN202010476319.7A patent/CN111652308B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3171297A1 (en) * | 2015-11-18 | 2017-05-24 | CentraleSupélec | Joint boundary detection image segmentation and object recognition using deep learning |
| CN107480649A (en) * | 2017-08-24 | 2017-12-15 | 浙江工业大学 | A kind of fingerprint pore extracting method based on full convolutional neural networks |
| KR101979654B1 (en) * | 2018-01-15 | 2019-05-17 | 주식회사 비엔인더스트리 | License plate recognition apparatus and the method thereof |
| KR20190098671A (en) * | 2018-02-14 | 2019-08-22 | 삼성전자주식회사 | High speed processing method of neural network and apparatus using thereof |
| CN108416440A (en) * | 2018-03-20 | 2018-08-17 | 上海未来伙伴机器人有限公司 | A kind of training method of neural network, object identification method and device |
| CN109325484A (en) * | 2018-07-30 | 2019-02-12 | 北京信息科技大学 | Flowers image classification method based on background priori conspicuousness |
| CN110188705A (en) * | 2019-06-02 | 2019-08-30 | 东北石油大学 | A kind of remote road traffic sign detection recognition methods suitable for onboard system |
| CN110929603A (en) * | 2019-11-09 | 2020-03-27 | 北京工业大学 | Weather image identification method based on lightweight convolutional neural network |
| CN110853009A (en) * | 2019-11-11 | 2020-02-28 | 北京端点医药研究开发有限公司 | Retina pathology image analysis system based on machine learning |
Non-Patent Citations (4)
| Title |
|---|
| 吕石磊，卢思华等.基于改进YOLOv3-LITE轻量级神经网络的柑橘识别方法.《农业工程学报》.2019,第35卷(第17期),205-214页. * |
| 应用卷积神经网络识别花卉及其病症;林君宇;李奕萱;郑聪尉;罗雯波;许蕾;;小型微型计算机系统(06);全文 * |
| 李兴维 ; 朱泽群 ; 万秋波 ; 屈金山.基于深度学习的实时车牌检测.《信息通信》.2019,全文. * |
| 自然背景下的梅花图像分割算法研究;张娟;韩殿元;黄心渊;;北京林业大学学报(03);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111652308A (en) | 2020-09-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110738207A (en) | character detection method for fusing character area edge information in character image | |
| CN111368896A (en) | Hyperspectral remote sensing image classification method based on dense residual three-dimensional convolutional neural network | |
| Kadam et al. | Detection and localization of multiple image splicing using MobileNet V1 | |
| CN109190752A (en) | The image, semantic dividing method of global characteristics and local feature based on deep learning | |
| CN111126202A (en) | Optical remote sensing image target detection method based on void feature pyramid network | |
| CN109190626A (en) | A kind of semantic segmentation method of the multipath Fusion Features based on deep learning | |
| CN106354735A (en) | Image target searching method and device | |
| CN105956560A (en) | Vehicle model identification method based on pooling multi-scale depth convolution characteristics | |
| CN107665324A (en) | A kind of image-recognizing method and terminal | |
| CN110633708A (en) | Deep network significance detection method based on global model and local optimization | |
| CN111709311A (en) | Pedestrian re-identification method based on multi-scale convolution feature fusion | |
| CN110188816B (en) | Image fine granularity identification method based on multi-stream multi-scale cross bilinear features | |
| CN111932511B (en) | Electronic component quality detection method and system based on deep learning | |
| CN113240691A (en) | Medical image segmentation method based on U-shaped network | |
| CN111652273B (en) | Deep learning-based RGB-D image classification method | |
| CN101169830A (en) | Human face portrait automatic generation method based on embedded type hidden markov model and selective integration | |
| CN112232229A (en) | Fine water body extraction method based on U-net neural network | |
| CN114694038A (en) | High-resolution remote sensing image classification method and system based on deep learning | |
| CN111368825A (en) | Pointer positioning method based on semantic segmentation | |
| CN115984172A (en) | Small target detection method based on enhanced feature extraction | |
| CN113269224A (en) | Scene image classification method, system and storage medium | |
| CN114882599A (en) | Off-line handwritten signature segmentation system and method based on double-branch neural network | |
| CN115272278A (en) | Method for constructing change detection model for remote sensing image change detection | |
| Yin et al. | Road Damage Detection and Classification based on Multi-level Feature Pyramids. | |
| CN116052110B (en) | Intelligent positioning method and system for pavement marking defects |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |