CN112464942A

CN112464942A - Computer vision-based overlapped tobacco leaf intelligent grading method

Info

Publication number: CN112464942A
Application number: CN202011166359.8A
Authority: CN
Inventors: 王欢; 刘振
Original assignee: Nanjing University of Science and Technology
Current assignee: Nanjing University of Science and Technology
Priority date: 2020-10-27
Filing date: 2020-10-27
Publication date: 2021-03-09
Anticipated expiration: 2040-10-27
Also published as: CN112464942B

Abstract

The invention discloses an intelligent grading method for overlapped tobacco leaves based on computer vision, which comprises the following steps: and simulating the synthesis of the overlapped tobacco leaves to obtain the flue-cured tobacco leaf image of the random-grade tobacco leaf overlapped tobacco leaves. And (3) according to the concave-convex point properties of the overlapped tobacco leaves, segmenting the overlapped tobacco leaves, and extracting shape texture characteristics, color characteristics and characteristics output by a VGG16 deep network for the complete tobacco leaves. And (3) training 3 different SVM classifiers by using the characteristics, and integrating the prediction results of the three individual classifiers by using the idea of ensemble learning. For segmented tobacco leaf identification, only the shape features need to be zeroed out. The invention can combine the traditional characteristic extraction technology with the deep neural network characteristic extraction technology by using a computer vision method, and can effectively grade complete single-leaf tobacco leaves and even overlapped tobacco leaves by using an integration technology, so that the automatic grading of the tobacco leaves can replace the manual grading.

Description

Computer vision-based overlapped tobacco leaf intelligent grading method

Technical Field

The invention belongs to an image segmentation technology, and particularly relates to an intelligent overlapping tobacco leaf grading method based on computer vision.

Background

As an important economic crop, the tobacco plays an important role in national economic construction. In order to process and produce cigarettes of different levels, the original flue-cured tobacco leaves need to be purchased in a grading manner. The standard for grading flue-cured tobacco leaves comprises: maturity, leaf structure, identity, oil content, color, length, disability level. Manual grading can only depend on comprehensive judgment of various senses such as vision, touch, taste and the like. On a large-scale tobacco leaf grading site, visual fatigue is often generated by workers, and emotional fluctuation exists, so that the traditional grading method has strong subjectivity. Normal workers often need years of grading training to achieve qualified grading accuracy. Tobacco leaf grading has significant practical value and great challenges.

The conventional vision method for grading the flue-cured tobacco leaves, namely the traditional vision processing technology, comprises the steps of feature extraction and classifier training, and can achieve certain intelligence. Since the features need to be designed by themselves, only a part of the known expert knowledge, such as the length, color, etc. of the tobacco leaves, can be utilized. However, as the season changes, the characteristics may change, such as the color of the dehydrated tobacco leaves becomes darker, the length of curls becomes shorter, and the like. This is a disadvantage of manually extracting features. With the development of deep learning and artificial intelligence technologies, examples of image classification and identification by using a deep learning method are enumerated, for example, the most common face identification technology applied to public safety is adopted, so that the deep learning method is an important way for solving flue-cured tobacco leaf grade identification.

At present, the tobacco classification problem is solved by means of good characteristics and a strong classifier, however, most of current researches classify one piece of tobacco, and most of the current researches only distinguish different parts, such as upper tobacco, middle tobacco and lower tobacco. In actual production, a stack of overlapped tobacco leaves needs to be classified, and the classification includes the differentiation of different grades at the same part. How to separate each piece of tobacco from a stack of tobacco leaves is a very difficult problem faced at present.

Disclosure of Invention

The invention aims to provide an intelligent overlapping tobacco leaf grading method based on computer vision.

The technical scheme for realizing the purpose of the invention is as follows: an intelligent overlapping tobacco leaf grading method based on computer vision comprises the following specific steps:

step 1: synthesizing overlapped tobacco leaves by using single flue-cured tobacco leaves;

step 2: segmenting the overlapped image according to the properties of the concave-convex points;

and step 3: extracting shape texture characteristics, color characteristics and depth network output characteristics of the flue-cured tobacco leaf image;

and 4, step 4: and (4) respectively training different classifiers by using the features extracted in the step (3), and integrating the prediction results of the different classifiers according to the idea of ensemble learning to obtain a grading result.

Preferably, the specific method for synthesizing the overlapped tobacco leaves by using the single flue-cured tobacco leaves comprises the following steps:

step 1.1: denoising the single flue-cured tobacco leaf image;

step 1.2: rotating and translating the denoised single flue-cured tobacco leaf image;

step 1.3: and overlapping the rotated and translated single-sheet flue-cured tobacco leaf images to obtain an overlapped tobacco leaf image.

Preferably, the method comprises the following steps of segmenting the overlapped tobacco leaves according to the concave-convex point properties of the overlapped tobacco leaves:

step 2.1: preprocessing the overlapped tobacco leaf images, detecting the number of connected domains, and directly obtaining segmented tobacco leaf images when the number of the connected domains is larger than N-1 connected domains, and performing the step 3; otherwise, performing step 2.2;

step 2.2: removing leaf ears of the overlapped tobacco leaves by utilizing opening operation;

step 2.3: detecting concave-convex points;

step 2.4: gathering the strong salient points into A areas by a kmeans algorithm, gathering the strong concave points into V areas, setting labels, and setting the same labels for contour points of the same area;

step 2.5: the contour between two adjacent pit areas is a contour segment, and the number CN of the contour segments is counted;

step 2.6: carrying out ellipse fitting on any segment I of the CN segment outline segment, finding the main stem of the tobacco leaf on the uppermost layer by utilizing the probability Hough transformation of a straight line, distinguishing the fitting ellipse of the tobacco leaf on the upper layer, and taking the rest part as the image of the tobacco leaf on the lower layer.

Preferably, the specific method for detecting the concave-convex points is as follows:

acquiring an outer contour image of the filtering image;

based on the outer contour image, dividing the contour points into concave points and convex points, and the specific method comprises the following steps:

expanding the boundary of the outer contour image outwards by R, wherein the pixel value of the expanded area is 0;

drawing a circle by taking the contour point as the center and the set numerical value as the radius, drawing a circle by taking the contour point as the center and the radius as R, and performing difference operation on the original contour map to obtain the area A of the disjoint areas_b

Order to

If A_bDetermining the contour point as a strong salient point if A is more than or equal to 1.2S_bIf the contour point is less than or equal to 0.8S, the contour point is a strong concave point.

Preferably, the extracted shape texture features include: perimeter, area, breakage rate, circularity, major axis length, minor axis length, aspect ratio.

Preferably, the extracted color features include: including 16384-dimensional HSV histogram color features, 11-dimensional HSV three-component mean and standard deviation, B, G color component mean and B, G, R standard deviation.

Preferably, the features extracted in step 3 are used to train different classifiers respectively, and the prediction results of the different classifiers are integrated according to the idea of ensemble learning, and the specific steps for obtaining the classification result are as follows:

step 4.1: performing distinguishable feature selection on the color features and the depth network features;

step 4.2: carrying out maximum and minimum value normalization on the three characteristics;

step 4.3: respectively inputting the normalized 3 types of features into an SVM trainer for training, wherein a kernel function of the SVM trainer is a radial basis function, and optimal parameters C and G are found through grid optimization to obtain 3 trained SVM models;

step 4.4: and (3) utilizing the 3 SVM models to carry out result prediction on the tobacco leaf image to obtain 3 different prediction results, wherein the prediction result is a 6-dimensional vector, the 1 st dimension is a prediction label, the last 5 dimensions are prediction probabilities, the 3 different prediction probabilities are summed, if the ratio of the maximum probability to the next maximum probability is greater than a set threshold T, the prediction label corresponding to the maximum probability is selected, and otherwise, the prediction label corresponding to the next maximum probability is selected.

Preferably, the specific steps of performing distinguishable feature selection on the color feature and the depth network feature are as follows:

the value of the dimension feature is calculated using the formula of the gradeability criterion:

wherein the content of the first and second substances,

is the average of the ith feature over the jth class data set,

the average of the ith feature over the entire data set,

class j kth sample point ith feature;

the characteristic value is according to F_iAnd sorting from large to small, and selecting the color characteristics of the front a dimension and the depth network of the b dimension.

Preferably, the formula for the maximum and minimum normalization of the three features is as follows:

in the formula, x_maxAnd x_minThe maximum value and the minimum value of a certain one-dimensional feature are obtained, x is an element of a one-dimensional feature vector, and y is an element after normalization.

Compared with the prior art, the invention has the following remarkable advantages:

according to the tobacco leaf grading method, the concave-convex point detection and the elliptical contour segment fitting technology are utilized, the complicated overlapped tobacco leaves can be segmented, so that not only can the single tobacco leaves be graded, but also the overlapped tobacco leaves can be graded, the more practical tobacco leaf grading problem can be solved, and the tobacco leaf grading efficiency is improved;

the method can distinguish and select the extracted tobacco leaf characteristics, integrates the prediction results of the 3 SVM models, and can remarkably improve the accuracy of tobacco leaf classification.

Drawings

Fig. 1 is a schematic diagram of an image preprocessing process, in which fig. 1(a) is an original image, fig. 1(b) is a grayscale image, fig. 1(c) is a binary image, and fig. 1(d) is an image with noise removed.

Fig. 2 is a schematic diagram of removing a petiole, in which fig. 2(a) is an outer contour region diagram of a tobacco leaf, fig. 2(b) is an open operation diagram, fig. 2(c) is a diagram after the petiole is removed, fig. 2(d) is an edge protection filter diagram, fig. 2(e) is a median filter diagram after the jaggy is removed, and fig. 2(f) is an outer contour region diagram after the jaggy is removed.

FIG. 3 is a schematic diagram of concave-convex point detection, in which FIG. 3(a) is a schematic diagram of concave-convex point, FIG. 3(b) is a diagram of drawing a circle with a certain contour point as the center, and FIG. 3(c) is a non-intersecting region area A_bFig. 3(d) is a schematic diagram of two highly overlapped tobacco leaves, and fig. 3(e) and 3(f) are visual diagrams of concave and convex points of the tobacco leaves.

Fig. 4 is a schematic diagram of setting labels for concave-convex points, where fig. 4(a) is a convex point on a binary image, fig. 4(b) is a convex point corresponding to a tobacco leaf, fig. 4(c) is a concave point on a binary image, fig. 4(d) is a concave point corresponding to a tobacco leaf, and fig. 4(e) is a concave-convex point corresponding to a tobacco leaf.

Fig. 5 is a schematic diagram of segmentation of an outline based on relief point labels.

FIG. 6 is a schematic diagram of four ellipse fits based on contour segments.

Fig. 7 is a schematic view of the main stem of tobacco leaf.

Fig. 8 is a schematic diagram of main stem detection based on probabilistic hough transform, fig. 8(a) is a tobacco gray scale map, fig. 8(b) is maximum and minimum filtering, fig. 8(c) is a difference image, fig. 8(d) is a canny edge image, fig. 8(e) is a binary image including main stems, and fig. 8(f) is a diagram of a probabilistic hough transform straight line detection result.

FIG. 9 is a diagram illustrating the segmentation result of the overlapped tobacco leaves.

FIG. 10 is a schematic view of tobacco leaves in two different color spaces. In which fig. 10(a) is RGB color space, fig. 10(b) is HSV color space, and fig. 10(c) is HSV cone model.

FIG. 11 is a histogram of HSV colors of different tobacco leaves, V, S, and H from left to right.

FIG. 12 is a network framework for vgg16 extraction of flue-cured tobacco leaf characteristics.

FIG. 13 is a confusion matrix of the 1 st model test results of data set 201907_ 1.

FIG. 14 is a confusion matrix of the 2 nd model test results of data set 201907_ 1.

FIG. 15 is a confusion matrix of the 3 rd model test results of data set 201907_ 1.

FIG. 16 is a confusion matrix for the integration of 3 model results in data set 201907_ 1.

Detailed Description

An intelligent overlapping tobacco leaf grading method based on computer vision firstly simulates the synthesis of overlapping tobacco leaves to obtain flue-cured tobacco leaf images of random-grade tobacco leaf overlapping. And then, segmenting the overlapped tobacco leaves according to the concave-convex point properties of the overlapped tobacco leaves. For the complete tobacco leaf, the shape texture feature, the color feature and the feature of the VGG16 deep network output are extracted. And (3) training 3 different SVM classifiers by using the characteristics, and integrating the prediction results of the three individual classifiers by using the idea of ensemble learning. For the identification of the segmented tobacco leaves, the shape features are only required to be set to zero. The invention can combine the traditional characteristic extraction technology with the deep neural network characteristic extraction technology by using a computer vision method, and can effectively grade complete single-leaf tobacco leaves and even overlapped tobacco leaves by using an integration technology, so that the automatic grading of the tobacco leaves can replace the manual grading. The method comprises the following specific steps:

step 1: synthesizing an overlapped tobacco leaf image by using the existing single-leaf flue-cured tobacco leaf image;

in a further embodiment, the method comprises the following specific steps:

step 1.1: denoising the single flue-cured tobacco leaf image;

in some embodiments, the specific method of denoising is:

the method comprises the steps of carrying out gray level processing on a single flue-cured tobacco leaf image, then carrying out thresholding to obtain a binary image, and eliminating noise by limiting the area of a communicated region. And traversing each connected region of the image, and if the area of the connected region is smaller than the set threshold value S, making the RGB pixel values of the region all be 0. And carrying out pixel-based operation on the mask and the original image in the maximum communication area to obtain a pure tobacco leaf image.

In some embodiments, the specific method for superimposing the rotated and translated single-leaf flue-cured tobacco leaf images is as follows:

an image matrix L is set to store labels of the composite image, each tobacco leaf corresponds to a mask of a different COLOR, the COLOR _ TAB is set to be a COLOR array, and the dimension is N. Synthesizing N images, wherein N-1 times of superposition operation is required, and the specific synthesis method comprises the following steps:

initial value setting: inputting two tobacco leaf images T1 and T2 processed in the step 1.2, wherein the synthesis time T is 1

1. Carrying out graying and binarization on T1 and T2 respectively to obtain images B1 and B2;

2. traversing the binary map B1 of the image T1;

2.1, if the traversed pixel is 0, then the pixel of the image T1 and the pixel of the region RGB corresponding to the image matrix L are also set to 0.

And 2.2, if the traversed pixel is not 0 and T is 1, setting the COLOR of the area corresponding to the image matrix L as the COLOR in 1 st of the COLOR _ TAB, and T is 2.

3. Traversing the binary map B2 of the image T2;

if the traversed pixel is not 0, the pixel value corresponding to T1 is modified to be the pixel value of T2, and the pixel COLOR corresponding to L is set to be the Tth COLOR of COLOR _ TAB.

4. And recording the next tobacco leaf image participating in the synthesis as T2, if T is equal to N, exiting the loop, otherwise, continuing the step 1.

After the above 4 steps are completed, a synthesized tobacco leaf image T1 and label image L can be obtained.

Step 2: according to the concave-convex point property of the overlapped tobacco leaves, the overlapped tobacco leaves are divided, and the method comprises the following specific steps:

step 2.1: preprocessing the overlapped tobacco leaf images, detecting the number of connected domains, and directly obtaining segmented tobacco leaf images when the number of the connected domains is larger than N-1, namely one connected domain corresponds to one segmented tobacco leaf image, and performing the step 3; otherwise, step 2.2 is performed.

Specifically, the overlapped tobacco leaf image is grayed and binarized, and the overlapped tobacco leaf original image and the mask image obtained by binarization are subjected to pixel-by-pixel and operation, so that an image with partial noise removed is obtained. In the image synthesis process, there may be no overlapping region between the two images, so the number C of connected regions that satisfy the condition by detecting the images is required, and if the number of pixels of the outline of the connected region is less than a certain threshold, the threshold taken in this embodiment is 100, the statistics of the number of connected regions is not performed, that is, the region is not considered to be the tobacco leaf image. And when the number C of the connected areas is larger than N-1, directly obtaining N independent tobacco leaf images, and performing the step 3. Otherwise, if the N tobacco leaf images are mutually overlapped, the step 2.2 is carried out.

Step 2.2: and removing leaf ears of the overlapped tobacco leaves by utilizing opening operation.

In some embodiments, the specific method is: performing morphological open operation on the outer contour region image, wherein the shape of the structural element is an ellipse; and performing pixel-by-pixel and operation and filtering processing on the overlapped original tobacco leaf image and the opening operation image to obtain a filtered image with the petioles removed.

Further, the filtering process adopts edge protection filtering and median filtering.

Step 2.3: the method for detecting the concave-convex points comprises the following steps:

acquiring an outer contour image of the filtering image;

and expanding the boundary of the outline image outwards by R, wherein the pixel value of the expanded area is 0.

The difference operation is performed between the original contour map (fig. 2(f)) and the circle 3(b) obtained by drawing a circle with the contour point as the center and the set numerical value as the radius, and drawing a circle with the contour point as the center and the radius of R in sequence to obtain the area a of the non-intersecting region_b(FIG. 3(c)) in

If A_bDetermining the contour point as a strong salient point if A is more than or equal to 1.2S_bIf the contour point is less than or equal to 0.8S, the contour point is a strong concave point;

counting the number V of strong pits and the total number N of contour points if

Step 3 is carried out, otherwise step 2.4 is carried out;

step 2.4: and (3) gathering the strong salient points into A areas and gathering the strong concave points into V areas by a kmeans algorithm, and setting labels. Contour points of the same area have the same label. Wherein, the label of the strong salient point is positive, and the label of the strong concave point is negative.

Step 2.5: based on the contour segment of the contour point label, the contour between two adjacent pit areas is a contour segment, and the number CN of all contour segments satisfying such a property is recorded.

Step 2.6: carrying out ellipse fitting on any segment I of the CN segment contour segment, finding the main stem of the tobacco leaf on the uppermost layer by utilizing the probability Hough transformation of a straight line, distinguishing the fitting ellipse of the tobacco leaf on the upper layer, and taking the rest parts as the image of the tobacco leaf on the lower layer, wherein the method specifically comprises the following steps:

step 2.6.1: carrying out ellipse fitting on any segment I (I is 1,2,3) of the CN segment profile segment, and recording the area of the obtained fitting ellipse as E_a. Recording the area of the outer contour of the tobacco leaf as C_aThe center point of the fitted ellipse is E_cIf 0.35C_a＜E_a＜2C_aAnd E is_cIf the coordinate value of the ellipse is in the tobacco leaf image, the fitted ellipse is a candidate ellipse. Defining ellipse compactness E_ρThe white pixels inside the fitted ellipse account for the proportion of the total pixels of the ellipse. Calculating included angles E between all candidate ellipses and the horizontal direction_βTightness of ellipse E_ρ。

Step 2.6.2: the median filtered image (fig. 2 e) from which the noise has been removed is subjected to a gradation process (fig. 8 a), and then subjected to a maximum-minimum filtering process (fig. 8 b). Then, the difference is calculated to obtain a difference image (fig. 8(c)), and then, the threshold is applied to the difference image, and canny edge detection is performed to obtain an edge image (fig. 8 (d)). Finally, the edge of the canny edge image is eliminated, and only the main stem area of the tobacco leaf area is reserved (fig. 8 (e)).

Step 2.6.3: probability hough line detection is performed on the binary image (fig. 8(e)) including the main stem obtained in step 2.6.2, and the longest straight line segment L is found (fig. 8 (f)). Calculating the included angle L between the straight line segment and the horizontal direction_α. The height of the image is H. Calculating the center E of the ellipse_cDistance D to the straight line L. One index is defined:

and sorting all the candidate ellipses from big to small according to score, and selecting the ellipse with the highest score as a fitting ellipse of the tobacco leaf on the uppermost layer. The uppermost leaf can be divided by the elliptical mask and subtracted from the original image to obtain the lowermost leaf image (fig. 9).

And step 3: and extracting shape texture characteristics and color characteristics of the flue-cured tobacco leaf image, inputting the flue-cured tobacco leaf image into a deep learning network, and extracting an output result of the last pooling layer as the deep network characteristics.

Specifically, the extracted shape texture features include: perimeter, area, breakage rate, circularity, major axis length, minor axis length, aspect ratio (major axis length/minor axis length).

The extracted color features include: including 16384-dimensional HSV histogram color features, 11-dimensional HSV three-component mean and standard deviation, B, G color component mean and B, G, R standard deviation.

The extracted VGG deep network features comprise: the convolutional neural network has strong feature extraction capability, so that the method selects the common classification network VGG16 to extract image features, and only the output result of the last pooling layer of the VGG16 is reserved as the final feature.

And 4, step 4: and training different classifiers by using different characteristics, and integrating prediction results of the different classifiers according to the idea of ensemble learning.

According to the obtained 3 types of different features, the color features and the features extracted by the deep neural network are selected by the differentiable features, and 110-dimension and 200-dimension are respectively selected. Maximum and minimum normalization is then performed. And finally, respectively training 3 SVM classifiers, and integrating the results of different classifiers by using the idea of ensemble learning, wherein the method specifically comprises the following steps:

in particular, the feature selection is implemented using the following indexable criteria:

wherein the content of the first and second substances,

is the average of the ith feature over the jth class data set,

the average of the ith feature over the entire data set,

class j kth sample point ith feature. The characteristic value is according to F_iSorting from big to small, and selecting the color characteristic of the front a dimension and the depth network of the b dimension;

in the embodiment, the color features of the first 100 dimensions are selected, the features extracted by the VGG deep learning network of the first 200 dimensions are selected, the serial numbers of the dimensions of the selected features are stored, and the features of the test set are selected in the test stage according to the serial numbers.

Step 4.2: and carrying out maximum and minimum value normalization on the three characteristics, wherein the normalization formula is as follows:

in the formula, x_maxAnd x_minThe maximum value and the minimum value of a certain one-dimensional characteristic. x is the element of the feature vector of a certain dimension, and y is the normalized element. And (3) respectively carrying out normalization processing on the 3 features extracted in the step (3), and keeping the minimum value and the maximum value of all dimensions of each feature.

Step 4.3: and (3) inputting the 3 types of features processed in the step (4.2) into an SVM trainer for training, wherein a kernel function of the SVM trainer is a Radial Basis Function (RBF), and finding an optimal parameter penalty coefficient C and an optimal kernel function parameter G through grid optimization to finally obtain 3 SVM models.

Specifically, before SVM training, it is necessary to select a kernel function of a trainer (RBF in this example), a fold number of cross validation (5-fold cross validation in this example), a stopping condition eps of training (0.001 in this example), whether probability estimation is performed (yes in this example), and the others are default values, set parameters C and G by using grid optimization, then SVM training is performed, and when training accuracy reaches the stopping condition, training is stopped.

The tobacco grade of the flue-cured tobacco has 42 grades, and the tobacco grade is mapped to different positive integer labels according to the quality of the tobacco. B2F corresponds to a label of 7, B3F corresponds to a label of 19, C2F corresponds to a label of 2, C3F corresponds to a label of 3, C4F corresponds to a label of 13, wherein the meaning of the ranking is: "B" represents the upper part of the tobacco leaf; "C" represents the middle of the tobacco leaf; "F" represents the color of the tobacco leaves is orange; "1, 2,3, 4" means that the quality of the tobacco leaves is excellent, good, and general.

Step 4.4: and (3) utilizing the 3 SVM models to carry out result prediction on the tobacco leaf image to obtain 3 different prediction results, wherein the prediction result is a 6-dimensional vector, the 1 st dimension is a prediction label, the last 5 dimensions are prediction probabilities, and the label corresponding to the maximum probability value is the 1 st dimension prediction label. And summing the 3 different prediction probabilities, if the ratio of the maximum probability to the second maximum probability is greater than a set threshold T, taking the classification label corresponding to the maximum probability, and otherwise, taking the prediction label corresponding to the second maximum probability.

Examples

An intelligent overlapping tobacco leaf grading method based on computer vision comprises the following specific steps:

step 1: and (3) synthesizing the overlapped tobacco leaves by utilizing the existing single-piece flue-cured tobacco leaves.

In this embodiment, two tobacco images are synthesized, and the segmentation problem of two overlapped tobacco leaves is discussed. This embodiment has three data sets 201907_1, 201907_2, 201907_3, respectively, each having 5 levels of tobacco leaf samples, B2F, B3F, C2F, C3F, C4F, respectively. The corresponding labels are 7, 19, 2,3, 13. Each level in turn has a front view and a back view. Each sample of tobacco leaves corresponds to a front view and a back view. Each data set was randomly partitioned into a training set and a test set in a 7:3 ratio. The training set is used for training an image grading model, and the testing set is used for synthesizing and grading the overlapped tobacco leaf images.

For any two tobacco leaf images, denoising the two tobacco leaf images, rotating the two tobacco leaves at random angles and randomly translating the positions of the two tobacco leaves, and finally superposing the images.

Step 1.1: denoising, namely performing graying treatment on the single flue-cured tobacco leaf image, then performing thresholding to obtain a binary image, and eliminating noise by limiting the size of the area of a communicated region. And traversing each connected region of the image, and if the area of the connected region is smaller than the set threshold value S, making the RGB pixel values of the region all be 0. And carrying out pixel-based operation on the mask and the original image in the maximum communication area to obtain a pure tobacco leaf image.

Step 1.2: and (3) image transformation, namely rotating the image obtained by denoising, setting a rotation center and a rotation angle of a rotation transformation matrix, and taking the center of the image by the rotation center. And then carrying out translation transformation, wherein a translation matrix is as follows:

δ_xand delta_yIndicating the amount of translation in the horizontal and vertical directions, respectively.

Step 1.3: and (3) image superposition, namely setting an image matrix L to store a label of a synthesized image, wherein each tobacco leaf corresponds to a mask with different colors, and the synthesis of the images of 2 tobacco leaves needs to be carried out for 1 time of superposition operation. The tobacco synthesis algorithm is as follows:

1. carrying out graying and binarization on the two images T1 and T2 after image transformation to obtain B1 and B2 respectively;

2. traversing the binary map B1 of the image T1;

2.1, if the traversed pixel is 0, then T1 and the RGB pixel of the region corresponding to the image matrix L are also set to 0.

2.2, if the traversed pixel is not 0, then set the region corresponding to L to red, and the corresponding RGB is (255,0,0), that is, the mask corresponding to T1 is red.

3. Binary map of traversal image T2

If the traversed pixel is not 0, the pixel value corresponding to T1 is modified to be the pixel value of T2, the pixel color corresponding to L is set to be blue, and the corresponding RGB is (0, 255).

After the above 3 steps are completed, two tobacco leaf composite images T1 and a label image L can be obtained. The red area of the label image is the mask corresponding to the bottom tobacco leaf, and the blue area is the mask corresponding to the top tobacco leaf.

Step 2: and segmenting the overlapped image according to the properties of the concave-convex points.

In the embodiment, a composite image of two tobacco leaf images is used for segmentation;

step 2.1: preprocessing the overlapped tobacco leaf images, detecting the number of connected domains, and directly obtaining segmented tobacco leaf images when more than 1 connected domain exists, namely one connected domain corresponds to one segmented tobacco leaf image, and performing the step 3; otherwise, step 2.2 is performed.

The image to be processed is grayed and then binarized, and the original image and the mask image obtained by binarization are subjected to pixel-by-pixel and operation, so that an image with partial noise removed can be obtained. In the image synthesis process, there may be no overlapping region between the two images, so the number C of connected regions that satisfy the condition by detecting the images is required, and if the number of pixels of the outline of the connected region is less than a certain threshold, the threshold taken in this embodiment is 100, the statistics of the number of connected regions is not performed, that is, the region is not considered to be the tobacco leaf image. And (3) when the number C of the connected areas is more than 1, directly obtaining two independent tobacco leaf images, and performing the step (3). If the connected region is 1, i.e. the two tobacco leaf images are superimposed on each other, step 2 is performed.

And performing morphological open operation on the outer contour region image, wherein the shape of the structural element is an ellipse, and the size of the structural element is (17, 17). Although two adjacent tobacco leaves may become a connected region after the image opening operation, the original image and the opening operation image are subjected to pixel-by-pixel operation, so that the problem can be avoided, and the petiole tobacco leaf removed image can be obtained. Then, the image is subjected to filtering processing without destroying edges as much as possible, so that edge protection filtering is adopted. In order to eliminate the bad influence of the jaggy on the concave-convex point detection, median filtering is used to remove the jaggy. And taking the image after the sawtooth is removed as an analysis object of image segmentation.

acquiring an outer contour image of the filtering image;

If A_bIf the contour point is more than or equal to 1.2S, determining the contour point as a strong salient point, and if A is greater than or equal to 1.2S, determining the contour point as a strong salient point_bIf the contour point is less than or equal to 0.8S, the contour point is a strong concave point;

Step 3 is carried out, otherwise step 2.4 is carried out;

step 2.4: set label for concave-convex points

The strong salient points are classified, and the strong salient points are concentrated at two ends of the tobacco leaves, and can be classified into 4 types through a kmeans algorithm, wherein the class labels after the kmeans classification are 0,1,2 and 3. An add 1 operation is required to bring its label to 1,2,3, 4. The strong pits are concentrated in the intersection area of the two tobacco leaves, the strong pits can be classified into 4 types through a kmeans algorithm, and after the kmeans classification, the class labels are 0,1,2 and 3. The operation of adding 1 and taking negative numbers is needed to make the label become-1, -2, -3, -4. The labels of such outlines are divided into positive, negative and 0. Positive numbers represent strong peaks and negative numbers represent strong valleys. 0 represents a general contour point of concavity and convexity. In the experimental process, the number of the categories can be increased a little, and the more general image overlapping problem can be solved.

Step 2.5: based on the contour segment of the contour point label, the contour between two adjacent pit areas is a contour segment, and the number of all contour segments satisfying such a property is recorded as CN.

In order to eliminate the influence of noise on the pits, that is, only a few strong pits appear in a section of the contour, we need to treat these strong pits as noise and assign their labels to 0. Specifically, a sliding window with the length of 10 is constructed from contour points with the first label being a negative number, and labels with 10 consecutive contour points are stored in the window. If the number of negatives in the sliding window is less than 3, then the segment is considered to have no strong pits. And the sliding window continuously slides to the contour point of the next negative label, the judgment of the number of the negative labels is still carried out, and after all contours are traversed, pits with insufficient response can be eliminated.

Observing the distribution of the concave-convex points of the overlapped tobacco leaves (fig. 4(e)), it can be found that the contour segments on both sides of the concave points belong to different tobacco leaves, and the contour segments on both sides of the convex points belong to the same tobacco leaf. For example, in a clockwise direction, 4 to-1 and-1 to 1 belong to two different leaves, whereas-1 to 1 and 1 to-4 belong to the same leaf. The strong pits are therefore the demarcation points for the outer contour segments. This is a necessary consequence of tobacco leaf overlap.

We consider contour points with the same label as the same segment contour. The labels of the contour points are also a large number of 0's in addition to positive and negative numbers. Therefore, we need to convert the labels of these contour points into positive or negative numbers according to a certain rule.

For example, the labels of the contour points are 11000-4-40011, and we replace 0 with the previous non-zero label, which becomes 11111-4-4-4-411. If the first label is 0, e.g., 0011000-4-400-20, we replace it with-2-211111-4-4-4-2-2. I.e., the tag beginning with 0 is converted to the first non-zero tag reciprocal from the end. Note that the above is only for the sake of simplicity in explanation of the label switching rule, and does not refer to noise removal of strong pits.

For contour sequences starting with a positive number label, 11111-4-4-411, the positions where the symbols change are counted, for example from 1 to-4, with the index 4 being the position where the symbols change, and from-4 to 1, with the index 7 being the position where the symbols change. All negative tag intervals are then calculated from the positions where the sign changes, e.g., the negative tag interval of the sequence can be represented by [5,7 ]. For the

21-1-13355-4-4667788-3-312 we need to handle different positive numbers that occur in succession in the middle, 3355 to 3333 and 667788 to 666666. For the header 21, this is converted to 22, i.e. the first consecutive positive number, all taking the value of the first label. For the tail 12, it is converted to 22, i.e. the positive number of the tail continuation, all taking the value of the last label. The results were:

2 2 -1 -1 3 3 3 3 -4 -4 6 6 6 6 6 6 -3 -3 2 2

finally, the negative label is converted into the next positive label:

2 2 3 3 3 3 3 3 6 6 6 6 6 6 6 6 2 2 2 2

for contour sequences starting with negative number labels, e.g.

-1 -1 2 2 2 3 -2 -2 6 6 7 -1 -1

Also, we need to find the negative tag interval. [0,1],[6,7],[11,12]. The intermediate consecutive positive tags are then processed to yield:

-1 -1 2 2 2 2 -2 -2 6 6 6 -1 -1

and finally, converting the negative number label into the next positive number label to obtain:

2 2 2 2 2 2 6 6 6 6 6 2 2

step 2.6: carrying out ellipse fitting on any segment I of the CN segment contour segment, finding the main stem of the tobacco leaf on the uppermost layer by utilizing the probability Hough transformation of a straight line, distinguishing the fitting ellipse of the tobacco leaf on the top, and taking the rest parts as the image of the tobacco leaf on the bottom, wherein the method specifically comprises the following steps:

step 2.6.1: tobacco shape fitting based on elliptical model

Labeling different outlinesDisplayed in different colors. It can be seen that fig. 5 is divided into 4 profile segments. Because the shape of flue-cured tobacco leaves is similar to an ellipse, candidate ellipses can be obtained by fitting ellipses of different contour segment combinations. Fitting an ellipse to an arbitrary I (I ═ 1,2,3) segment of the CN segment profile segment (fig. 6), and the area of the obtained fitted ellipse is denoted as E_a. Recording the area of the outer contour of the tobacco leaf as C_aThe center point of the fitted ellipse is E_cIf 0.35C_a＜E_a＜2C_aAnd E is_cIf the coordinate value of the ellipse is in the tobacco leaf image, the fitted ellipse is a candidate ellipse. Defining ellipse compactness E_ρThe white pixels inside the fitted ellipse account for the proportion of the total pixels of the ellipse. Calculating included angles E between all candidate ellipses and the horizontal direction_βTightness of ellipse E_ρ。

Step 2.6.2: canny edge detection

The top tobacco leaf image of the two overlapped tobacco leaves can observe the complete main stem (figure 7), so the top tobacco leaf is selected according to the angle (direction) of the long axis of the fitting ellipse, the long axis length, the center of the long axis (the center of the ellipse) and the main stem detected by Hough transform.

First, the median filtered image (fig. 2 e) from which noise has been removed is subjected to a graying process (fig. 8 a), and then subjected to maximum-minimum filtering (fig. 8 b). Then, the difference is calculated to obtain a difference image (fig. 8(c)), and then, the threshold is applied to the difference image, and canny edge detection is performed to obtain an edge image (fig. 8 (d)). Finally, the edge of the canny edge image is eliminated, and only the main stem area of the tobacco leaf area is reserved (fig. 8 (e)).

Step 2.6.3: probabilistic Hough transform detection of main stems

Probability hough line detection is performed on the binary image (fig. 8(e)) including the main stem obtained in step 2.6.2, and the longest straight line segment L is found (fig. 8 (f)). Calculating the included angle L between the straight line segment and the horizontal direction_α. The height of the image is H. Calculating the center E of the ellipse_cDistance D to the straight line L. One index is defined:

And step 3: and extracting shape texture characteristics and color characteristics of the flue-cured tobacco leaf image, inputting the flue-cured tobacco leaf image into a deep learning network VGG16, and extracting an output result of the last pooling layer as deep network characteristics.

Three types of features are extracted, including shape texture features, color features, and depth features extracted by VGG 16.

a) Shape texture characteristics: the shape features include 7 and the texture features include 5.

The shape characteristics include perimeter, area, breakage rate, circularity, major axis length, minor axis length, aspect ratio.

The calculation of the perimeter C only needs to count the number of pixels of the outer contour of the maximum connected region.

And calculating the area A, namely counting the hole number h with the pixel value of 0 and the non-zero pixel number n of the area by traversing the binary image corresponding to the maximum communication area, so as to obtain the appearance area A of the tobacco leaf, wherein A is h + n.

The breakage rate is B, and B is h/A.

The calculation formula of the circularity CIL is as follows:

and carrying out ellipse fitting on the tobacco leaf image to obtain the long axis length, the short axis length and the ratio of the long axis length and the short axis length.

Texture features include entropy, energy, contrast, inverse distance, autocorrelation. Entropy represents the amount of information in an image, reflecting the degree of non-uniformity or complexity of texture in the image. The energy is the sum of squares of the gray level co-occurrence matrix element values, and reflects the uniformity degree and the texture thickness of the image gray level distribution. The contrast reflects the sharpness of the image and the depth of the texture grooves. The inverse difference reflects the homogeneity of the image texture and measures the local change of the image texture. And the autocorrelation is used to measure the degree of similarity of the elements of the spatial gray level co-occurrence matrix in the row or column direction.

b) Color characteristics: including 16395 dimensions, including 16384 dimensions of HSV histogram color features, 11 dimensions of mean and standard deviation of HSV three components, B, G color components of mean and B, G, R standard deviation (R mean is the same as V mean and is omitted).

Firstly, converting the color space of the image containing the minimum circumscribed rectangle of the tobacco leaves from RGB (figure 10(a)) into HSV (figure 10(b)), wherein the range of each component of the HSV is 0-255. Since the HSV colour space (fig. 10(c)) is more computer-distinguishable from differences in colour of tobacco. Then, the color grade quantification is carried out, because the colors of the flue-cured tobacco leaves are concentrated between red and yellow, the corresponding chroma is 0-60 degrees and 340-360 degrees. And the saturation and the brightness are both between 0.6 and 1. HSV is therefore quantified within these intervals.

For H, 0-60 degrees and 340-360 degrees respectively correspond to 0-42.5,240.8-255. For S and V, 0.6-1 corresponds to 153-255.

H, S, V are quantized into 32, 16 levels respectively. The quantization formula is as follows:

wherein h is_i、s_iv_iRespectively representing H, S, V color component values of the ith pixel. Quantifying HSV to 0-31, 0-31 andand the value range of 0-15. Then, carrying out pixel-by-pixel statistics to obtain a histogram function H', and finally carrying out histogram normalization:

wherein A is the area of the tobacco leaf.

Normalizing the sum of the color components of the HSV channel and the RGB channel to obtain

Wherein

And a color brightness value of the jth color component in the ith pixel, which represents the color space as c. Wherein N is the number of traversed pixels. Thus we have 16384-dimensional HSV color histogram features. And 5-dimensional color component mean features. Traversing the tobacco leaf area again to obtain the standard deviation of each color component:

we have obtained the standard deviation feature of the color component in 6 dimensions, plus the histogram feature in 16384 dimensions and the mean feature of the color component in 5 dimensions, so we can finally obtain the color feature in 16395 dimensions.

c) Features extracted by convolutional neural network (VGG16)

The convolutional neural network has strong feature extraction capability, so a common classification network VGG16 is selected to extract image features, the output result of the last pooling layer of VGG16 is only reserved, 512 feature maps are arranged in the layer, the dimension of each feature map is 7 x 7, and the feature matrix is subjected to flattening stretching to obtain 25088-dimensional features. The invention uses a dark learning framework of dark darknet to extract the features of the convolutional neural network. Flue-cured tobacco leaf images at 8000 x 4096 resolution are first scaled to 256 x 256 as input to the darknet frame and then to 224 x 224 as input to the VGG16 network. The network framework is shown in fig. 12, where the blue modules represent the convolutional layers (conv) and the activation function layers (relu) and the red modules represent the max pooling layers.

a) Feature selection and feature normalization

And extracting three different features of the training set according to the feature extraction method. Recording three characteristics as F₁,F₂,F₃. Wherein

N is the number of samples in the training set, D_iIs the dimension of the ith feature. D₁＝12,D₂＝16395,D₃＝131328。

Principal component analysis is the main means of feature dimension reduction, but when the feature dimension is very high, the requirement for memory is also increased, and the calculation time overhead is very large, so that the following gradable criterion is adopted to realize feature selection.

Wherein

Is the average of the ith feature over the jth class data set,

the average of the ith feature over the entire data set,

class j kth sample point ith feature. In the experimental process, the dimension of the color features is reduced to 100 dimensions, and the features extracted by the VGG deep learning network are usedReducing to 200 dimensions.

The maximum and minimum normalization is performed for the three features. The normalization formula is as follows:

the normalized interval in this embodiment is [ -1,1 [ ]]I.e. y_max＝1,y_min＝-1。x_maxAnd x_minThe maximum value and the minimum value of a certain one-dimensional characteristic. x is the element of the feature vector of a certain dimension, and y is the normalized element. Are respectively paired with F₁,F₂,F₃And (6) carrying out normalization processing. And the minimum and maximum values of all dimensions of each feature are retained for normalization of the test set data.

b) Integrated model training

The SVM is a simple and effective classification model, and the search of the optimal model is equivalent to the search of the optimal segmentation hyperplane. Knowing the feature vectors and labels, three different SVM classification models can be trained by using the LIBSVM tool library. The kernel function is RBF kernel, and the optimal classification model is searched by adopting five-fold cross validation. The RBF optimizing function can find the local optimal solution within the specified range only by continuously adjusting the values of C and G. And adding the classification probability results of the three different classifiers, if the ratio of the maximum probability to the next maximum probability is greater than a certain threshold, taking the classification label corresponding to the maximum probability, and otherwise, taking the classification result of the second classifier. The results are shown in tables 2-4, and FIGS. 13-16.

The average grading accuracy of the three types of characteristics in three different data sets is respectively as follows: 54.64%, 61.03%, 54.38%, the highest belongs to classifiers trained on color features, and the average accuracy of the integration of results by three classifiers is 65.95%. It can be seen that ensemble learning has good classification gain. The test set of table 5 was derived from the overlapping tobacco leaf combination of the 201807_1 test set, which was also the 201807_1 training set, except that no shape feature extraction was performed. It can be seen that the integration of classifiers is effective.

TABLE 1 number of samples from three data sets

Data set	Training set quantity (sheet)	Number of samples (sheet) of test set
			201907_1	817	348
201907_2	1706	731
			201907_3	966	415

TABLE 2 data set 201907_1 data set Experimental results

Table 3 data set 201907_2 data set experimental results

TABLE 4 data set 201907_3 data set Experimental results

TABLE 5 overlapped tobacco leaf data set experimental results

Claims

1. An intelligent overlapping tobacco leaf grading method based on computer vision is characterized by comprising the following specific steps:

2. The computer vision-based intelligent grading method for overlapped tobacco leaves according to claim 1, wherein the specific method for synthesizing the overlapped tobacco leaves by using single flue-cured tobacco leaves is as follows:

step 1.1: denoising the single flue-cured tobacco leaf image;

3. The computer vision-based intelligent grading method for overlapped tobacco leaves according to claim 1, wherein the overlapped tobacco leaves are segmented according to the concave-convex point properties of the overlapped tobacco leaves, and the method comprises the following specific steps:

step 2.3: detecting concave-convex points;

4. The computer vision-based overlapped tobacco leaf intelligent grading method according to claim 3, wherein the specific method for detecting concave-convex points is as follows:

acquiring an outer contour image of the filtering image;

Order to

If A_bIf the contour point is more than or equal to 1.2S, determining the contour point as a strong salient point, and if A is greater than or equal to 1.2S, determining the contour point as a strong salient point_b≤0.And 8S, the contour point is a strong concave point.

5. The computer vision based intelligent grading method of overlapped tobacco leaves according to claim 1, characterized in that the extracted shape texture features comprise: perimeter, area, breakage rate, circularity, major axis length, minor axis length, aspect ratio.

6. The computer vision based intelligent grading method of overlapping tobacco leaves according to claim 1, characterized in that the extracted color features comprise: including 16384-dimensional HSV histogram color features, 11-dimensional HSV three-component mean and standard deviation, B, G color component mean and B, G, R standard deviation.

7. The computer vision-based overlapped tobacco leaf intelligent grading method according to claim 1, wherein the features extracted in step 3 are used for respectively training different classifiers, and the prediction results of the different classifiers are integrated according to the idea of ensemble learning, and the specific steps for obtaining the grading result are as follows:

8. The computer vision-based intelligent grading method for overlapped tobacco leaves according to claim 7, characterized in that the specific steps of performing distinguishable feature selection on color features and depth network features are as follows:

wherein the content of the first and second substances,

is the average of the ith feature over the jth class data set,

the average of the ith feature over the entire data set,

class j kth sample point ith feature;

9. The computer vision based intelligent grading method of overlapped tobacco leaves according to claim 7, characterized in that the formula for maximum and minimum normalization of the three features is as follows: