CN113192090B

CN113192090B - Juicy peach sorting method and device based on machine learning

Info

Publication number: CN113192090B
Application number: CN202110500535.5A
Authority: CN
Inventors: 郑祺文; 邬莉莉; 孙康廷; 何立栋
Original assignee: Nanjing University of Science and Technology
Current assignee: Nanjing University of Science and Technology
Priority date: 2021-05-08
Filing date: 2021-05-08
Publication date: 2022-08-16
Anticipated expiration: 2041-05-08
Also published as: CN113192090A

Abstract

The invention discloses a juicy peach sorting method and device based on machine learning. The control analysis module consists of embedded AI computing equipment and an ARM control board. The basic workflow is as follows: the method is characterized by comprising the following steps of combining a machine learning and statistical method to perform morphological analysis on the juicy peaches, then using a data acquisition module to perform image acquisition on juicy peach samples in a visual field, and classifying the sizes and colors of the juicy peaches; meanwhile, in a near-infrared detection module, detecting the internal condition of the juicy peaches by using near-infrared light, detecting the sugar content of the juicy peaches, and screening the juicy peaches; outputting relevant control parameters to a sorting mechanism of the device, and sorting different types of juicy peaches into different sorting frames by utilizing a push rod; and finally, the information of the shape, the sugar content and the like of the honey peaches is printed on the LCD module.

Description

Juicy peach sorting method and device based on machine learning

Technical Field

The invention belongs to the field of machine vision, near infrared light detection and mechanical control, and particularly relates to a juicy peach sorting method and device based on machine learning.

Background

Juicy peaches are one of the common fruits in the Chinese market. With the continuous improvement of the quality of life, the requirements of the people on the quality of the honey peaches are also continuously improved. Consumers particularly favor honey peaches with large volume and high sugar content in the process of selecting honey peaches. With the increasing demand of honey peaches, the requirements on sorting efficiency and accuracy in sorting work are higher and higher. The following are currently common:

1. pure manual sorting: this sorting method requires a lot of manpower. Due to the increasing labor cost in recent years, only a few small-scale fruit sortings still adopt this mode at present.

2. Mechanical rigid sorting: a machine is used instead of a part of the manual operation. This sorting is primarily based on the size and weight of the fruit. Mainly comprises roller sieve type sorting equipment and roller belt type sorting equipment. The equipment has the advantages of simple mechanical structure, high sorting speed, low manufacturing cost and capability of improving the working efficiency to a certain extent. The defects that in the sorting process, the friction force between fruits and between the fruits and equipment is long, so that the fruit surface tissues are easily extruded and damaged, and the appearance of the fruits is influenced; the fruit process of dropping also easily causes the impact damage, influences the fruit quality. And when the shape difference of the fruits is large, the sorting result has large deviation.

3. Mechanical arm type sorting: the terminal actuating mechanism of arm cooperation vision realizes fixing a position fruit, sorts the operation to fruit through snatching or adsorbing the action. This is a cross technology combining the subjects of vision, control, computer, machinery, etc., but still cannot avoid rubbing or shaking the fruit and dropping the fruit to cause external and internal damage to the fruit.

4. Flexible sorting by machine vision technology: with the continuous development of science and technology, the machine vision technology is gradually mature in the fields of fruit and vegetable quality nondestructive testing, food, processing and packaging, logistics sorting and the like, and compared with other sorting modes, the technology has the advantages of flexible operation, long working time, no fatigue reaction and the like in the aspect of finishing work with high repeatability, has a promoting significance for improving the fruit production efficiency and popularizing the execution of fruit grade standards, and still has a certain promotion space in the aspects of sorting accuracy and efficiency.

Disclosure of Invention

The invention aims to provide a juicy peach sorting method and device based on machine learning, which are used for sorting juicy peaches in a non-contact mode and measuring the sugar content, so that the loss in the sorting process is effectively reduced.

The technical solution for realizing the purpose of the invention is as follows:

a juicy peach sorting method based on machine learning comprises the following steps:

step 1, collecting images and infrared signals of the juicy peaches to obtain the relation between the size of the optical signals and the sugar content of the juicy peaches to obtain the sugar content of the juicy peaches, and removing the juicy peaches which do not meet the requirement of the sugar content;

step 2, distinguishing the sizes and the shapes of the honey peaches: recognizing the image by using the trained target detection model to obtain the position and size data of the prediction frame; respectively drawing the area data of the prediction frames of the juicy peaches with different sizes as Gaussian normal distribution curves by utilizing a Gaussian normal distribution principle, respectively solving the mean value and the variance of the area normal distribution curves of the juicy peaches with different sizes, calculating to obtain the size threshold value for distinguishing the juicy peaches with different sizes, and distinguishing the sizes of the juicy peaches according to the size threshold value;

step 3, distinguishing the color and the shape of the juicy peaches: taking the image of the juicy peach prediction frame region as an ROI (region of interest), counting the ratio of the red region to the area of the prediction frame, further obtaining a corresponding threshold value of the color form of the juicy peach by using a Gaussian normal distribution curve, judging whether the juicy peach is a red peach or not according to the threshold value, and removing the juicy peaches which are not red peaches;

and 4, selecting the juicy peaches with different sizes which accord with the red peaches to corresponding areas by combining the results of the step 2 and the step 3.

Honey peach sorting device based on machine learning, include

The image data acquisition module is used for acquiring image information of the juicy peaches;

the near-infrared detection module is used for collecting the reflection data of the juicy peaches to near-infrared light;

the control analysis module is used for judging the sugar content of the honey peaches according to the data collected by the near-infrared detection module and the image data collection module, distinguishing the size form and the color form of the honey peaches and controlling the operation of the transportation and sorting module;

and the transportation and sorting module is used for transmitting the honey peaches to be distinguished, sorting the distinguished honey peaches according to the distinguishing result, and transmitting the honey peaches which do not meet the sugar content requirement to be offline.

Compared with the prior art, the invention has the following remarkable advantages:

(1) the provided juicy peach size classification method and the juicy peach and non-juicy peach classification method effectively combine a statistical method with machine learning, quantize the related data of the juicy peaches, have clear process, do not depend on model parameters and a large number of classification data sets, and have higher sorting accuracy and wider application range.

(2) For the sorting of the honey peaches, the device realizes the multi-layer sorting including the size, the color and the sugar content of the honey peaches, and has strong comprehensive sorting capacity.

Drawings

FIG. 1 is a schematic view of the apparatus used in the present invention and the connection of the respective parts.

Fig. 2 is a schematic structural diagram of the present invention.

FIG. 3 is a schematic diagram of a model for detecting sugar content of honey peaches by a near-infrared module according to the present invention.

FIG. 4 is a flow chart of the near infrared module for detecting sugar content of honey peaches in the present invention.

FIG. 5 is a schematic representation of the morphological analysis of honey peaches in the present invention.

FIG. 6 is a normal distribution curve of the area of the prediction box of honey peaches in the present invention.

FIG. 7 is a normal distribution curve of the area ratio of the juicy peach red region to the prediction frame in the invention.

FIG. 8 is a flow chart of honey peach classification in the present invention.

Detailed Description

The invention is further described with reference to the following figures and embodiments.

With reference to fig. 1, the combination device for sorting the shapes and sugar contents of the juicy peaches comprises a power supply and voltage stabilization module, a data acquisition module, a near infrared detection module, a control analysis module, a transportation and sorting mechanical structure and an LCD module. The transportation and sorting module consists of a crawler belt and can be used for simultaneously transporting a plurality of juicy peaches. A two-dimensional cradle head carrying an industrial camera is constructed on the front half part of the transportation and sorting crawler, the model of the selected industrial camera is HBVCAM-4K 1928V 11, the conditions of a plurality of juicy peaches at different positions of the track can be detected simultaneously, and the acquired pictures are transmitted to the control analysis module. The embedded AI computing equipment Jetson TX2 in the control analysis module imports image data acquired by a camera, classifies the size form and the color form, the near-infrared detection module selects NIRONE Sensor X, the sugar content of the fruit can be detected through the reflection condition of the fruit to near-infrared light, and finally the data are sent to the control analysis module. The STM32F407 is used by a main control chip of the control analysis module and used for receiving the juicy peach shape information transmitted by the Jetson TX2 and receiving the information such as the fruit sugar content parameter returned by the near-infrared detection module, and controlling the push rod device to sort the juicy peaches and put the juicy peaches into three different baskets to be selected, wherein the baskets are large, medium and small. In addition, STM32F407 outputs the information such as the shape and sugar content of the currently detected honey peaches and the number of the honey peaches which meet the standard after detection to an LCD screen so as to facilitate real-time monitoring.

Referring to fig. 2, honey peaches first need to be manually loaded into the fruit basket 14. The fruit basket 14 is provided with a chute, and can only pass through one juicy peach at a time. The honey peaches arriving on the crawler 2 follow the movement of the crawler 2. The track 2 is provided with a tensioning device, the tensioning of which can be adjusted from the side. The juicy peach reaches the acquisition module 13, and the carried industrial camera (image data acquisition module) and the near infrared light detection module (near infrared detection module) can feed back information such as the shape, the sugar content and the like of the juicy peach. The honey peaches on the crawler 2 are transited to the crawler 1 through the sliding grooves, the honey peaches classified as big peaches can be pushed into the sorting basket 3 through the control push rod 7, the honey peaches classified as middle peaches are pushed into the sorting basket 4 through the control push rod 8, and the honey peaches classified as small peaches are pushed into the sorting basket 5 through the control push rod 9. The rest honey peaches which do not meet the standard are transported forwards along with the crawler 1. Sorting basket 3, 4, 5 all are furnished with elevating platform 6 down, can adjust the position of sorting basket, convenient subsequent processing.

With reference to fig. 3 and 4, a fiber diffuse reflection near infrared light detection technology in "near infrared spectrometry of sugar content and effective acidity of juicy peaches" by liu yand, which should be introduced, is adopted, 820nm wavelength is selected as a characteristic wavelength of sugar content in juicy peaches, firstly, juicy peach samples meeting the size requirement are selected, a relation between the magnitude of an optical signal and the sugar content is obtained by utilizing the near infrared diffuse reflection technology, a mathematical model is established by utilizing a least square method to predict the sugar content of the juicy peaches, then, for the juicy peaches to be detected, the infrared signal is converted into an electric signal by utilizing an photoelectric sensor, the electric signal passing through an amplifying circuit, a filter circuit, a rectifying circuit and an AD conversion circuit is transmitted to a main control chip, and the main control chip calculates the sugar content of the juicy peaches by utilizing the pre-established mathematical model.

Referring to fig. 5, the present invention needs to perform morphological analysis on honey peaches, including size morphological analysis and color morphological analysis, and mainly comprises the following steps:

firstly, model training, wherein the model training method comprises the following steps:

1.1, preparing a juicy peach dataset, wherein the manufacturing method comprises the following steps: the honey peaches move forwards on the conveyor belt, the fixed cameras are used for photographing the honey peaches in the visual field one by one, and 750 honey peach pictures of the same pixel and different sizes and colors are acquired respectively. This example uses the juicy peach dataset published on the net.

And 1.2, performing data enhancement on the acquired image, including horizontal translation, vertical translation, image turning, small-amplitude random rotation, contrast transformation and channel transformation to obtain 9000 data sets including red, large, medium and small juicy peaches.

1.3, preparing a model to be trained, wherein an open-source Yolov4 target detection model is used, the model uses a CSPDarknet53 trunk feature extraction network, uses a Mish activation function, and uses SPP and PANet structures in a feature pyramid.

1.4, putting 9000 enhanced honey peach pictures into a Yolov4 model for training to finally obtain a target detection model for honey peach detection and positioning;

the Map is calculated by using the juicy peach pictures disclosed on the internet as a data set and a target detection model obtained after training, so that the recognition rate close to 100% can be achieved.

Secondly, performing target detection by using the model obtained by training to obtain data such as the position, the size and the like of a prediction frame;

and thirdly, the area size of the prediction frame can indirectly represent the shape size of the juicy peach, the area size of the prediction frame is counted, and then the corresponding threshold value of the juicy peach size is obtained by utilizing the Gaussian normal distribution curve. Supplementary explanation is carried out on the steps, the target detection model is utilized, the juicy peaches can be detected and identified, the position of the prediction frame is obtained, and the yolk 4 target detection model is used, so that the juicy peach positioning accuracy is high, and the data such as the position and the size of the prediction frame have high reliability. The area of the prediction frame is taken as an analysis object, obviously, the size of the area of the prediction frame can indirectly represent the size of the juicy peach shape, and then, the area of the prediction frame of the juicy peaches in different shapes can be obtained by only counting and analyzing the area of the prediction frame of the juicy peaches in different shapes. It should be noted here that the classification principle used in the present invention is not limited to classifying honey peaches into three categories, namely large, medium and small, but also into more categories according to the requirement.

Furthermore, by utilizing the Gaussian normal distribution principle, the area data of the prediction frames of the juicy peaches with different sizes are respectively drawn as Gaussian normal distribution curves, the area of the prediction frames of the juicy peaches with different sizes is mu according to the mathematical expectation, and the variance is sigma ² Is the mean of the area of the prediction box, σ ² Is the variance of the predicted box area.

Further, according to the principle of gaussian normal distribution, the probability of falling on a region is approximately equal to 95.44%, and also according to the basic principle of "small probability event", which generally refers to an event having a probability of occurrence of less than 5%, it is considered that the event is almost impossible to occur in one experiment, and thus μ +2 σ and μ -2 σ can be used as two thresholds for classification. For better illustration, the juicy peach dataset disclosed on the network is used, and in combination with fig. 6, the juicy peaches divided into large, medium and small juicy peaches are taken as an example, the juicy peaches are obtained by using small peaches, medium peaches and large peachesRespectively solving the mean value and the variance of the normal distribution curve of the area of the prediction frame of the small peach according to the normal distribution curve of the area of the prediction frame as follows: mu.s _s ＝1.668×10 ⁴ ，σ _s ＝0.3096×10 ⁴ Respectively solving the mean value and the variance of the area normal distribution curve of the peach prediction frame as mu by the area normal distribution curve of the peach prediction frame _m ＝1.991×10 ⁴ ，σ _m ＝0.2411×10 ⁴ Respectively solving the mean value and the variance of the area normal distribution curve of the large peach prediction frame as mu according to the area normal distribution curve of the large peach prediction frame _b ＝2.554×10 ⁴ ，σ _b ＝0.3189×10 ⁴ Then μ _m -2σ _m ＝1.51×10 ⁴ ，μ _m +2σ _m ＝2.47×10 ⁴ Binding of mu _s -2σ _s ＝1.05×10 ⁴ ,μ _s +2σ _s ＝2.29×10 ⁴ And mu _b -2σ _b ＝1.92×10 ⁴ ，μ _b +2σ _b ＝3.19×10 ⁴ Making an adjustment to

And

i.e. two thresholds of 1.9 x 10 respectively ⁴ And 2.2X 10 ⁴ Then the predicted frame area can be made smaller than α ₁ The honey peaches are small peaches, and the prediction frame area is larger than alpha ₂ The juicy peaches are large peaches, and the area of a prediction frame is [ alpha ] ₁ ,α ₂ ]The inner one is named as middle peach.

Fourthly, sampling the red area of the juicy peach to obtain a proper HSV range, and setting a mask, wherein the method in the step comprises the following steps:

4.1, selecting 100 pictures from the red juicy peach dataset in the step 1.1, changing the channels of the pictures, and changing the BGR channel into the HSV channel.

And 4.2, compiling a mouse click event, clicking a mouse, randomly sampling HSV in a red area in 100 juicy peach pictures, recording the value of the HSV, taking the lowest value and the highest value, determining the maximum range of the HSV value, and taking the value range of the HSV as a mask.

The maximum range of the HSV value is determined because the difference between the HSV of the juicy peach red region and the HSV of other regions is large, the value range is large, so that the influence is not large, and the subsequent determination of the juicy peach red region is convenient. Examples using the juicy peach dataset published on the web, the HSV values obtained ranged from [0,70,67] to [13,164,220 ];

and fifthly, the ratio of the red area to the area of the prediction frame can be used for indirectly judging the color form of the honey peach, the ratio of the red area to the area of the prediction frame is counted, then the corresponding threshold value of the color form of the honey peach is obtained by utilizing the Gaussian normal distribution curve, and the method for counting the ratio of the red area to the area of the prediction frame is as follows:

5.1, regarding whether the peach is judged to be the red peach, only the condition of the juicy peach in the prediction frame is involved, and therefore, the image in the prediction frame is taken as the region of interest, namely the ROI.

And 5.2, combining the mask obtained in the step 4.2, obtaining a moment by using a CV algorithm, drawing a contour, calculating the area of the contour by using the moment, and taking the maximum value of the area of the contour, namely the area of the juicy peach red area.

5.3, determining the color and form threshold of the juicy peaches:

the statistical method of the step is the same as that in the third step, only the analysis object is changed into the ratio of the area of the red area in the prediction frame to the area of the prediction frame, the average value and the variance of the normal distribution curve of the ratio of the area of the red peach to the area of the prediction frame are respectively mu by using the juicy peach data set disclosed on the network and combining with the figure 7 _r ＝0.3408，σ _r 0.1298, by the principle of Gaussian normal distribution, the probability of a variable falling on (μ - σ, + ∞) is 0.8413, which does not fit the basic principle of "small probability events", but μ is chosen considering that part of the red region of a juicy peach is not counted in for placement reasons _r -σ _r And (4) taking 0.211 as a threshold value of the color form, namely dividing the juicy peach red area with the ratio of the area of the prediction frame to the juicy peach red area larger than 0.211 into red peaches, and dividing the juicy peach red area into non-red peaches.

The reason why the ratio of the red region to the area of the prediction frame is selected as the analysis object rather than directly using the area of the red region as the analysis object is that the areas of the prediction frames of honey peaches in different forms are different, and even the areas of the prediction frames of some honey peaches are different greatly, the areas of the prediction frames are small, and the areas of the red regions are relatively small. In the juicy peach dataset disclosed on the internet, the red peaches are usually big peaches, but a plurality of red peaches exist in the middle peaches and even small peaches, and the red peaches in the middle peaches and the small peaches can be excluded from the red peaches by selecting the red region as the threshold value obtained by the statistical analysis object, so that the ratio of the area of the red region to the area of the prediction frame is more reasonable to be used as the statistical analysis object.

With reference to fig. 8, the main steps of classifying honey peaches are:

a. the fixed industrial camera collects the juicy peach images on the conveyor belt one by one, and the juicy peach images have the same pixels as those of the juicy peach images in the data set.

b. And identifying the image by using the trained Yolov4 target detection model, accurately positioning the juicy peach, and outputting the relevant information of the prediction frame.

c. Calculating the area of prediction frame of honey peach, and 1.9 × 10 threshold value in the third step ⁴ And 2.2X 10 ⁴ And comparing the sizes of the juicy peaches with each other, judging the section to which the area of the prediction frame belongs, and classifying the sizes and the shapes of the juicy peaches according to the determined classification standard in the step three.

d. And (5) taking the image of the juicy peach predicted frame region as an ROI region, and calculating the area of a red region in the ROI region according to the method in the step 5.2.

e. And (4) calculating the ratio of the area of the red region to the area of the prediction frame, comparing the ratio with the threshold value of 0.211 in the step five, judging the region to which the red region belongs, and classifying the color and the shape of the juicy peaches according to the classification standard determined in the step five, namely judging whether the juicy peaches are red peaches or non-red peaches.

Claims

1. A juicy peach sorting method based on machine learning is characterized by comprising the following steps:

the sizes and shapes of the honey peaches are divided into three types, namely large, medium and small, and two thresholds are set:

and

wherein mu _m ，σ _m Respectively mean value and variance, mu, of the area normal distribution curve of the prediction frame of the peach _s ，σ _s Respectively mean and variance, mu, of the area normal distribution curve of the prediction frame of the peach _b ，σ _b Respectively obtaining the mean value and the variance of the area normal distribution curve of the prediction frame of the large peach;

setting the area of the prediction frame less than the threshold value alpha ₁ The juicy peaches are determined to be small peaches, and the area of the prediction frame is larger than a second threshold value alpha ₂ The juicy peaches are large peaches, and the area of a prediction frame is [ alpha ] ₁ ,α ₂ ]The inner is named as middle peach;

the threshold value of the color morphology is mu _r -σ _r

Wherein mu _r ，σ _r Are respectively red peachThe mean and variance of the normal distribution curve of the area ratio of the red area relative to the prediction frame;

2. The juicy peach sorting method according to claim 1, wherein the determining of the juicy peach color morphology threshold specifically comprises the following steps:

(a) taking the image in the prediction frame as a region of interest;

(b) sampling the red area of the juicy peach, setting a mask, obtaining a moment by using a CV algorithm, drawing a contour, calculating the area of the contour by using the moment, and taking the maximum value of the area of the contour, namely the area of the juicy peach red area;

(c) determining a color and form threshold value of the juicy peaches: counting the ratio of the red area to the area of the prediction frame, obtaining the mean value and the variance of the ratio of the red area to the area of the prediction frame by utilizing a Gaussian normal distribution curve, and selecting mu according to the Gaussian normal distribution principle _r -σ _r As a threshold for color morphology.

3. Honey peach sorting device based on machine learning, which is characterized by comprising

the transportation and sorting module is used for transporting the juicy peaches to be distinguished, sorting the differentiated juicy peaches according to the distinguishing result, and transporting the juicy peaches which do not meet the sugar content requirement to a lower line;

the control analysis module comprises a sugar content distinguishing unit, a size form distinguishing unit, a color form distinguishing unit and a control unit;

the sugar amount discrimination unit: according to the relation between the magnitude of the optical signal and the sugar content, the sugar content of the juicy peaches is obtained, and the juicy peaches which do not meet the requirement of the sugar content are removed;

the size form distinguishing unit: recognizing the image by using the trained target detection model to obtain the position and size data of the prediction frame; respectively drawing the area data of the prediction frames of the juicy peaches with different sizes as Gaussian normal distribution curves by utilizing a Gaussian normal distribution principle, respectively solving the mean value and the variance of the area normal distribution curves of the juicy peaches with different sizes, calculating to obtain the size threshold value for distinguishing the juicy peaches with different sizes, and distinguishing the sizes of the juicy peaches according to the size threshold value;

the color form distinguishing unit: taking the image of the juicy peach prediction frame region as an ROI (region of interest), counting the ratio of the red region to the area of the prediction frame, further obtaining a corresponding threshold value of the color form of the juicy peach by using a Gaussian normal distribution curve, judging whether the juicy peach is a red peach or not according to the threshold value, and removing the juicy peaches which are not red peaches;

the control unit is used for controlling the operation of the transportation and sorting module.

4. A sorting device according to claim 3, characterised in that the transport and sorting module comprises:

the device comprises a to-be-selected basket arranged at the end part of a conveying belt and used for storing honey peaches to be distinguished and transmitting the honey peaches which do not meet the requirement of sugar content to be offline;

the conveying belt is used for conveying the juicy peaches to be distinguished;

the push rods are arranged at the side ends of the conveying belts and used for pushing the differentiated honey peaches;

the sorting baskets are arranged at the side end of the conveying belt and used for storing the honey peaches after being distinguished.