CN110548699A - Automatic pineapple grading and sorting method and device based on binocular vision and multispectral detection technology - Google Patents
Automatic pineapple grading and sorting method and device based on binocular vision and multispectral detection technology Download PDFInfo
- Publication number
- CN110548699A CN110548699A CN201910938730.9A CN201910938730A CN110548699A CN 110548699 A CN110548699 A CN 110548699A CN 201910938730 A CN201910938730 A CN 201910938730A CN 110548699 A CN110548699 A CN 110548699A
- Authority
- CN
- China
- Prior art keywords
- pineapple
- pineapples
- binocular
- grading
- point cloud
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 235000007119 Ananas comosus Nutrition 0.000 title claims abstract description 202
- 238000001514 detection method Methods 0.000 title claims abstract description 28
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000005516 engineering process Methods 0.000 title claims abstract description 13
- 244000099147 Ananas comosus Species 0.000 title 1
- 241000234671 Ananas Species 0.000 claims abstract description 201
- 238000001228 spectrum Methods 0.000 claims abstract description 15
- 230000003595 spectral effect Effects 0.000 claims abstract description 9
- 238000004364 calculation method Methods 0.000 claims abstract description 4
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 claims abstract description 4
- 150000001720 carbohydrates Chemical class 0.000 claims abstract description 3
- 239000011664 nicotinic acid Substances 0.000 claims description 42
- 230000007246 mechanism Effects 0.000 claims description 18
- 239000011159 matrix material Substances 0.000 claims description 16
- 229910052736 halogen Inorganic materials 0.000 claims description 15
- 150000002367 halogens Chemical class 0.000 claims description 15
- 238000002310 reflectometry Methods 0.000 claims description 15
- 230000001681 protective effect Effects 0.000 claims description 8
- 238000010586 diagram Methods 0.000 claims description 7
- 230000005540 biological transmission Effects 0.000 claims description 6
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 238000012937 correction Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 230000007306 turnover Effects 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 claims description 3
- 238000013178 mathematical model Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000013528 artificial neural network Methods 0.000 claims description 2
- 230000003287 optical effect Effects 0.000 claims description 2
- 230000011218 segmentation Effects 0.000 claims description 2
- 238000012549 training Methods 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 230000009471 action Effects 0.000 description 7
- 235000013399 edible fruits Nutrition 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 230000033001 locomotion Effects 0.000 description 2
- 230000001737 promoting effect Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 235000012055 fruits and vegetables Nutrition 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000011065 in-situ storage Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000007789 sealing Methods 0.000 description 1
- 210000004872 soft tissue Anatomy 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
- B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
- B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
- B07C5/34—Sorting according to other particular properties
- B07C5/342—Sorting according to other particular properties according to optical properties, e.g. colour
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B07—SEPARATING SOLIDS FROM SOLIDS; SORTING
- B07C—POSTAL SORTING; SORTING INDIVIDUAL ARTICLES, OR BULK MATERIAL FIT TO BE SORTED PIECE-MEAL, e.g. BY PICKING
- B07C5/00—Sorting according to a characteristic or feature of the articles or material being sorted, e.g. by control effected by devices which detect or measure such characteristic or feature; Sorting by manually actuated devices, e.g. switches
- B07C5/34—Sorting according to other particular properties
- B07C5/342—Sorting according to other particular properties according to optical properties, e.g. colour
- B07C5/3422—Sorting according to other particular properties according to optical properties, e.g. colour using video scanning devices, e.g. TV-cameras
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Investigating Or Analysing Materials By Optical Means (AREA)
- Sorting Of Articles (AREA)
Abstract
The invention discloses a method and a device for automatically grading and sorting pineapples based on binocular vision and multispectral detection technologies, wherein a binocular vision system is adopted to detect the size and the color maturity of the pineapples, and the external quality grading of the pineapples is carried out; then, whether the interior of the pineapple is damaged or not is detected by adopting multiple spectrums, the interior quality of the pineapple is detected and judged according to the spectral curve characteristics of the interior water content and the carbohydrate content of the normal pineapple and the black-heart pineapple, and the interior quality of the pineapple is graded; and finally, automatically classifying and sorting the pineapples according to the detection result. The method can accurately and efficiently automatically sort the pineapples in a grading manner, has high algorithm speed and less calculation amount, is suitable for host equipment with low configuration, has low cost and is very suitable for field operation; the device has low cost, simple structure, small part volume and convenient operation.
Description
Technical Field
the invention relates to the technical field of fruit and vegetable sorting, in particular to a method and a device for automatically sorting pineapples in a grading manner based on binocular vision and a multispectral detection technology.
Background
pineapple is one of four big fruits in the south of China, and the demand and planting amount of the pineapple are increasing continuously. The collection and sorting mechanization of the pineapples is vigorously developed, and the method has important significance for promoting the adjustment of the industrial structure of the pineapples, promoting the improvement of the product value of the pineapples, improving the income of fruit growers and the like. The pineapples are sorted and collected in a grading manner before being processed or sold, and most of the existing similar sorting methods are that the pineapples are carried to a fruit field after being picked manually, and then are sorted manually and simply according to the sizes and colors of the pineapples. The screening according to the senses of people has no clear indexes, the accuracy of classification is difficult to ensure, the black-heart pineapples are difficult to detect, the labor intensity is high, the labor is wasted, and the working efficiency is low. Therefore, the invention provides an automatic grading and sorting method and device for pineapples.
Disclosure of Invention
the invention aims to overcome the defects in the prior art and provides the automatic pineapple grading and sorting method and device based on the binocular vision and multispectral detection technology, which have high grading and sorting accuracy and are convenient to use.
The purpose of the invention is realized by the following technical scheme:
a method for automatically grading and sorting pineapples based on binocular vision and multispectral detection technology comprises the steps of firstly, detecting the size and the color maturity of pineapples by using a binocular vision system, and grading the external quality of the pineapples; then, whether the interior of the pineapple is damaged or not is detected by adopting multiple spectrums, the interior quality of the pineapple is detected and judged according to the spectral curve characteristics of the interior water content and the carbohydrate content of the normal pineapple and the black-heart pineapple, and the interior quality of the pineapple is graded; and finally, automatically classifying and sorting the pineapples according to the detection result.
The detection of the size of the pineapple comprises the following steps:
(1) Binocular calibration and correction: performing three-dimensional calibration on the binocular cameras 10, namely performing monocular calibration on the two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, simultaneously carrying out binocular stereo vision calibration on the two cameras to obtain a reprojection matrix for binocular correction and a conversion relation between a pixel distance and a real physical distance; performing binocular correction on the pineapple image shot by the binocular camera to obtain a binocular corrected image;
(2) Segmenting the pineapple and the background: firstly, collecting more than 100 pineapple images for neural network training of MaskR-CNN algorithm; then, obtaining a semantic segmentation model with curve convergence by using a Mask R-CNN algorithm, and segmenting pineapples from the binocular corrected images respectively to obtain segmented left and right pineapple images;
(3) generating a disparity map by adopting an SGBM stereo matching algorithm, and obtaining three-dimensional point cloud on the surface of the pineapple by utilizing the segmented left and right images of the pineapple, wherein the origin of the point cloud is defined at the optical center position of a left camera;
(4) Processing the three-dimensional point cloud on the surface of the pineapple by adopting a directional bounding box algorithm to obtain the size of the pineapple;
(5) And setting a size threshold value a (the setting range of a is 60-80 mm), and judging that the pineapple is unqualified when the diameter of the pineapple is smaller than a.
The method for calculating the three-dimensional point cloud on the surface of the pineapple by adopting the directional bounding box algorithm comprises the following steps:
(1) Firstly, solving a covariance matrix of the three-dimensional point cloud, wherein the matrix can reflect the linear correlation degree of each point in the point cloud, and a covariance calculation formula of each point in the three-dimensional point cloud is as follows:
cov(X,Y,Z)＝E(X-EX)(Y-EY)(Z-EZ) (1)
EX, EY and EZ are mathematical expectations of random variables X, Y, Z of each point in the three-dimensional point cloud in the x, y and z dimensions respectively, and then a three-dimensional covariance matrix C of the three-dimensional point cloud is obtained:
(2) carrying out diagonal transformation on a three-dimensional covariance matrix of the three-dimensional point cloud to obtain a characteristic value and a characteristic vector of the three-dimensional covariance matrix; representing the eigenvector with the largest eigenvalue as the longest side direction of the directional bounding box, and searching the pineapple point cloud length boundary point according to the eigenvector direction to obtain the long side of the bounding box;
(3) projecting the characteristic vector with the maximum characteristic value onto a base plane parallel to an imaging plane of the camera, making a normal vector perpendicular to the characteristic vector on the base plane, and searching a width boundary point of the pineapple point cloud along the direction of the normal vector to obtain a wide edge of the bounding box, wherein the wide edge represents the diameter of the pineapple;
(4) Performing cross multiplication on the eigenvector with the maximum eigenvalue in the step (2) and the normal vector in the step (3) to obtain a normal vector of a base plane, searching a height boundary point of the pineapple along the normal vector direction of the base plane to obtain the height of the bounding box, wherein the height represents the radius of the pineapple;
(5) and measuring the long edge, the wide edge and the height of the bounding box, and converting to obtain the size of the pineapple according to pixel statistics and the conversion relation between the pixel distance and the real physical distance.
The detection of the maturity of the pineapple comprises the following steps:
(1) Finding pineapples in the binocular corrected image by using the Mask R-CNN algorithm, segmenting the pineapple image, counting the area S 1 of one side image of the pineapple, wherein the unit is a pixel, turning the pineapple over, and counting the area S 2 of the other side image of the same pineapple;
(2) setting a threshold value T (the setting range of T is 120-150), and segmenting the G component image by adopting a formula (3) to obtain an output image, wherein G src (x, y) represents the G component image of the pineapple, and dst (x, y) represents the output image;
(3) counting an area C 1 with a gray scale value of 255 in the output image dst (x, y) and the unit is pixel, C 1 represents the green part area of the pineapple skin on the pineapple image, and similarly, the green part area on the other side of the pineapple image is marked as C 2, calculating the occupation ratio i of the green part area of the pineapple skin:
(4) Setting a maturity threshold m (the setting range of m is 0.5-0.75), and judging that the pineapple is immature when i is larger than m.
Multispectral detection of the interior of the pineapple comprises the following steps:
(1) establishing the spectral region of normal mature pineapples: putting the normal ripe pineapples into a spectrum detector, creating a mathematical model according to the reflectivity of the normal ripe pineapples in light of 600-900 nm wave band, fitting an average spectrum curve, and drawing out a spectrum area of the normal ripe pineapples;
(2) And (3) placing the pineapple to be detected in a multispectral detector, detecting the reflectivity of the pineapple at a 600-900 nm waveband, creating a scatter diagram according to the reflectivity, and when more than 85% of points in the scatter diagram fall within the spectral region of the normal mature pineapple, the interior of the pineapple to be detected is not damaged and is the pineapple with normal quality.
the method for establishing the spectrum region of the normal mature pineapple comprises the following steps:
(1) more than 100 normal ripe pineapples are picked out by a needle tube sampling mode and put into a spectrum detector, and a halogen lamp is turned on to irradiate light on the pineapples and can generate light with a wide waveband range;
(2) adjusting a spectrometer to enable the imaging spectral range of the spectrometer to be 600-900 nm, and collecting the reflectivity of light of each wave band after the light irradiates on the normal mature pineapple; collecting and recording the reflectivity of the normal mature pineapples to the light of the wave band every 5-10 nm by taking the light of the wave band of 600nm as a starting point until the reflectivity of the light of 900nm is sampled;
(3) Sampling more than 100 normal ripe pineapples for more than 20 times respectively, and averaging the reflectivity collected in the same wave band; then drawing a scatter diagram according to the average value of the reflectivity of each wave band, wherein the horizontal axis is the wave band of light, the unit nm is, and the vertical axis is the average value of the reflectivity;
(4) establishing a mathematical model for the scatter diagram in the step (3) by using a polynomial regression method, and fitting to obtain an average spectral curve of the normal mature pineapples at the wave band of 600-900 nm; and respectively translating the average spectrum curve to positive and negative directions of a longitudinal axis by a certain distance (the range is +/-10% to +/-15%) by taking the average spectrum curve as a reference, and marking out a spectrum region of the normal ripe pineapple.
The polynomial regression method in the step (4) is based on the principle that a power function can approximate any function:
N is the order of the polynomial and is generally an odd number close to but not greater than the number of sampling points; k is the coefficient of the polynomial. The polynomial regression is a planar scatter regression algorithm with short algorithm running time. The average value of the reflectivity is used as a sampling point of polynomial regression, so that the data volume can be reduced, the operation time of the algorithm can be reduced, and the operation efficiency can be improved.
the principle of multispectral detection of the interior of the pineapple is as follows: since part of the wavelength band of light can pass through the pineapple and the spectrometer can collect the intensity of this part of the light, the lost light intensity can be considered as being reflected by the pineapple. Compared with normal mature pineapples, the internally damaged pineapples have more water content and low sugar content in cells, so that the internally damaged pineapples are easier to reflect light in a specific wave band (600nm-900nm), namely, the spectrum detection result of the internally damaged pineapples is reflected as high reflectivity and low light transmittance.
a binocular vision and multispectral detection technology pineapple automatic grading sorting device adopts the above pineapple automatic grading sorting method, and comprises a bilateral bent plate chain conveyor belt 2, a U-shaped groove 3, a diversion groove 6, a push rod 8, a binocular camera 10 and a multispectral detector 13; a conveying belt shell 1 is provided with a double-side bent plate chain conveying belt 2, and a U-shaped groove 3 is positioned above the double-side bent plate chain conveying belt 2; the two shunting grooves 6 are respectively positioned at two sides of the transmission belt shell 1; the two push rod fixing supports 7 are respectively positioned at two sides of the conveyor belt shell 1 and correspond to the two shunting grooves 6, and the push rod 8 is positioned above the push rod fixing supports 7; the camera support 9 is fixed above the transmission belt shell 1, and the binocular camera 10 is installed on the camera support 9; the closed box 12 is arranged between the two shunting grooves 6, and a multispectral detector 13 and a halogen lamp are arranged in the closed box 12; the binocular camera 10, the first splitter box and the first push rod, the multispectral detector 13, the second splitter box and the second push rod are sequentially arranged in the conveying direction of the double-side bent plate chain conveyor belt 2.
the two ends of the arc-shaped elastic connecting plate 4 are installed above the two adjacent U-shaped grooves 3 through the two hinges 5, one page of each hinge 5 is connected with the arc-shaped elastic connecting plate 4, the other page of each hinge 5 is fixed on the U-shaped groove 3, and the U-shaped grooves 3 are connected in pairs through the arc-shaped elastic connecting plates 4 and the hinges 5. When the U-shaped groove 3 is reversed on the double-side bent plate chain conveyor belt 2, one page of the hinge 5 fixed on the U-shaped groove 3 is fixed relative to the U-shaped groove 3, and the other page of the hinge rotates; meanwhile, when in reversing, the U-shaped groove 3 positioned in front drives the next U-shaped groove 3 to complete reversing by drawing the arc elastic connecting plate 4; after reversing, the hinge returns to the original position under the action of gravity. Therefore, under the elastic action of the arc-shaped elastic connecting plate 4, the distance between two adjacent U-shaped grooves 3 can be kept consistent after horizontal movement and reversing.
the bionic shell protective cover 11 is positioned right above the horizontal rod of the camera support 9, the cross section of the bionic shell protective cover 11 is approximately elliptical, and the edge of the bionic shell protective cover is contracted inwards. The principle of the bionic shell protective cover 11 is to simulate the action of the clam shell to protect the internal soft tissue, and the structure can prevent the binocular camera 10 from being directly irradiated by sunlight and rainfall with a certain angle, and prevent the binocular camera 10 from being heated and damaged and reduce the precision.
The bionic ant leg overturning mechanisms 16 are arranged on the U-shaped grooves 3, and more than 1 bionic ant leg overturning mechanism 16 is arranged on two sides of each U-shaped groove at least.
The bionic ant leg overturning mechanism 16 is provided with four limb joints which are connected in sequence through four joints; the first joint is a base joint, a driving motor of the first joint is arranged on the U-shaped groove 3, and an output shaft of the driving motor is connected with the base of the first limb section of the bionic ant leg and is used for driving the whole bionic ant leg to rotate; the driving motors of the second joint and the third joint are respectively arranged at the tail end of the first limb section and the tail end of the second limb section of the bionic ant leg and are respectively used for driving the second limb section and the third limb section to rotate, so that the bionic ant leg is more flexible and has better folding performance, and the motion space layout of the bionic ant leg is optimized; the fourth joint is driven by air pressure, and because the air pressure driven joint is high in loading and unloading speed, when the pressure value of the air pressure drive of the fourth joint is larger than a certain range (the thrust reaches more than 1.5 kilograms), the air pressure can be quickly unloaded to prevent the pineapple from being punctured by the fourth limb joint. When the pineapple is turned, when the bionic ant legs reach the excircle tangent of the cross section of the pineapple, the four joints can rapidly drive the four limbs of the bionic ant legs to fold, stretch or rotate, and the bionic ant legs rotate to the initial positions to continue turning the pineapple. The tail end of the bionic ant leg overturning mechanism 16 is provided with a plurality of fine seta for increasing the friction force when contacting with the pineapple.
The bionic ant leg overturning mechanism 16 has the following action principle: when the rigid body is subjected to two equivalent reverse acting forces, the action point connecting line of the two acting forces passes through the mass center of the rigid body, and the directions of the two acting forces are vertical to the connecting line, the rigid body rotates around the mass center. When the bionic ant leg overturning mechanism 16 moves, the bionic ant legs positioned on the two sides of the U-shaped groove 3 respectively apply equivalent reverse thrust to the skin of the pineapple, the acting point connecting line of the two thrusts on the skin of the pineapple penetrates through the center of the cross section of the pineapple, the pineapple can be overturned by 180 degrees around the central axis of the pineapple in the U-shaped groove, and the maturity condition of the other side of the pineapple can be detected by the binocular camera. When the stroke of the bionic ant leg reaches the maximum and the pineapple does not turn over by 180 degrees, all joints of the bionic ant leg move jointly, so that the bionic ant leg is folded and contracted to return to the initial position and is ready to execute the next turning action. The bionic ant leg overturning mechanism 16 is simple in structure, small in size and rapid in action, can meet the requirement of accurately overturning pineapples in situ in a U-shaped groove with narrow space, and cannot influence the shooting of a binocular camera.
a halogen lamp angle adjusting device 14 is arranged at the lower part inside the closed box 12, and a multispectral detector 13 is arranged at the upper part inside the closed box; the halogen lamp 15 is mounted at the front end of the halogen lamp angle adjusting device 14, and faces the multispectral detector 13 above. Before use, the adjustable halogen lamp angle adjusting device 14 enables light rays of the halogen lamp 15 to illuminate the pineapple more intensively, and light leakage is reduced.
Compared with the prior art, the invention has the following advantages and effects:
(1) The method can accurately and efficiently automatically sort the pineapples in a grading manner, has high algorithm speed and less calculation amount, is suitable for host equipment with low configuration, has low cost and is very suitable for field operation.
(2) The device has the advantages of low cost, simple structure, small part volume and convenient operation, can sort the external characteristics of the size and maturity of the pineapples and detect whether the pineapples are black or not, and saves the labor cost of sorting workers.
(3) The pineapple picking manipulator can be used together with a picking manipulator and a transport vehicle, and is an important ring for realizing efficient integration of field operations from pineapple picking to fruit transportation and the like.
Drawings
Fig. 1 is a front view of the automatic pineapple grading and sorting device of the invention.
Fig. 2 is a left side view of the automatic pineapple grading and sorting device of the invention.
fig. 3 is a top view of the automatic pineapple grading and sorting device of the present invention.
Fig. 4 is an enlarged view of the connection of the U-shaped slot to the double-sided dog chain conveyor.
Fig. 5 is an enlarged view of the multispectral detection portion.
Fig. 6 is a schematic diagram of the bionic ant leg overturning mechanism.
1. a conveyor belt housing; 2. a double-sided dog chain conveyor; 3. a U-shaped groove; 4. an arc-shaped elastic connecting plate; 5. a hinge; 6. a shunt slot; 7. a push rod fixing bracket; 8. a push rod; 9. a camera support; 10. a binocular camera; 11. a shell bionic protective cover; 12. sealing the box; 13. a multispectral detector; 14. a halogen lamp angle adjusting device; 15. a halogen lamp; 16. bionic ant leg overturning mechanism.
Detailed Description
the present invention will be described in further detail with reference to examples, but the embodiments of the present invention are not limited thereto.
Example 1
As shown in fig. 1, 2 and 3, the automatic pineapple grading and sorting device adopting the binocular vision and multispectral detection technology comprises a double-side bent plate chain conveyor belt 2, a U-shaped groove 3, a diversion groove 6, a push rod 8, a binocular camera 10 and a multispectral detector 13; a conveying belt shell 1 is provided with a double-side bent plate chain conveying belt 2, and a U-shaped groove 3 is positioned above the double-side bent plate chain conveying belt 2; the two shunting grooves 6 are respectively positioned at two sides of the transmission belt shell 1; the two push rod fixing supports 7 are respectively positioned at two sides of the conveyor belt shell 1 and correspond to the two shunting grooves 6, and the push rod 8 is positioned above the push rod fixing supports 7; the camera support 9 is fixed above the transmission belt shell 1, the binocular camera 10 is installed on the camera support 9, and the shell bionic protection cover 11 is located right above a horizontal rod of the camera support 9; the closed box 12 is arranged between the two shunting grooves 6, and a multispectral detector 13 and a halogen lamp are arranged in the closed box 12; the binocular camera 10, the first splitter box and the first push rod, the multispectral detector 13, the second splitter box and the second push rod are sequentially arranged in the conveying direction of the double-side bent plate chain conveyor belt 2. As shown in fig. 4, two ends of the arc-shaped elastic connecting plate 4 are mounted above two adjacent U-shaped grooves 3 through two hinges 5, one page of each hinge 5 is connected with the arc-shaped elastic connecting plate 4, the other page of each hinge 5 is fixed on the U-shaped groove 3, and the two U-shaped grooves 3 are connected through the arc-shaped elastic connecting plates 4 and the hinges 5. As shown in fig. 6, the bionic ant leg overturning mechanism 16 is installed on the U-shaped groove 3, and at least more than 1 bionic ant leg overturning mechanism 16 is installed on both sides of each U-shaped groove. Bionic ant leg tilting mechanism 16 has four limbs, and four limbs loop through four joints and connect, and bionic ant leg tilting mechanism 16's end is provided with a lot of tiny setae for frictional force when increasing with the pineapple contact. As shown in fig. 5, a halogen lamp angle adjusting device 14 is mounted on the lower portion of the sealed case 12, and a halogen lamp 15 is mounted on the front end of the halogen lamp angle adjusting device 14 and faces the multispectral detector 13 on the upper side.
When in use, the method comprises the following steps:
(1) placing the pineapple: the pineapples are placed in a first empty U-shaped groove 3 at the right end of the double-side bent plate chain conveyor belt. Each U-shaped groove 3 can only contain one pineapple, so that a fixed interval is kept in the conveying process of the pineapples, the binocular camera 10 is enabled to shoot only one pineapple at a time, and the push rod 8 is enabled to push only one pineapple at a time;
(2) conveying the pineapple: the pineapples are placed on the U-shaped groove 3, and the U-shaped groove 3 is driven by the double-side bent plate chain conveyor belt 2 to horizontally move on the automatic pineapple grading and sorting device;
(3) Binocular vision detection: when the U-shaped groove 3 loaded with the pineapples reaches the position right below the binocular camera 10, a digital image of one side of the pineapples is shot through the binocular camera 10; then, the bionic ant leg overturning mechanism 16 is started, the pineapple is rotated in place around the axis by 180 degrees, and the binocular camera 10 shoots a digital image of the other side of the pineapple; sending an image obtained by shooting by a binocular camera to a background processing system, calculating the size of the pineapple by adopting a directional bounding box algorithm, and detecting and calculating the color maturity of the pineapple;
(4) grading the external quality of the pineapple: if the size or color maturity of the pineapples does not meet the requirements, an instruction is sent to the first push rod 8, the pineapples which do not meet the requirements are pushed out to the first diversion trench, and external quality grading of the pineapples is achieved;
(5) And (3) multispectral detection classification: the pineapples with qualified external quality are continuously horizontally transmitted on the automatic pineapple grading and sorting device, enter the closed boxes 12 one by one, turn on the halogen lamp 15 and the adjusting spectrometer 13, detect the reflectivity of the pineapples in the wave band of 600-900 nm, and detect whether the pineapples are damaged or not;
(6) grading the external quality of the pineapple: if the damage in the pineapples is detected, an instruction is sent to the second push rod 8, the pineapples which do not meet the requirement are pushed out to a second diversion groove, and the grading of the internal quality of the pineapples is realized; finally obtaining the pineapple with qualified external and internal quality.
The above description is only an example of the present invention, but the present invention is not limited to the above example, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention and are equivalent to each other are included in the protection scope of the present invention.
Claims (10)
1. A binocular vision and multispectral detection technology-based automatic grading and sorting method for pineapples is characterized by comprising the following steps: firstly, detecting the size and the color maturity of the pineapples by adopting a binocular vision system, and grading the external quality of the pineapples; then, whether the interior of the pineapple is damaged or not is detected by adopting multiple spectrums, the interior quality of the pineapple is detected and judged according to the spectral curve characteristics of the interior water content and the carbohydrate content of the normal pineapple and the black-heart pineapple, and the interior quality of the pineapple is graded; and finally, automatically classifying and sorting the pineapples according to the detection result.
2. the method for automatically grading and sorting pineapples based on binocular vision and multispectral detection technology according to claim 1, wherein the method comprises the following steps: the detection of the size of the pineapple comprises the following steps:
(1) binocular calibration and correction: performing three-dimensional calibration on binocular cameras, namely performing monocular calibration on the two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, simultaneously carrying out binocular stereo vision calibration on the two cameras to obtain a reprojection matrix for binocular correction and a conversion relation between a pixel distance and a real physical distance; performing binocular correction on the pineapple image shot by the binocular camera to obtain a binocular corrected image;
(2) segmenting the pineapple and the background: firstly, collecting more than 100 pineapple images for neural network training of Mask R-CNN algorithm; then, obtaining a semantic segmentation model with curve convergence by using a Mask R-CNN algorithm, and segmenting pineapples from the binocular corrected images respectively to obtain segmented left and right pineapple images;
(3) generating a disparity map by adopting an SGBM stereo matching algorithm, and obtaining three-dimensional point cloud on the surface of the pineapple by utilizing the segmented left and right images of the pineapple, wherein the origin of the point cloud is defined at the optical center position of a left camera;
(4) Processing the three-dimensional point cloud on the surface of the pineapple by adopting a directional bounding box algorithm to obtain the size of the pineapple;
(5) and setting a size threshold value a, wherein the setting range of a is 60-80 mm, and when the diameter of the pineapple is smaller than a, judging that the pineapple is unqualified in size.
3. the automatic grading and sorting method for pineapples based on binocular vision and multispectral detection technology according to claim 2, wherein the method comprises the following steps: the method for calculating the three-dimensional point cloud on the surface of the pineapple by adopting the directional bounding box algorithm comprises the following steps:
(1) Firstly, solving a covariance matrix of the three-dimensional point cloud, wherein the matrix can reflect the linear correlation degree of each point in the point cloud, and a covariance calculation formula of each point in the three-dimensional point cloud is as follows:
cov(X,Y,Z)＝E(X-EX)(Y-EY)(Z-EZ) (1)
EX, EY and EZ are mathematical expectations of random variables X, Y, Z of each point in the three-dimensional point cloud in the x, y and z dimensions respectively, and then a three-dimensional covariance matrix C of the three-dimensional point cloud is obtained:
(2) carrying out diagonal transformation on a three-dimensional covariance matrix of the three-dimensional point cloud to obtain a characteristic value and a characteristic vector of the three-dimensional covariance matrix; representing the eigenvector with the largest eigenvalue as the longest side direction of the directional bounding box, and searching the pineapple point cloud length boundary point according to the eigenvector direction to obtain the long side of the bounding box;
(3) Projecting the characteristic vector with the maximum characteristic value onto a base plane parallel to an imaging plane of the camera, making a normal vector perpendicular to the characteristic vector on the base plane, and searching a width boundary point of the pineapple point cloud along the direction of the normal vector to obtain a wide edge of the bounding box, wherein the wide edge represents the diameter of the pineapple;
(4) Performing cross multiplication on the eigenvector with the maximum eigenvalue in the step (2) and the normal vector in the step (3) to obtain a normal vector of a base plane, searching a height boundary point of the pineapple along the normal vector direction of the base plane to obtain the height of the bounding box, wherein the height represents the radius of the pineapple;
(5) And measuring the long edge, the wide edge and the height of the bounding box, and converting to obtain the size of the pineapple according to pixel statistics and the conversion relation between the pixel distance and the real physical distance.
4. The method for automatically grading and sorting pineapples based on binocular vision and multispectral detection technology according to claim 1, wherein the method comprises the following steps: the detection of the maturity of the pineapple comprises the following steps:
(1) Finding pineapples in the binocular corrected image by using a Mask R-CNN algorithm, segmenting the pineapple image, counting the area S 1 of one side image of the pineapples, wherein the unit is a pixel, turning the pineapples, and counting the area S 2 of the other side image of the same pineapple;
(2) setting a threshold value T, wherein the setting range of T is 120-150, and the G component image is segmented by adopting a formula (3) to obtain an output image, wherein G src (x, y) represents the G component image of the pineapple, and dst (x, y) represents the output image;
(3) Counting an area C 1 with a gray scale value of 255 in the output image dst (x, y) and the unit is pixel, C 1 represents the green part area of the pineapple skin on the pineapple image, and similarly, the green part area on the other side of the pineapple image is marked as C 2, calculating the occupation ratio i of the green part area of the pineapple skin:
(4) Setting a maturity threshold m, wherein the setting range of m is 0.5-0.75, and when i is larger than m, judging the pineapple to be immature.
5. the method for automatically grading and sorting pineapples based on binocular vision and multispectral detection technology according to claim 1, wherein the method comprises the following steps: multispectral detection of the interior of the pineapple comprises the following steps:
(1) Establishing the spectral region of normal mature pineapples: putting the normal ripe pineapples into a spectrum detector, creating a mathematical model according to the reflectivity of the normal ripe pineapples in light of 600-900 nm wave band, fitting an average spectrum curve, and drawing out a spectrum area of the normal ripe pineapples;
(2) and (3) placing the pineapple to be detected in a multispectral detector, detecting the reflectivity of the pineapple at a 600-900 nm waveband, creating a scatter diagram according to the reflectivity, and when more than 85% of points in the scatter diagram fall within the spectral region of the normal mature pineapple, the interior of the pineapple to be detected is not damaged and is the pineapple with normal quality.
6. The utility model provides a pineapple automatic grading sorting device based on binocular vision and multispectral detection technique which characterized in that: the device comprises a double-side bent plate chain conveyor belt, a U-shaped groove, a shunting groove, a push rod, a binocular camera and a multispectral detector; a conveying belt shell is provided with a double-side bent plate chain conveying belt, and a U-shaped groove is positioned above the double-side bent plate chain conveying belt; the two shunting grooves are respectively positioned on two sides of the transmission belt shell; the two push rod fixing supports are respectively positioned at two sides of the conveyor belt shell and correspond to the two shunting grooves, and the push rods are positioned above the push rod fixing supports; the camera support is fixed above the transmission belt shell, and the binocular camera is installed on the camera support; the sealed box is arranged between the two shunting grooves, and a multispectral detector and a halogen lamp are arranged in the sealed box; a binocular camera, a first splitter box, a first push rod, a multispectral detector, a second splitter box and a second push rod are sequentially arranged in the conveying direction of the double-side bent plate chain conveyor belt.
7. The automatic grading and sorting device for pineapples according to claim 6, wherein: the two ends of the arc-shaped elastic connecting plate are arranged above the two adjacent U-shaped grooves through the two hinges, one page of each hinge is connected with the arc-shaped elastic connecting plate, the other page of each hinge is fixed on the U-shaped groove, and the U-shaped grooves are connected in pairs through the arc-shaped elastic connecting plates and the hinges.
8. the automatic grading and sorting device for pineapples according to claim 6, wherein: the shell bionic protective cover is located right above the camera support horizontal rod and used for preventing the binocular camera from being directly irradiated by sunlight and being wetted by rainfall, the cross section of the shell bionic protective cover is approximately elliptical, and the edge of the shell bionic protective cover is retracted inwards.
9. The automatic grading and sorting device for pineapples according to claim 6, wherein: the bionic ant leg turnover mechanisms are arranged on the U-shaped grooves, and more than 1 bionic ant leg turnover mechanism is arranged on two sides of each U-shaped groove at least.
10. The automatic grading and sorting device for pineapples according to claim 9, wherein: the bionic ant leg turnover mechanism is provided with four limbs which are connected through four joints in sequence; the first joint is a base joint, the driving motor is arranged on the U-shaped groove, and an output shaft of the driving motor is connected with the base part of the first limb section of the bionic ant leg and is used for driving the whole bionic ant leg to rotate; the driving motors of the second joint and the third joint are respectively arranged at the tail end of the first limb section and the tail end of the second limb section of the bionic ant leg and are respectively used for driving the second limb section and the third limb section to rotate; the fourth joint is driven by air pressure; tiny seta are arranged at the tail end of the bionic ant leg overturning mechanism.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910938730.9A CN110548699B (en) | 2019-09-30 | 2019-09-30 | Automatic grading and sorting method and device for pineapples based on binocular vision and multispectral detection technology |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910938730.9A CN110548699B (en) | 2019-09-30 | 2019-09-30 | Automatic grading and sorting method and device for pineapples based on binocular vision and multispectral detection technology |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110548699A true CN110548699A (en) | 2019-12-10 |
| CN110548699B CN110548699B (en) | 2024-04-05 |
Family
ID=68742047
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910938730.9A Active CN110548699B (en) | 2019-09-30 | 2019-09-30 | Automatic grading and sorting method and device for pineapples based on binocular vision and multispectral detection technology |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110548699B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112604987A (en) * | 2020-12-17 | 2021-04-06 | 广州市永合祥自动化设备科技有限公司 | Vehicle for harvesting field pineapples |
| CN112775008A (en) * | 2020-12-17 | 2021-05-11 | 广州市永合祥自动化设备科技有限公司 | Visual sorting detection system |
| CN112990063A (en) * | 2021-03-30 | 2021-06-18 | 北京林业大学 | Banana maturity grading method based on shape and color information |
| CN113641517A (en) * | 2021-08-10 | 2021-11-12 | 平安科技（深圳）有限公司 | Service data sending method and device, computer equipment and storage medium |
| CN113751345A (en) * | 2021-09-15 | 2021-12-07 | 广东省农业科学院农业质量标准与监测技术研究所 | Pineapple nondestructive testing and grading device and method |
| CN114170535A (en) * | 2022-02-11 | 2022-03-11 | 北京卓翼智能科技有限公司 | Target detection positioning method, device, controller, storage medium and unmanned aerial vehicle |
| CN116453111A (en) * | 2023-06-16 | 2023-07-18 | 广东海洋大学 | Pineapple maturity analysis method based on lightweight YOLOv4 |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4843561A (en) * | 1985-05-01 | 1989-06-27 | Sun Controls Design, Inc. | Apparatus for locating and manipulating agricultural products |
| CN1664568A (en) * | 2005-03-28 | 2005-09-07 | 浙江大学 | Fruit quality detection system based on image information fusion technology |
| US20130126399A1 (en) * | 2010-07-02 | 2013-05-23 | Strube Gmbh & Co. Kg | Method for classifying objects contained in seed lots and corresponding use for producing seed |
| CN203578234U (en) * | 2013-11-19 | 2014-05-07 | 中国科学院沈阳自动化研究所 | Red date appearance online visual detecting and grading system |
| CN105772410A (en) * | 2016-03-09 | 2016-07-20 | 华南农业大学 | Intelligent illumination control and sorting system for fruit and vegetable quality detection |
| CN206161515U (en) * | 2016-08-31 | 2017-05-10 | 华中农业大学 | Globular agricultural product inside quality detects classification equipment based on spectral technique |
| CN106733701A (en) * | 2016-12-23 | 2017-05-31 | 郑州轻工业学院 | Potato multi-stage combination dysnusia detecting system and detection method |
| CN206384229U (en) * | 2017-01-12 | 2017-08-08 | 上海烨得自动化科技股份有限公司 | Vanning collating unit |
| CN110125044A (en) * | 2019-05-27 | 2019-08-16 | 山东工商学院 | A kind of apple sorting equipment based on binocular vision |
| CN211134667U (en) * | 2019-09-30 | 2020-07-31 | 华南农业大学 | Pineapple automatic grading sorting device based on binocular vision and multispectral detection technology |
-
2019
- 2019-09-30 CN CN201910938730.9A patent/CN110548699B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4843561A (en) * | 1985-05-01 | 1989-06-27 | Sun Controls Design, Inc. | Apparatus for locating and manipulating agricultural products |
| CN1664568A (en) * | 2005-03-28 | 2005-09-07 | 浙江大学 | Fruit quality detection system based on image information fusion technology |
| US20130126399A1 (en) * | 2010-07-02 | 2013-05-23 | Strube Gmbh & Co. Kg | Method for classifying objects contained in seed lots and corresponding use for producing seed |
| CN203578234U (en) * | 2013-11-19 | 2014-05-07 | 中国科学院沈阳自动化研究所 | Red date appearance online visual detecting and grading system |
| CN105772410A (en) * | 2016-03-09 | 2016-07-20 | 华南农业大学 | Intelligent illumination control and sorting system for fruit and vegetable quality detection |
| CN206161515U (en) * | 2016-08-31 | 2017-05-10 | 华中农业大学 | Globular agricultural product inside quality detects classification equipment based on spectral technique |
| CN106733701A (en) * | 2016-12-23 | 2017-05-31 | 郑州轻工业学院 | Potato multi-stage combination dysnusia detecting system and detection method |
| CN206384229U (en) * | 2017-01-12 | 2017-08-08 | 上海烨得自动化科技股份有限公司 | Vanning collating unit |
| CN110125044A (en) * | 2019-05-27 | 2019-08-16 | 山东工商学院 | A kind of apple sorting equipment based on binocular vision |
| CN211134667U (en) * | 2019-09-30 | 2020-07-31 | 华南农业大学 | Pineapple automatic grading sorting device based on binocular vision and multispectral detection technology |
Non-Patent Citations (1)
| Title |
|---|
| 汪懋华 等: "现代农业理论与实践-安徽现代农业博士科技论坛论文集", 中国铁道出版社有限公司, pages: 227 - 231 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112604987A (en) * | 2020-12-17 | 2021-04-06 | 广州市永合祥自动化设备科技有限公司 | Vehicle for harvesting field pineapples |
| CN112775008A (en) * | 2020-12-17 | 2021-05-11 | 广州市永合祥自动化设备科技有限公司 | Visual sorting detection system |
| CN112990063A (en) * | 2021-03-30 | 2021-06-18 | 北京林业大学 | Banana maturity grading method based on shape and color information |
| CN112990063B (en) * | 2021-03-30 | 2023-06-30 | 北京林业大学 | Banana maturity grading method based on shape and color information |
| CN113641517A (en) * | 2021-08-10 | 2021-11-12 | 平安科技（深圳）有限公司 | Service data sending method and device, computer equipment and storage medium |
| CN113641517B (en) * | 2021-08-10 | 2023-08-29 | 平安科技（深圳）有限公司 | Service data transmitting method, device, computer equipment and storage medium |
| CN113751345A (en) * | 2021-09-15 | 2021-12-07 | 广东省农业科学院农业质量标准与监测技术研究所 | Pineapple nondestructive testing and grading device and method |
| CN114170535A (en) * | 2022-02-11 | 2022-03-11 | 北京卓翼智能科技有限公司 | Target detection positioning method, device, controller, storage medium and unmanned aerial vehicle |
| CN116453111A (en) * | 2023-06-16 | 2023-07-18 | 广东海洋大学 | Pineapple maturity analysis method based on lightweight YOLOv4 |
| CN116453111B (en) * | 2023-06-16 | 2023-11-21 | 广东海洋大学 | Pineapple maturity analysis method based on lightweight YOLOv4 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110548699B (en) | 2024-04-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110548699B (en) | Automatic grading and sorting method and device for pineapples based on binocular vision and multispectral detection technology | |
| CN211134667U (en) | Pineapple automatic grading sorting device based on binocular vision and multispectral detection technology | |
| CN103808669B (en) | A kind of apple small holes caused by worms fast non-destructive detection method based on high light spectrum image-forming technology | |
| CN113920376B (en) | Wheat seed variety identification method based on lightweight convolutional neural network | |
| CN101832926B (en) | Method for performing apple powder materialization non-destructive inspection by using hyper-spectral image technique | |
| US20150302567A1 (en) | System and method for sun glint correction of split focal plane visible and near infrared imagery | |
| CN106525732A (en) | Apple interior and exterior quality fast nondestructive testing method based on hyperspectral imaging technology | |
| CN102890092B (en) | Characteristic angle cosine value method for detecting defects of honey peach brown rot | |
| CN103698009B (en) | A kind of multispectral image acquisition method based on line sweep high light spectrum image-forming and system | |
| CN109001218A (en) | Apple surface defect quick nondestructive recognition methods based on high light spectrum image-forming technology | |
| CN114354637A (en) | Fruit quality comprehensive grading method and device based on machine vision and X-ray | |
| CN114419311B (en) | Multi-source information-based passion fruit maturity nondestructive testing method and device | |
| CN210377552U (en) | Fruit is multiaspect image acquisition device for classification | |
| CN112098415A (en) | Nondestructive testing method for quality of waxberries | |
| CN114486939A (en) | Lens scratch detection system and method | |
| WO2023050603A1 (en) | Accurate person recognition apparatus and method for using same | |
| CN113670934B (en) | Omnibearing rapid detection device and method for defects of notebook computer shell | |
| CN115496729A (en) | Cherry defect and classification detection method based on improved YOLOX model | |
| CN108593104A (en) | A kind of small-sized high s/n ratio hand-held spectral detection system | |
| CN115015180B (en) | Fruit sugar degree nondestructive testing method based on spectral reflectance reconstruction technology | |
| CN215613293U (en) | Seed sorting device based on machine vision | |
| CN116448769B (en) | Multi-mode information fusion plate defect detection system and detection method thereof | |
| CN112729545B (en) | Miniature spectrometer based on variable-pitch grating | |
| CN210321639U (en) | Curved surface mirror surface three-dimensional detection device based on reflection imaging | |
| CN114740007B (en) | Non-contact surface defect detection device and use method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |