CN116977274A - Intelligent nut detection method and system suitable for mobile production line - Google Patents
Intelligent nut detection method and system suitable for mobile production line Download PDFInfo
- Publication number
- CN116977274A CN116977274A CN202310575519.1A CN202310575519A CN116977274A CN 116977274 A CN116977274 A CN 116977274A CN 202310575519 A CN202310575519 A CN 202310575519A CN 116977274 A CN116977274 A CN 116977274A
- Authority
- CN
- China
- Prior art keywords
- nut
- point cloud
- point
- algorithm
- fitting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 46
- 238000004519 manufacturing process Methods 0.000 title claims abstract description 24
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 55
- 238000000034 method Methods 0.000 claims abstract description 53
- 230000007547 defect Effects 0.000 claims abstract description 16
- 230000003287 optical effect Effects 0.000 claims abstract description 14
- 230000009467 reduction Effects 0.000 claims abstract description 9
- 239000011159 matrix material Substances 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 12
- 230000011218 segmentation Effects 0.000 claims description 12
- 238000007781 pre-processing Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000002950 deficient Effects 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 238000013135 deep learning Methods 0.000 claims description 5
- 206010027146 Melanoderma Diseases 0.000 claims description 2
- 230000001788 irregular Effects 0.000 abstract description 3
- 235000013399 edible fruits Nutrition 0.000 description 31
- 238000013527 convolutional neural network Methods 0.000 description 12
- 238000005516 engineering process Methods 0.000 description 12
- 238000005259 measurement Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 239000013598 vector Substances 0.000 description 8
- 238000011161 development Methods 0.000 description 6
- 230000018109 developmental process Effects 0.000 description 6
- 244000061456 Solanum tuberosum Species 0.000 description 4
- 235000002595 Solanum tuberosum Nutrition 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 235000007688 Lycopersicon esculentum Nutrition 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 238000000691 measurement method Methods 0.000 description 3
- 238000011176 pooling Methods 0.000 description 3
- 241000227653 Lycopersicon Species 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 2
- 235000012730 carminic acid Nutrition 0.000 description 2
- 238000013170 computed tomography imaging Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 235000012015 potatoes Nutrition 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- SUBDBMMJDZJVOS-UHFFFAOYSA-N 5-methoxy-2-{[(4-methoxy-3,5-dimethylpyridin-2-yl)methyl]sulfinyl}-1H-benzimidazole Chemical compound N=1C2=CC(OC)=CC=C2NC=1S(=O)CC1=NC=C(C)C(OC)=C1C SUBDBMMJDZJVOS-UHFFFAOYSA-N 0.000 description 1
- 206010063385 Intellectualisation Diseases 0.000 description 1
- 240000003768 Solanum lycopersicum Species 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000007664 blowing Methods 0.000 description 1
- 235000020971 citrus fruits Nutrition 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 230000008676 import Effects 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000013441 quality evaluation Methods 0.000 description 1
- 238000007430 reference method Methods 0.000 description 1
- 230000008521 reorganization Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Abstract
The application relates to the field of metering and image recognition of irregular objects, and discloses an intelligent nut detection method and system suitable for a mobile production line, wherein the intelligent nut detection method comprises the steps of acquiring nut point cloud data by using an Azure Kinect camera, and performing noise reduction and point cloud cutting pretreatment; dividing the background point cloud and the nut point cloud; performing plane fitting on the background point cloud, and rotating the background point cloud to a reference plane; searching a single nut peak as the highest point of the local point cloud; registering by using a Horn's Method algorithm; the PNN algorithm is provided for dividing the plurality of nut point clouds into single nut point clouds; removing edge noise of the single nut point cloud based on an improved SBF algorithm; fitting the denoised single nut point cloud fitting parameters based on a least square ellipsoid fitting algorithm to obtain nut volumes; acquiring a nut defect position by combining the optical image and the thermal infrared image; the application can be widely applied to the field of metering and identifying irregular three-dimensional objects.
Description
Technical Field
The application relates to the field of metering and image recognition of irregular objects, in particular to an intelligent nut detection method and system suitable for a mobile production line.
Background
In the process of rapid development of nut production and processing industry in China, various problems are faced. The post-treatment technology after picking the nuts has relatively low adoption rate, and the commercialized treatment mode is relatively backward. These problems not only severely affect the quality of the nuts, but also limit the price of the nuts. Along with the continuous healthy development of national economy, the income of people is steadily improved, and the consumption structure is continuously optimized and upgraded. In particular to the consumer level of nuts, consumers will necessarily be more demanding with respect to the appearance, uniformity and inherent quality of nuts. In such situations, there is naturally also a growing demand for commercial processing of nuts. The first step of nut commercialization treatment is to sort and classify nuts, classify nuts according to quality inspection standards and remove defective products in time; in the dynamic quality screening process of nuts, bad nuts can be automatically identified and the positions of the defects of the nuts can be obtained. While current machine vision techniques are rapidly evolving, there are a number of shortcomings in the practical application of the process to nut production. For example, the size and the volume of nuts are mainly combined with manual and mechanical equipment, and the identification of bad nuts of nuts is mainly judged manually by relying on the experience of fruit farmers. However, in the time of automation, the manual sorting cost is high, the time is long, the accuracy is low, errors caused by experience differences easily exist, and the quality problem of nuts after sorting can not be guaranteed.
In recent years, with the development of machine vision technology, computer hardware technology, image processing technology, pattern recognition and artificial intelligence technology, and the perfection of nut classification standards and systems, more and more modern technologies are combined with traditional nut classification technologies, and a new approach is provided for nut counting, intelligent measurement and quality detection.
Image measurement method: in 2010, zhou Pingdeng in the text of "calculating method of egg volume and surface area based on machine vision", the volume and surface area of an egg are measured by using a machine vision technology, and an image pixel is proposed to measure the volume Vp and the surface area Sp of the egg, and a drainage method and a gyroscope are applied to measure the volume V and the surface area S of the egg, so that a relation model between the volume Vp and the volume V and between the surface area Sp and the surface area S is established, the correlation coefficient r is 0.965 and 0.971 respectively, the accuracy of the egg volume compared with the measured volume is 92%, the accuracy of the egg surface compared with the measured volume is 88%, and a new reference method is provided for online measurement of geometric characteristic parameters of the egg, so that the method has a certain practical value. The image measuring method has the advantages of simple method, low cost and low precision and has high requirement on ambient light.
CT imaging method: in 2015, rogge et al in A3D contour based geometrical model generator for complex-shaped horticultural products, propose a new method for establishing a new agricultural product geometric model, acquire a CT image of a fruit by adopting X rays, form A3D contour of the fruit by 2D image contour signals through coordinate transformation, directly import the 3D contour model into CAD software to carry out grid division on the model to generate a geometric model, and measure the volume and the surface area of the geometric model. The CT imaging method has the advantages of high precision, complex operation, high equipment price and complex data processing.
Binocular imaging method: in 2010, m.omid et al, estimating volume and mass of citrus fruits by image processing technique, in one article, used two cameras to reconstruct a three-dimensional model of fruit. The fruit volume is calculated by dividing the fruit image into a number of basic elliptical frustums. The volumes were calculated as the sum of the volumes of the individual frustums using the VB program. The calculated volume matches well with the actual volume measured by the drainage method. The binocular imaging method has the advantages of low cost, mature algorithm and certain dependence on ambient light and physical surface textures.
Depth camera measurement: in 2017, qinghuaSu et al in Potato feature prediction based on machine vision and 3D model rebuilding built an image acquisition system using a PrimeSense Carmine depth camera, acquired 110 potato depth images, processed the images, extracted characteristic parameters such as image length, width, thickness and volume, and calculated the error between the image measurement and the manual measurement. The results show that the average error between length, width and thickness is 2.5%,3.5% and 4.4%, respectively, and that the 3D volume model is able to accurately distinguish normal potatoes from abnormal potatoes, with a volume correct fraction of 93% and a surface area correct fraction of 73%. The PrimeSense Carmine depth camera can measure and grade the characteristic parameters of the agricultural products such as size, shape, volume and surface area. The depth camera measuring method has the advantages of high precision and low cost, and the existing algorithm is not mature.
Three-dimensional point cloud measurement: the method and the system for measuring the single tree yield of the fruit tree acquire three-dimensional point cloud data of the fruit tree and preprocess the three-dimensional point cloud data to acquire a three-dimensional point cloud data set of the fruit tree; dividing the three-dimensional point cloud data set of the fruit tree to obtain a three-dimensional point cloud data set of the fruit only containing fruit information; counting the three-dimensional point cloud data sets of the fruits to obtain N three-dimensional point cloud data subsets of the fruits and N numbers of the fruits; calculating the corresponding fruit radius of each fruit according to the three-dimensional point cloud data subsets of the fruits to obtain a fruit radius array; and calculating the fruit quality corresponding to each fruit radius in the fruit radius array one by one according to basic parameters in the relation model of the fruit radius and the fruit quality, and calculating the yield of the whole fruit tree in an accumulated way. The method is complicated for processing the point cloud data, depends on the existing method and software more, is not technically and algorithmically innovative, and is not high enough in measurement accuracy of the phenotype parameters due to the shielding among different fruit trees and among organs and the influence of other external conditions (such as wind blowing).
Machine vision technology: with the development of artificial intelligence, many students study visual angle turning to deep learning. The Baigvand.M designs a fig grading system based on computer vision, firstly, a CCD camera is adopted to finish quality detection of figs, and then the figs are graded into 5 grades through a conveying mechanism; the tomato grading system based on machine vision is developed and designed in Araker.M.P. Computer Vision Based Fruit Grading System for Quality Evaluation of Tomato in Agriculture industry, and the automation degree and grading efficiency of the grading system are improved. However, both studies do not achieve the dynamic quality detection of tomatoes, so that the quality detection cannot be more intelligent and refined.
In summary, there is a certain theory and technology for researching fruit phenotype at home and abroad, but the existing measurement method needs advanced equipment and instruments, researchers need to master a certain operation technology, the cost and threshold are high, the average price of the nut sorting machine on the market at present reaches about fifty thousand, and the result is easy to have errors caused by experience differences. Therefore, the low-cost, high-accuracy, automatic and intelligent phenotype parameter dynamic measurement method is the development direction of future sorting classification and phenotype parameter measurement. The application can normally operate by only one depth camera of about 2000 yuan, one microcomputer of about 2000 yuan and one thermal infrared camera of 800 yuan and one conveyor belt.
Disclosure of Invention
In order to solve the technical problems, the application aims to provide the intelligent nut detection method and system suitable for the mobile production line, which not only can accurately, quickly and nondestructively count the number of nuts and measure the size (including volume, length and width) of each nut, but also can dynamically detect the quality of nuts on a conveyor belt and position the defective nuts.
The first technical scheme adopted by the application is as follows: the intelligent nut detection method suitable for the mobile production line comprises the following steps of:
collecting nut point cloud data by using an Azure Kinect camera, and carrying out noise reduction and point cloud clipping pretreatment;
dividing the background point cloud and the nut point cloud based on the point cloud color threshold;
performing plane fitting on the background point cloud, and rotating the background point cloud and the nut point cloud to a reference plane;
searching a single nut peak as the highest point of the local point cloud based on the K nearest neighbor searching method;
registering the highest points of the single nut point clouds of different frames based on a horns' Method algorithm;
the PNN algorithm is provided for dividing the plurality of nut point clouds into single nut point clouds;
removing edge noise of the single nut point cloud based on an improved SBF algorithm;
fitting the denoised single nut point cloud based on a least square ellipsoid fitting algorithm, and obtaining nut volume, length and width phenotype parameters according to fitting parameters;
and (3) combining the optical image and the thermal infrared image, carrying out dynamic quality detection on nuts on the conveyor belt based on the Faster R-CNN network, and positioning the positions of the nut defects.
Further, the nut point cloud data acquisition and noise reduction and point cloud clipping preprocessing specifically comprise:
the center of the Azure Kinect camera is vertically opposite to the measured object and data are collected at a specific height of 0.4m from the measured object;
noise reduction processing is carried out on the original point cloud data based on a statistical filtering algorithm and a Gaussian filtering algorithm;
extracting the central area of the nut point cloud data, and removing redundant points.
Further, the step of dividing the background point cloud and the nut point cloud based on the point cloud color threshold specifically includes:
setting R, G, B corresponding threshold according to the background color;
judging that the nut point cloud is the nut point cloud if the threshold value is exceeded;
and judging that the threshold value is not exceeded, and obtaining the background point cloud.
Further, the step of performing plane fitting on the background and rotating the background point cloud and the nut point cloud to the reference plane, that is, the xOy plane, specifically includes:
fitting the background point cloud by using a RASANC plane fitting algorithm to obtain plane equation parameters;
rotating the background point cloud to an xOy plane according to a plane equation to obtain a rotation matrix;
the nut point cloud is rotated based on the rotation matrix.
Further, the step of searching for a single nut vertex as a local highest point based on the K nearest neighbor searching method specifically includes:
searching the highest point of each nut point cloud based on a specified radius range based on the K nearest neighbor point searching method, wherein the local highest point is the highest point of a single nut, and simultaneously counting nuts.
Further, the registering of the highest point of the single nut point cloud by using different frames is based on a horns's Method algorithm, which specifically comprises the following steps:
extracting the three-dimensional coordinates of the highest point of the single nut point cloud, and registering the nut point clouds of two adjacent frames based on a Horn's Method algorithm by utilizing the coordinates of the highest points of the nut point clouds among different frames.
Further, the step of dividing the plurality of nut point clouds into single nut point clouds by the PNN algorithm specifically includes:
arranging all nut point clouds in a descending order according to the height, and sequentially judging the plane distance between each point and the highest point of the single nut point cloud;
if the plane distance between a certain point and the highest point of the nuts is minimum, the point is judged to be the nut point cloud with the nearest plane distance.
Further, the step of removing the edge noise of the single nut point cloud based on the improved SBF algorithm specifically comprises the following steps:
judging whether the edge of the single nut point cloud is a noise point or not based on PCA algorithm projection, and removing edge noise based on an angle threshold;
further, the step of fitting the denoised single nut point cloud based on a least square ellipsoid fitting algorithm, and obtaining the nut volume according to fitting parameters, wherein the formula of the least square ellipsoid fitting formula is expressed as follows:
wherein x, y and Z are single nut point cloud coordinates, phi and omega are rotation angles about X, Y and Z axes, a, b and c are half-axis parameters of ellipsoids, and x c 、y c 、z c U and v are angle parameters for the center of the ellipsoid;
and (3) carrying out dynamic quality detection on nuts on the conveyor belt based on the Faster R-CNN network, and positioning the positions of the nut defects.
Further, the combining of the optical image and the thermal infrared image is based on the fast R-CNN network to perform dynamic defect detection on nuts on the conveyor belt, and the positioning of the defect positions of the nuts specifically comprises:
and acquiring the optical image and the thermal infrared image of the nuts by using an Azure Kinect camera, carrying out image detection on broken black spot nuts based on a fast R-CNN network, and positioning the defective nuts.
The second technical scheme adopted by the application is as follows: nut intelligent detection system suitable for remove on production line includes:
the point cloud data acquisition module acquires original point cloud data by using Azure Kinect;
the point cloud preprocessing module is used for denoising and point cloud clipping preprocessing of the original point cloud;
the nut point cloud and background point cloud segmentation module segments background point cloud data and nut data based on a point cloud color threshold;
different frames of nut point cloud registration modules are used for registering the highest points of different frames of single nut point clouds based on a Horn's method algorithm;
the single nut point cloud segmentation module is used for providing a PNN algorithm to segment single nut point clouds and simultaneously calculating the number of nuts;
the single nut point cloud denoising module is used for removing single nut point cloud edge noise based on an improved SBF algorithm;
the nut phenotype parameter calculation module is used for calculating parameters after fitting based on least square ellipsoid fitting and calculating the volume, length and width of nuts according to the parameters;
based on the deep learning defect nut detection module, combining an optical image and a thermal infrared image, and carrying out dynamic quality detection on nuts on a conveyor belt based on a fast R-CNN network to obtain nut defect positions.
The method and the system have the beneficial effects that: the application uses an Azure Kinect depth camera to collect the image data and depth information of nuts; setting threshold segmentation background point cloud data and nut data for point cloud color by a threshold segmentation method and a morphological processing method in an image processing technology, so as to ensure the efficiency and accuracy of nut measurement; firstly, a PNN algorithm is put forward in a point cloud segmentation module to segment single nut point clouds, and the nut number is calculated; obtaining the phenotype characteristic parameters of each nut by using an optimized point cloud denoising algorithm and a fitting model with higher precision; and combining the optical image and the thermal infrared image, and utilizing a Faster R-CNN convolutional neural network to perform dynamic quality detection on nuts on the conveyor belt, so as to identify bad fruits. The method systematically measures the phenotypic parameters of nuts, provides technical support for nut cultivation in China, and provides a reference idea for measuring phenotypic parameters of other crops. In nut quality detection, deep learning is used for carrying out transfer learning on fruit growers experience, so that errors caused by experience differences are reduced. The technical process, the working efficiency and the quality of the whole application embody incomparable superiority and uniqueness of the traditional means, can accurately, rapidly and nondestructively count nuts and measure the phenotypic characteristic parameters, has low use cost, can effectively assist agricultural automation, intellectualization and fine management, meets the market demands of industrial high-speed development and food quality safety, and has wide market prospect.
Drawings
FIG. 1 is a flow chart of steps of a method for intelligent detection of nuts on a mobile manufacturing line in accordance with an embodiment of the present application;
FIG. 2 is a schematic view of a measuring device according to an embodiment of the present application;
FIG. 3 is a schematic representation of nut point cloud data after pretreatment in accordance with an embodiment of the present application;
FIG. 4 is a schematic representation of multiple cloud nut point extraction in accordance with an embodiment of the present application;
FIG. 5 is a schematic diagram of a PNN algorithm according to an embodiment of the present application;
FIG. 6 is a schematic representation of a single nut point cloud segmentation in accordance with an embodiment of the present application;
FIG. 7 is a schematic diagram of ellipsoid volume calculation according to the embodiment of the present application;
FIG. 8 is a schematic diagram of detection of nut defects based on deep learning;
FIG. 9 is a schematic diagram of the acquisition of nut defect locations by dynamic quality inspection of nuts on a conveyor belt based on the fast R-CNN network in combination with optical images and thermal infrared images in accordance with an embodiment of the present application
FIG. 10 is a block diagram of an intelligent nut detection system suitable for use in a mobile manufacturing line in accordance with an embodiment of the present application;
Detailed Description
The application will now be described in further detail with reference to the drawings and to specific examples. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.
As shown in fig. 1, the present application provides an intelligent nut detection method suitable for use in a mobile production line, the method comprising the steps of:
s1, acquiring nut point cloud data by using an Azure Kinect camera, and performing noise reduction and point cloud clipping pretreatment;
specifically, referring to fig. 2, a kernel Kinect camera is used to obtain a nut original point cloud and preprocess the original point cloud, noise reduction processing is performed on original point cloud data based on a statistical filtering algorithm and a gaussian filtering algorithm, an image is cropped, and a cropped nut point cloud is obtained, and referring to a point cloud schematic diagram after preprocessing in fig. 3.
S2, dividing the nut point cloud and the background point cloud based on the point cloud color threshold;
s2.1, setting R, G, B corresponding threshold values according to background colors;
s2.2, judging that the threshold value is exceeded, and determining that the threshold value is the nut point cloud, and referring to a nut point cloud schematic diagram after segmentation in FIG. 4;
and S2.3, judging that the threshold value is not exceeded, and obtaining the background point cloud.
S3, performing plane fitting on the background point cloud, and rotating the background point cloud and the nut point cloud to an xOy plane;
s3.1, fitting the background point cloud by using a RASANC plane fitting algorithm to obtain plane equation parameters;
s3.2, rotating the background point cloud to an xOy plane according to a plane equation to obtain a rotation matrix;
and S3.3, rotating the nut point cloud based on the rotation matrix.
Specifically, a background point cloud is fitted according to a RASCAN plane algorithm, a plane normal vector is obtained, and then included angles phi and omega between the plane normal vector and the X, Z axis of a scanning coordinate system are obtained:
in the above formula, phi and omega respectively represent rotation angles of X, Z axes in three directions, R 1 、R 3 The rotation matrixes respectively representing the X and Z axes, X, Y, Z respectively representing the input three-dimensional coordinates of the point cloud, and X ', Y ', Z ' representing the three-dimensional coordinates of the point cloud after rotation through two rotation angles;
s4, searching a single nut peak as a local highest point based on a K nearest neighbor searching method;
specifically, referring to the nut highest point schematic diagram of fig. 4, searching the highest point of each nut point cloud within a specified radius range based on the K nearest neighbor point searching method, wherein the local highest point is the highest point of a single nut:
l＝max{Z i } (2)
in the above formula, Z is the Z coordinate of the point cloud, and l is the maximum value of the Z coordinate of the point cloud.
S5, registering the highest points of nut point clouds based on different frames by using a Horn' S Method algorithm;
specifically, the Horn's Method algorithm includes the steps of calculating a center point of a point set, moving the point set to an origin, calculating an optimal rotation matrix M, and calculating a translation matrix T xyz 。
The calculation center point formula is expressed as follows:
in the above formula, P represents the point cloud set coordinates, N represents the point cloud set point cloud number, mu A Represents the point cloud A center, mu B Representing the point cloud B center.
The point set re-centering formula is expressed as follows:
in the above, A' i Representing the point cloud after the original point cloud A is centralized, B' i Representing the point cloud after the original point cloud B is centered.
The covariance matrix H between the set of computation points is formulated as follows:
in the above formula, H represents a covariance matrix between the point cloud a and the point cloud B, and N represents the number of point clouds.
The rotation matrix M and translation matrix between the set of computation points are formulated as follows:
[U,S,V]＝SVD(H) (9)
M＝VU T (10)
T xyz ＝-Rμ A +μ B (11)
in the above formula, U, S, V represents three values decomposed by SVD, M represents a rotation matrix, T xyz Representing a translation matrix.
The registration calculation formula is as follows:
in the above description, M represents a rotation matrix, X ", Y", Z "respectively represent input three-dimensional coordinates of the highest point of the nut point cloud, and X '", Y ' ", Z '" represent three-dimensional coordinates of the highest point of the nut point cloud after rotation of the rotation matrix, T xyz Representing a translation matrix.
S6, providing a PNN algorithm to divide the plurality of nut point clouds into single nut point clouds;
arranging all nut point clouds in a descending order according to the height, and sequentially judging the plane distance between each point and the highest point of the single nut point cloud;
if the plane distance between a certain point and the highest point of the nuts is minimum, judging that the point belongs to a nut point cloud with the nearest plane distance;
specifically, referring to the PNN algorithm schematic of fig. 5, points a and B are the vertices of a single nut, and points C are correctly assigned to nut #1 by comparing the planar distances between the classified points and the nut vertices to correctly classify the points as different nuts, for example, when dAC < dBC is satisfied. Then, the plane distance between the point D and the points B and C is compared, and the point D is divided into nuts #2. Finally, and by analogy, all the nut point clouds are correctly segmented into single nuts, and reference is made to the single nut point cloud segmentation schematic diagram of fig. 6.
S7, removing edge noise of the single nut point cloud based on an improved SBF algorithm;
specifically, search point x is searched based on K nearest neighbor method i The point with radius d is recorded as a point set in the neighborhoodCalculating point set based on PCA algorithm>Is composed of the normal vector n and the other two component vectors u, v (n T U T V), and the query point x in the point set i Sum pointForm a vector x j -x i Vector (x) j -x i ) uv Is the vector x j -x i Projection onto a normal plane formed by u and v. Beta j Is a vector (x) j -x i ) uv The included angle with u is expressed as follows:
β j ＝arccos{(x j -x i ) uv ,u} (13)
calculating to obtain an included angle set delta= { beta j I j e {1,.,. K }, if the maximum angle difference satisfies max { beta } j+1 -β j }>β threshold X is then i The noise is regarded as edge points to be removed.
S8, fitting the denoised single nut point cloud based on a least square ellipsoid fitting algorithm, and obtaining nut volume, length and width phenotype parameters according to fitting parameters;
specifically, the ellipsoidal model is formulated as follows
In the above, x, y and Z are single nut point cloud coordinates, phi and omega are rotation angles about X, Y and Z axes, a, b and c are half-axis parameters of ellipsoids, and x c 、y c 、z c Is the center of an ellipsoid, and u and v are angle parameters;
specifically, an ellipsoid parameter is obtained by using a Gaussian Newton least square algorithm, and the estimated parameter is expressed as follows:
s9, calculating the volume, length and width of a single nut according to the ellipsoid fitting parameters;
specifically, referring to the ellipsoid volume calculation schematic of fig. 7, the length is 2a, the width is 2b, and the single nut volume formula is expressed as follows:
in the above formula, S is the surface area of the nut ellipsoidal slice and V is the volume of the nut.
S10, detecting defective nuts according to a Faster R-CNN artificial neural network by combining an optical image and a thermal infrared image, and positioning;
specifically, after the optical nut photo and the thermal infrared photo after linear stretching algorithm are subjected to resolution downsampling, the thermal infrared photo enters a fast R-CNN extraction network to extract possible characteristics of classified nuts, and a characteristic Map (Feature Map) is generated. The signature with nut-artifact features then enters the region candidate network (Region Proposal Network) portion of the target detection network, which consists of a3 x 3 convolution layer, two 1 x 1 convolution layers, two reorganization layers, and an activation layer, predicting all regions in the input signature that are likely to contain nut-artifact (i.e., candidate regions). The output results are placed in the Proposal layer, which is responsible for correcting the location of nut defects in the graph in the RPN network. Then, the Propos al layer and the feature map are put together into a region of interest Pooling layer (ROI Pooling layer), which is a special Pooling layer, and the positions of the nut bad fruit candidate region frames in the Propos al layer are collected and extracted from the feature map at corresponding positions to generate feature candidate frames. The characteristic candidate frames are respectively classified and positioned on the occurrence of the bad nuts through the full-connection layer and the classification layer formed by regression and activation operation, so that detection and identification of the bad nuts are realized. Referring to FIG. 8 and FIG. 9, faster R-CNN artificial neural network framework and defective nut detection scenario.
As shown in fig. 10, an intelligent nut detection system suitable for use in a mobile manufacturing line, comprising:
the point cloud data acquisition module acquires original point cloud data by using Azure Kinect;
the point cloud preprocessing module is used for denoising and point cloud clipping preprocessing of the original point cloud;
the nut point cloud and background point cloud segmentation module segments background point cloud data and nut data based on a point cloud color threshold;
the single nut point cloud segmentation module is used for providing a PNN algorithm to segment single nut point clouds and simultaneously calculating the number of nuts;
the nut point cloud registration module of different frames registers by utilizing the highest point of single nut point clouds of different frames;
the single nut point cloud denoising module is used for removing single nut point cloud edge noise based on an improved SBF algorithm;
the nut phenotype parameter calculation module is used for calculating parameters after fitting based on least square ellipsoid fitting and calculating the volume, length and width of nuts according to the parameters;
and the nut dynamic quality detection module is used for identifying bad nuts based on a fast R-CNN network by combining the optical image and the thermal infrared image, and obtaining the positions of nut defects.
The content in the method embodiment is applicable to the system embodiment, the functions specifically realized by the system embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.
Nut intelligent detection system suitable for remove on production line:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement a nut intelligent detection system as described above as being suitable for use on a mobile production line.
The content in the method embodiment is applicable to the embodiment of the device, and the functions specifically realized by the embodiment of the device are the same as those of the method embodiment, and the obtained beneficial effects are the same as those of the method embodiment.
A storage medium having stored therein instructions executable by a processor, characterized by: the processor-executable instructions, when executed by the processor, are for implementing a low cost nut three-dimensional measurement and quality intelligent detection suitable for use on a mobile production line as described above.
The content in the method embodiment is applicable to the storage medium embodiment, and functions specifically implemented by the storage medium embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.
While the preferred embodiment of the present application has been described in detail, the application is not limited to the embodiment, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the application, and these equivalent modifications and substitutions are intended to be included in the scope of the present application as defined in the appended claims.
Claims (10)
1. The intelligent nut detection method suitable for the mobile production line is characterized by comprising the following steps of:
collecting nut point cloud data by using an Azure Kinect camera, and carrying out noise reduction and point cloud clipping pretreatment;
dividing the background point cloud and the nut point cloud based on the point cloud color threshold;
performing plane fitting on the background point cloud, and rotating the background point cloud and the nut point cloud to a reference plane;
searching a single nut peak as the highest point of the local point cloud based on the K nearest neighbor searching method;
registering the highest points of the single nut point clouds of different frames based on a horns' Method algorithm;
the PNN algorithm is provided for dividing the plurality of nut point clouds into single nut point clouds;
removing edge noise of the single nut point cloud based on an improved SBF algorithm;
fitting the denoised single nut point cloud based on a least square ellipsoid fitting algorithm, and obtaining nut volume and/or length-width phenotype parameters according to fitting parameters;
and combining the optical image and the thermal infrared image, and carrying out dynamic quality detection on nuts on the conveyor belt based on the Faster R-CNN network to locate the positions of the nut defects.
2. The intelligent nut detection method suitable for a mobile production line according to claim 1, wherein the steps of collecting nut point cloud data and performing noise reduction and point cloud clipping pretreatment comprise:
the center of the Azure Kinect camera is vertically opposite to the measured object and data are collected at a specific height of 0.4m from the measured object;
noise reduction processing is carried out on the original point cloud data based on a statistical filtering algorithm and a Gaussian filtering algorithm;
extracting the central area of the nut point cloud data, and removing redundant points.
3. The intelligent nut detection method suitable for use on a mobile production line according to claim 1, wherein the step of dividing the background point cloud and the nut point cloud based on the point cloud color threshold value specifically comprises:
setting R, G, B corresponding threshold according to the background color;
judging that the nut point cloud is the nut point cloud if the threshold value is exceeded;
and judging that the threshold value is not exceeded, and obtaining the background point cloud.
4. The intelligent nut detection method as defined in claim 1, wherein the step of performing a plane fitting on the background point cloud and rotating the background point cloud and the nut point cloud to the reference plane specifically comprises:
fitting the background point cloud by using a RASANC plane fitting algorithm to obtain plane equation parameters;
rotating the background point cloud to an xOy plane according to a plane equation to obtain a rotation matrix;
the nut point cloud is rotated based on the rotation matrix.
5. The intelligent nut detection method suitable for a mobile production line according to claim 1, wherein the step of searching for a single nut vertex as a local highest point based on the K nearest neighbor searching method specifically comprises:
searching the highest point of each nut point cloud based on a specified radius range based on the K nearest neighbor point searching method, wherein the local highest point is the highest point of a single nut, and simultaneously counting nuts.
6. The intelligent nut detection Method suitable for a mobile production line according to claim 1, wherein the step of registering the highest point of the single nut point cloud with different frames based on a horns' Method algorithm specifically comprises the following steps: extracting the three-dimensional coordinates of the highest point of the single nut point cloud, and registering the nut point clouds of two adjacent frames based on a Horn's Method algorithm by utilizing the coordinates of the highest points of the nut point clouds among different frames.
7. The intelligent nut detection method as defined in claim 1, wherein said step of presenting PNN algorithm to divide a plurality of nut point clouds into individual nut point clouds comprises:
arranging all nut point clouds in a descending order according to the height, and sequentially judging the plane distance between each point and the highest point of the single nut point cloud;
and if the plane distance between a certain point and the highest point of the nuts is minimum, judging that the point is classified as a nut point cloud with the nearest plane distance.
8. The intelligent nut detection method suitable for a mobile production line according to claim 1, wherein the step of fitting the denoised single nut point cloud based on a least square ellipsoid fitting algorithm and obtaining the nut volume according to fitting parameters is characterized in that the ellipsoid model is expressed as follows:
wherein x, y and Z are single nut point cloud coordinates, phi and omega are rotation angles about X, Y and Z axes, a, b and c are half-axis parameters of ellipsoids, and x c 、y c 、z c U and v are angle parameters for the center of the ellipsoid;
and calculating the volume and/or length-width phenotype parameters of the single nuts according to the parameters obtained by Gaussian Newton least squares fitting.
9. The intelligent nut detection method for mobile production line according to claim 1, wherein the step of combining the optical image and the thermal infrared image to dynamically detect the quality of nuts on the conveyor belt based on the fast R-CNN network and locate the positions of nut defects comprises the following steps:
collecting optical pictures and thermal infrared pictures of nuts by using an Azure Kinect camera and a thermal infrared camera, and performing linear processing
And (3) carrying out stranding algorithm processing, carrying out image detection on broken black spot nuts based on a fast R-CNN network, and positioning defective nuts.
10. Nut intelligent detection system suitable for remove on production line, its characterized in that includes:
the point cloud data acquisition module acquires original point cloud data by using Azure Kinect;
the point cloud preprocessing module is used for denoising and point cloud clipping preprocessing of the original point cloud;
the nut point cloud and background point cloud segmentation module segments background point cloud data and nut data based on a point cloud color threshold;
different frames of nut point cloud registration modules are used for registering the highest points of different frames of single nut point clouds based on a Horn's method algorithm;
the single nut point cloud segmentation module is used for providing a PNN algorithm to segment single nut point clouds and simultaneously calculating the number of nuts;
the single nut point cloud denoising module is used for removing single nut point cloud edge noise based on an improved SBF algorithm;
the nut phenotype parameter calculation module is used for calculating parameters after fitting based on least square ellipsoid fitting and calculating the volume, length and width of nuts according to the parameters;
based on the deep learning defect nut detection module, combining an optical image and a thermal infrared image, and based on a fast R-CNN network, carrying out dynamic quality detection on nuts on a conveyor belt to obtain nut defect positions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310575519.1A CN116977274A (en) | 2023-05-22 | 2023-05-22 | Intelligent nut detection method and system suitable for mobile production line |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310575519.1A CN116977274A (en) | 2023-05-22 | 2023-05-22 | Intelligent nut detection method and system suitable for mobile production line |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116977274A true CN116977274A (en) | 2023-10-31 |
Family
ID=88480369
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310575519.1A Pending CN116977274A (en) | 2023-05-22 | 2023-05-22 | Intelligent nut detection method and system suitable for mobile production line |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116977274A (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105115976A (en) * | 2015-06-24 | 2015-12-02 | 上海图甲信息科技有限公司 | Detection system and method for wear defects of track |
| JP2016031321A (en) * | 2014-07-30 | 2016-03-07 | アイシン精機株式会社 | Defect extraction device and defect extraction method |
| CN113469950A (en) * | 2021-06-08 | 2021-10-01 | 海南电网有限责任公司电力科学研究院 | Method for diagnosing abnormal heating defect of composite insulator based on deep learning |
| CN114119488A (en) * | 2021-10-29 | 2022-03-01 | 成都建工第一建筑工程有限公司 | Intelligent size and quality detection method for prefabricated laminated plate facing factory |
| CN115035195A (en) * | 2022-08-12 | 2022-09-09 | 歌尔股份有限公司 | Point cloud coordinate extraction method, device, equipment and storage medium |
| CN115661072A (en) * | 2022-10-25 | 2023-01-31 | 山东省计算中心（国家超级计算济南中心） | Disc rake surface defect detection method based on improved fast RCNN algorithm |
| CN115791541A (en) * | 2021-09-10 | 2023-03-14 | 苏州米顿智能视觉有限公司 | Discrete and recognition device for sticky coarse aggregate |
-
2023
- 2023-05-22 CN CN202310575519.1A patent/CN116977274A/en active Pending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2016031321A (en) * | 2014-07-30 | 2016-03-07 | アイシン精機株式会社 | Defect extraction device and defect extraction method |
| CN105115976A (en) * | 2015-06-24 | 2015-12-02 | 上海图甲信息科技有限公司 | Detection system and method for wear defects of track |
| CN113469950A (en) * | 2021-06-08 | 2021-10-01 | 海南电网有限责任公司电力科学研究院 | Method for diagnosing abnormal heating defect of composite insulator based on deep learning |
| CN115791541A (en) * | 2021-09-10 | 2023-03-14 | 苏州米顿智能视觉有限公司 | Discrete and recognition device for sticky coarse aggregate |
| CN114119488A (en) * | 2021-10-29 | 2022-03-01 | 成都建工第一建筑工程有限公司 | Intelligent size and quality detection method for prefabricated laminated plate facing factory |
| CN115035195A (en) * | 2022-08-12 | 2022-09-09 | 歌尔股份有限公司 | Point cloud coordinate extraction method, device, equipment and storage medium |
| CN115661072A (en) * | 2022-10-25 | 2023-01-31 | 山东省计算中心（国家超级计算济南中心） | Disc rake surface defect detection method based on improved fast RCNN algorithm |
Non-Patent Citations (6)
| Title |
|---|
| HUI ZHANG等: "Nondestructive Internal Defect Detection of In-Shell Walnuts by X-ray Technology Based on Improved Faster R-CNN", 《DOI:10.20944/PREPRINTS202304.1271.V1》, 30 April 2023 (2023-04-30), pages 1 - 10 * |
| XIONG PENG等: "MEASURING NUT VOLUMES USING THE AZURE KINECT", 《RESEARCHGATE》, 13 December 2022 (2022-12-13), pages 1 - 8 * |
| 康硕等: "基于红外和可见光图像融合的铺丝缺陷检测方法", 《航空学报》, vol. 43, no. 3, 25 March 2022 (2022-03-25), pages 425187 - 1 * |
| 张恒: "基于机器视觉的坚果尺寸与缺陷分类在线检测系统设计", 《中国优秀硕士学位论文全文数据库 工程科技Ⅰ辑》, no. 03, 15 March 2022 (2022-03-15), pages 024 - 664 * |
| 张成龙等: "农产品内部异物检测方法研究进展", 《安徽农业科学》, no. 16, 8 June 2018 (2018-06-08), pages 30 - 34 * |
| 齐琦: "基于高斯过程回归的车载激光雷达地面点滤波研究", 《中国优秀硕士学位论文全文数据库 工程科技Ⅱ辑》, no. 02, 15 February 2021 (2021-02-15), pages 13 - 17 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111062915B (en) | Real-time steel pipe defect detection method based on improved YOLOv3 model | |
| CN111299815B (en) | Visual detection and laser cutting trajectory planning method for low-gray rubber pad | |
| CN115082419B (en) | Blow-molded luggage production defect detection method | |
| CN109784204B (en) | Method for identifying and extracting main fruit stalks of stacked cluster fruits for parallel robot | |
| US9002072B2 (en) | System for detection of non-uniformities in web-based materials | |
| Li et al. | A leaf segmentation and phenotypic feature extraction framework for multiview stereo plant point clouds | |
| CN107230203A (en) | Casting defect recognition methods based on human eye vision attention mechanism | |
| CN108107444A (en) | Substation's method for recognizing impurities based on laser data | |
| CN115187803B (en) | Positioning method for picking process of famous tea tender shoots | |
| Zhang et al. | Computer vision estimation of the volume and weight of apples by using 3d reconstruction and noncontact measuring methods | |
| Wang et al. | 3D point cloud instance segmentation of lettuce based on PartNet | |
| Li et al. | An improved binocular localization method for apple based on fruit detection using deep learning | |
| CN113093216A (en) | Irregular object measurement method based on laser radar and camera fusion | |
| Liang et al. | Automatic defect detection of texture surface with an efficient texture removal network | |
| CN105654042B (en) | The proving temperature character identifying method of glass-stem thermometer | |
| CN105825215B (en) | It is a kind of that the instrument localization method of kernel function is embedded in based on local neighbor and uses carrier | |
| Ferrer-Ferrer et al. | Simultaneous fruit detection and size estimation using multitask deep neural networks | |
| Gu et al. | Surface defect detection of cabbage based on curvature features of 3d point cloud | |
| Zhang et al. | Corn ear test using SIFT-based panoramic photography and machine vision technology | |
| Zhao et al. | Automatic sweet pepper detection based on point cloud images using subtractive clustering | |
| Wu et al. | A Dense Litchi Target Recognition Algorithm for Large Scenes | |
| CN117079125A (en) | Kiwi fruit pollination flower identification method based on improved YOLOv5 | |
| CN116977274A (en) | Intelligent nut detection method and system suitable for mobile production line | |
| Mohana et al. | Stem-calyx recognition of an apple using shape descriptors | |
| CN115995017A (en) | Fruit identification and positioning method, device and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |