CN109359531B - Fruit harvesting area automatic positioning method facing natural scene - Google Patents
Fruit harvesting area automatic positioning method facing natural scene Download PDFInfo
- Publication number
- CN109359531B CN109359531B CN201811062603.9A CN201811062603A CN109359531B CN 109359531 B CN109359531 B CN 109359531B CN 201811062603 A CN201811062603 A CN 201811062603A CN 109359531 B CN109359531 B CN 109359531B
- Authority
- CN
- China
- Prior art keywords
- neighborhood
- fruit
- hue
- extracting
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 235000013399 edible fruits Nutrition 0.000 title claims abstract description 108
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000003306 harvesting Methods 0.000 title claims abstract description 21
- 238000001514 detection method Methods 0.000 claims abstract description 16
- 238000012549 training Methods 0.000 claims abstract description 6
- 238000010586 diagram Methods 0.000 claims description 28
- 238000000605 extraction Methods 0.000 claims description 16
- 238000012545 processing Methods 0.000 claims description 14
- 230000011218 segmentation Effects 0.000 claims description 14
- 238000003711 image thresholding Methods 0.000 claims description 8
- 230000009471 action Effects 0.000 claims description 7
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 230000003321 amplification Effects 0.000 claims description 6
- 230000005484 gravity Effects 0.000 claims description 6
- 238000003199 nucleic acid amplification method Methods 0.000 claims description 6
- 239000000284 extract Substances 0.000 claims description 5
- 230000000877 morphologic effect Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 230000007797 corrosion Effects 0.000 claims description 2
- 238000005260 corrosion Methods 0.000 claims description 2
- 238000007670 refining Methods 0.000 claims description 2
- 241001629511 Litchi Species 0.000 abstract description 43
- 230000002708 enhancing effect Effects 0.000 abstract 1
- 230000000694 effects Effects 0.000 description 22
- 238000005286 illumination Methods 0.000 description 12
- 238000003384 imaging method Methods 0.000 description 4
- 230000010339 dilation Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000007613 environmental effect Effects 0.000 description 3
- 230000003628 erosive effect Effects 0.000 description 3
- 238000013507 mapping Methods 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 239000012636 effector Substances 0.000 description 2
- 238000005429 filling process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 238000012804 iterative process Methods 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000004806 packaging method and process Methods 0.000 description 2
- 241000238370 Sepia Species 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 235000015203 fruit juice Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 239000004615 ingredient Substances 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 239000002420 orchard Substances 0.000 description 1
- 235000012173 organic fruit juice Nutrition 0.000 description 1
- 238000012856 packing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000013138 pruning Methods 0.000 description 1
- 238000002187 spin decoupling employing ultra-broadband-inversion sequences generated via simulated annealing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/77—Retouching; Inpainting; Scratch removal
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a natural scene-oriented automatic positioning method for a fruit harvesting area, which comprises the following steps: extracting litchi fruit areas in the training data, counting weighted tricolor brightness distribution characteristics of the litchi fruit areas as objective brightness reference, and enhancing weighted tricolor brightness components by adopting an iterative Retinex algorithm; extracting the fruit region with enhanced brightness by combining the corrected color difference map, thresholding treatment and mathematical morphology; reconstructing hue components in an HSI color space through the position relation of local neighborhood pixels and hue information, and extracting branch regions according to hue distribution characteristics; the branch framework is extracted by adopting a thinning algorithm, the key angular points on the framework are extracted through angular point detection and the mode distribution rule of angular point neighborhood pixels, and the litchi fruit harvesting area is automatically positioned by combining the relative position constraint of the fruits and the branches and the spatial distribution characteristic of the key angular points. The method can improve the adaptivity and the accuracy of automatic positioning of the litchi fruit harvesting area in a natural scene.
Description
Technical Field
The invention relates to the technical field of agricultural informatization, fine agriculture, machine vision and image processing, in particular to a natural scene-oriented automatic positioning method for a fruit harvesting area.
Background
Modern orchards are increasingly expensive to manage and less labor intensive to pick fruit, and tend to decrease in sustainable development in fruit picking based on labor intensive techniques (Gongal a., Amatya s., karke m., et al,2015.Sensors and systems for free detection and localization. com. electron. concrete.116, 8-19). The intelligent fruit harvesting equipment is automatic operation equipment for changing the traditional manual mode, generally comprises a mobile platform, an operation and control system, forest fruit detection, an end effector and other key components, is gradually put into automatic harvesting operation of forest fruits in various mountain areas at present, and has one main problem of reliable detection of a fruit area and accurate positioning of a harvesting point/area under a natural imaging condition.
Fruit detection and picking point/area location typically employs machine vision and image processing techniques. Aiming at the problem that the imaging quality of a forest fruit area in a natural scene is easily influenced by illumination conditions, the fruit image collected under the condition of weak illumination is processed by a homomorphic filtering method by Xu and the like (Xu L.M., Lv J.D.,2018 Recognition method for apple fruited base on SUSAN and PCNN. multimed. tools appl.77(6), 7205-. Wang et al (Wang C.L., Lee W.S, Zou X.J., et al,2018.Detection and counting of image green circulating on the Local Binary Patterns (LBP) characterization and minimization-normalized images.precision Agric.17(6), 678-. However, the above methods do not consider intelligent identification of the illumination property of the input image, and processing an image with high brightness or uniform brightness distribution is easy to cause an overexposure phenomenon or change the original hue information of a fruit region, and may affect the accuracy of subsequent fruit region detection and picking point/region positioning, so that the methods are not suitable for automatically processing forest fruit images in natural scenes. Zhuang et al (Zhuang j.j., Luo s.m., Hou c.j., et al,2018.Detection of organic fruit juice using a monolithic video-based method for automatic fruit juice packaging applications. com. electron. array.152, 64-73) automatically processes the problem of uneven illumination of the fruit region image by using a local block homomorphic filtering method, but the method contains more adjusting parameters, and can need to be readjusted when environmental factors are significantly changed, and the processing process is complicated. Wang et al (Wang C.L., Zou X.J., Tang Y.C., et al,2016. localization of litch in an unstructured electronic vision. Biosyst. Eng.145,39-51) combined with a K-means clustering method and a label template-based registration method extract a red litchi fruit region in a binocular vision image sequence, and have a better detection effect on a forest fruit image with a smaller background color difference. Aiming at the positioning problem of litchi harvesting points in a pneumatic environment, Xiong (Xiong J.T., He Z.L., Lin R., et al,2018.Visual position technology of packaging robots for dynamic litchis with distribution of ingredients. electronic. array.151, 226-237) and the like extract mature litchi fruit areas and centroids thereof from hue components of HSI images based on hue distribution knowledge of the litchi fruit areas, and estimate harvesting points from binocular Visual images through a pendulum criterion, but have larger automatic positioning errors and are suitable for end effectors with wider pruning operation range. In order to avoid The influence of environmental factors on The positioning accuracy of litchi harvesting points, a night imaging system based on LED directional light filling is adopted by Xiong and The like (Xiong J.T., Lin R., Liu Z., et al,2018.The registration of litchiclusterics and The computing of packing point in a normal environment. biosystem. Eng.166,44-57), non-forest fruit and interfering branch regions in night scene images are filtered by a fuzzy C mean value clustering method, and harvesting points of litchi fruits are positioned by combining Otsu threshold segmentation and Harris corner detection methods.
Although the current litchi fruit picking point/area automatic positioning method based on machine vision and image processing technology has achieved certain success, in a natural scene with constantly changing environmental factors, a more reliable and accurate automatic positioning method for picking points or picking areas still needs to be further explored.
Disclosure of Invention
In order to overcome at least one defect (deficiency) in the prior art, the invention provides a fruit harvesting area automatic positioning method oriented to a natural scene, and aims to improve the self-adaptability and accuracy of a fruit detection and harvesting area positioning system based on a machine vision technology.
In order to realize the purpose of the invention, the following technical scheme is adopted for realizing the purpose:
a fruit harvesting area automatic positioning method facing to natural scenes comprises the following steps:
(1) extracting weighted RGB brightness component V of current RGB image I, and generating corrected RX (Red) from I by using corrected RX color difference map (Modified Red and Green/blue (X) Chromatic Mapping, MRXCM)&Green/Blue) color difference map CmFrom CmExtracting local fruit region RoIs (regions of interest), calculating statistical characteristics of the luminance distribution of the RoIs, processing V by adopting a Retinex algorithm in an iterative mode according to the statistical characteristics, reconstructing I, and outputting an RGB image I' with enhanced luminance through a color space conversion model when the iterative process is ended;
(2) extracting and correcting RX color difference chart C from the image by MRXCM methodm', for Cm' performing image-thresholding segmentation to obtain Cm"and processing C using mathematical morphological algorithmsm"from which potentially mature or immature fruit regions O are extractedf；
(3) Extracting hue component H 'of I', measuring the position relation between local neighborhood pixels and neighborhood centers, measuring the hue information difference between the local neighborhood pixels and the neighborhood centers, establishing a weight distribution model of local neighborhood to center pixel hue values, and reconstructing hue component H 'in a weighted mean manner by isolating neighborhood center pixels polluted by noise'cTo H'cPerforming image thresholding segmentation to obtain Hc"from which potentially tan shoot region O is extracteds；
(4) Extraction of OsSkeleton Q ofs', detecting Qs'the angular point on the' is extracted, the key angular point related to the branch bifurcation property is extracted, and O is utilized under the action of gravityfAnd QsThe relative position constraint of the' and the spatial distribution characteristic of the key angular points position the distribution interval of the fruit picking points.
Further, in the step (1), the weighted three-primary color luminance component V of the current RGB image I is extracted, specifically, the gray values of the pixels of the red, green, and blue color channels of I are respectively represented by variables R, G and B, and the weighted three-primary color luminance component V is calculated according to the following formula:
V＝αR+βG+γB
further, the step (1) of generating a corrected RX (Red) from I by using a corrected RX color difference map (Modified Red and Green/blue (X) chromatographic Mapping, MRXCM) method&Green/Blue) color difference map CmSpecifically, according to the red hue or the green hue shown in the fruit region, the ratio of the single color of R to G or R to B is calculated as a scale factor, and the scale factor is weighted into the conventional red-green color difference diagram or the conventional red-blue color difference diagram to generate a corrected color difference diagram Cm。
Further, the slave C of step (1)mExtracting local fruit region RoIs (regions of interest), calculating the statistical characteristics of the luminance distribution of the RoIs, specifically for CmPerforming threshold segmentation and corrosion operation to obtain a binary image, extracting local fruit regions RoIs as mask images, and calculating the average brightness value M of the region covered by the mask images in VRoIs。
Further, in the step (1), the Retinex algorithm is adopted to process the V and reconstruct the I in an iterative manner according to the statistical characteristics, when the iterative process is terminated, the RGB image I' with enhanced brightness is output through the color space conversion model, specifically, the three primary color weighted brightness components of a plurality of training images are extracted, and the average brightness value M of the fruit region is countedtrainAnd executing the following steps:
s1, judging whether M is presentRoIs≥MtrainIf yes, go to step S2, otherwise go to step S3;
s2, taking the I as an RGB image I 'with enhanced brightness and outputting the I';
s3, obtaining an enhanced brightness component V ' by utilizing a Retinex algorithm, extracting hue H and color saturation S of the current RGB image through an RGB/HSI color space conversion model, fusing V ' to generate an RGB image I ' with enhanced brightness, and re-extracting the average M of IRoIs"judging whether M is presentRoIs＇≥MtrainIf not, continuing to execute the step S3, otherwise, continuing to execute the step S4;
s4, outputting I'.
Preferably, step (2) of processing C by using mathematical morphology algorithmm"from which potentially mature or immature fruit regions O are extractedfSpecifically, a morphological erosion operation based on the large-size structural elements, an expansion operation based on the small-size structural elements and a hole filling process C are sequentially adoptedm"from processed C using 8-connectivity criteriam"extraction of potential fruit region Of。
Further, the step (3) of measuring the position relationship between the local neighborhood pixels and the neighborhood center is specifically that (x) is measured at the neighborhood centerc,yc) In the m × n neighborhood of (c), the calculated coordinate is (x)i,yi) Is from the center of the neighborhood ((x)i,yi),(xc,yc) The position relation w between the ith pixel and the neighborhood center is obtained according to the following formulapi：
wpi＝exp{-λPmax(dis((xi,yi),(xc,yc)))}
In the formula, λpThe position amplification factor is 1,2, …, m × n.
Further, the step (3) of measuring the hue information difference between the local neighborhood pixels and the neighborhood center is to obtain the coordinate (x) of the neighborhood centerc,yc) In the m × n neighborhood of (c), the coordinates in the neighborhood are calculated as (x)i,yi) Tone value h of the ith pixel ofiAnd the median value h of the neighborhood tonemDifference of (h)i,hm) And the standard deviation sigma of the hue values of the neighborhood pixels excluding the center pointcCalculating the difference information w between the ith pixel and the neighborhood center according to the following formulaHi：
In the formula, λHI is 1,2, …, m × n, which is a hue difference amplification factor.
Further, the step (3) of establishing a weight distribution model of local neighborhood to central pixel hue value, and reconstructing hue component H 'in a weighted mean manner by isolating neighborhood central pixels polluted by noise'cSpecifically, by wpiAnd wHiThe dot product operation of (2) assigns a weight w of the ith pixelciIf the hue value h of the neighborhood centercAnd hmIf the deviation is larger, setting the weight of the center of the neighborhood as 0, otherwise, setting the weight of the center of the neighborhood as 1, and calculating the weighted average value of the hue values of all the pixels in the neighborhood as the reconstructed hue value h 'of the pixel in the center of the neighborhood'cTraversing all the pixels of H 'based on the flow to obtain a reconstructed hue component H'c。
Preferably, the para H 'of step (3)'cPerforming image thresholding segmentation to obtain Hc"from which potentially tan shoot region O is extractedsSpecifically, a segmentation threshold is set according to the branch tone information distribution in the plurality of training images, and H 'is obtained'cPerforming image thresholding segmentation to obtain Hc"from H using the 8-connectivity criterionc"extracting potential branch region Os。
Further, the step (4) of extracting OsSkeleton Q ofs', detecting Qs' the angular point on the surface of the branch is extracted, and the key angular point related to the branch bifurcation property is extracted, in particular, the refining algorithm is used for extracting OsSkeleton Q ofs' extracting Q by Harris angular point detection algorithmsAccording to the branching characteristics of different fruit branches under the condition of multiple fruit adhesion, all the angular points on the' judge whether the angular points at the branching position belong to key angular points or not by using the mode distribution rule and the skeleton connectivity of the pixels in the neighborhood of the angular points at the branching position and extract the key angular points.
Further, the step (4) utilizes the action of gravity to perform OfAnd Qs' the relative position constraint and the spatial distribution characteristic of the key angular points position the distribution interval of fruit picking points, specifically, according to the characteristic that the gravity center position of the fruit is lower than the connecting branch thereof under the action of gravity, and the strong connectivity between the fruit stalk and the branch, the fruit stalk and branch connecting area A is calculatedrThe statistical characteristics of spatial coordinates of all pixels in the image are combined with the statistical characteristics of coordinates of key corner points to extract the statistical characteristics of the coordinates of the key corner points only with OfExtracting a key angular point far away from a fruit stalk on the skeleton as a fruit harvesting central point, and taking A asrAnd the discrete degree of the spatial column coordinates of the middle pixel and the key corner points is used as the regional standard deviation deviating from the picking point, and the distribution interval of the fruit picking point is positioned according to the picking central point and the regional standard deviation.
Compared with the prior art, the technical scheme of the invention has the beneficial effects that: (1) the iterative Retinex algorithm can automatically adjust the overall brightness distribution of the input RGB image according to the brightness characteristics of the litchi fruit region, can still maintain the color information of each region of the original image when improving the overall brightness distribution of the image, has better scene adaptivity, and is suitable for image preprocessing in a natural environment; (2) meanwhile, the hue component in the HSI color space is reconstructed by utilizing the position relation and the hue information of the local neighborhood pixels, so that the noise interference can be eliminated, the actual hue information of the neighborhood center pixels can be restored to the maximum extent, and the reliability of branch region extraction can be improved; (3) the key angular points are extracted based on a Harris angular point detection algorithm and a mode distribution rule of angular point neighborhood pixels, so that the interference of noise angular points can be avoided to a certain extent, and the positioning accuracy of litchi fruit picking points or picking areas can be improved.
Drawings
Fig. 1 is a diagram illustrating an embodiment of a flow of a method for automatically positioning a fruit picking area facing a natural scene in this embodiment.
Fig. 2a is a diagram of an embodiment of an image with sufficient illumination.
FIG. 2b is a diagram illustrating an embodiment of adaptive luminance enhancement effect of the image shown in FIG. 2 a.
FIG. 2c is a diagram of an embodiment of an image with weak illumination.
FIG. 2d is a diagram illustrating an embodiment of the adaptive luminance enhancement effect of the image shown in FIG. 2 c.
FIG. 3a is a diagram illustrating an embodiment of the litchi fruit extraction effect of the image shown in FIG. 2 a.
FIG. 3b is a diagram illustrating the litchi fruit extraction effect of the image shown in FIG. 2 c.
Fig. 4a is a diagram of an example of a tone value reconstruction result when the neighborhood center is not contaminated by noise.
Fig. 4b is a diagram of an example of tone value reconstruction results when the neighborhood center has been contaminated with noise.
Fig. 5a is a diagram illustrating an embodiment of the litchi branch extraction effect of the image shown in fig. 2 a.
Fig. 5b is a diagram illustrating an embodiment of the litchi branch extraction effect of the image shown in fig. 2 c.
Fig. 6a is a diagram illustrating an embodiment of the positioning effect of the litchi fruit picking point of the image shown in fig. 2 a.
Fig. 6b is a diagram illustrating an embodiment of the positioning effect of the harvesting point of the litchi fruit according to the image shown in fig. 2 c.
Detailed Description
The drawings are for illustrative purposes only and are not to be construed as limiting the patent;
for the purpose of better illustrating the embodiments, certain features of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product;
it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
In the description of the present invention, "a plurality" means two or more unless otherwise specified.
The technical solution of the present invention is further described below with reference to the accompanying drawings and examples.
Examples
As shown in fig. 1, a method for automatically positioning a fruit harvesting area facing a natural scene includes the following steps:
(1) extracting weighted RGB brightness component V of current RGB image I, and generating corrected RX (Red) from I by using corrected RX color difference map (Modified Red and Green/blue (X) Chromatic Mapping, MRXCM)&Green/Blue) color difference map CmFrom CmExtracting local fruit region RoIs (regions of interest), calculating the statistical characteristics of the luminance distribution of the RoIs, and outputting an RGB image I' with enhanced luminance according to the statistical characteristics;
(2) extracting and correcting RX color difference chart C from the image by MRXCM methodm', for Cm' performing image-thresholding segmentation to obtain Cm"and processing C using mathematical morphological algorithmsm"from which potentially mature or immature fruit regions O are extractedf；
(3) Extracting hue component H 'of I', measuring the position relation between local neighborhood pixels and neighborhood centers, measuring the hue information difference between the local neighborhood pixels and the neighborhood centers, establishing a weight distribution model of local neighborhood to center pixel hue values, and reconstructing hue component H 'in a weighted mean manner by isolating neighborhood center pixels polluted by noise'cTo H'cPerforming image thresholding segmentation to extract potential dark brown branch region Os；
(4) Extraction of OsSkeleton Q ofs', detecting Qs'the angular point on the' is extracted, the key angular point related to the branch bifurcation property is extracted, and O is utilized under the action of gravityfAnd QsThe relative position constraint of the' and the spatial distribution characteristic of the key angular points position the distribution interval of the fruit picking points.
The imaging effect of the fruit area in the visual image is easily affected by the illumination condition, so the illumination compensation of the image is needed to be executed before the fruit detection and picking point is positioned, the hue information of the compensation real area cannot be changed significantly, the illumination compensation is not considered for the input image with uniform brightness information and higher brightness value, otherwise, the phenomenon of 'overexposure' occurs to cause the significant change of the original hue information of the fruit area. Therefore, the invention automatically enhances the brightness of the input image through an iterative Retinex algorithm according to the brightness information of the input image and the brightness distribution characteristics of the fruit area in the image with sufficient/uniform illumination.
Take litchi fruits as an example.
When the algorithm is started, the current RGB image I is set as the input RGB image Ic. In order to compensate for the luminance information of I without changing the original hue information of the image as much as possible, the weighted three-primary color luminance component V of I is extracted according to formula (1).
V＝αR+βG+γB
Where R, G and B represent the pixel gray values of the red, green and blue color channels of I, respectively, and s.t. represents the constraint. In this embodiment, the detection of the litchi fruit area with red hue is taken as a target, and three parameters in the formula (1) are set to be α ═ 0.6, β ═ 0.3, and γ ═ 0.1, respectively, so as to highlight the grayscale information of the red component in the image.
In order to separate background regions from I, which differ from the fruit color shade information, the ratio of the red component (R) to the green (G) or blue component (B) is set as an influencing factor, the higher the value of which, the more pronounced is the factorThe difference between the bright fruit and other background areas is larger, the difference is weighted into a conventional color difference map method to obtain a corrected RX color difference map calculation model shown by formulas (2) to (4), and a color difference map C is extracted from Im。
Cm＝λ×R-X (2)
λ＝R/X (3)
In order to extract the area of the litchi fruits with red tone, beta is set to be more than gamma in the embodiment.
Segmentation C using Otsu algorithmmThereby obtaining corresponding binary image
In order to ensure that the subsequently extracted regions are litchi fruit parts, the litchi fruit parts are corroded by circular structural elements with the radius exceeding 8 pixels in the embodiment
Obtaining a mask image corresponding to the fruit region, extracting the brightness image of the litchi fruit region by the dot product operation of the mask image and V, and calculating the average value M of the brightness imageRoIs. On the other hand, fruit regions are manually selected from training image data containing litchi fruits with red tones, and weighted tricolor brightness components and average brightness value M of the regions are calculatedtrain。
Comparison MRoIsAnd Mtrain：
If M isRoIs≥MtrainThen, without performing the luminance enhancement based on the Retinex algorithm on V, I' is directly output as Ic；
If M isRoIs＜MtrainThen, performing a Retinex algorithm-based luminance enhancement on V to obtain a new luminance component V', and extracting I through an RGB/HSI color space conversion modelcThe hue component H and the saturation component S are fused with V 'to generate an RGB image I' with enhanced brightness, and I is resetcContinue to calculate I according to the above flowcM of (A)RoIsEnhancing I in an iterative mannercUntil the condition M is satisfiedRoIs≥MtrainThen, the RGB image I' after the luminance enhancement is output.
Fig. 2a to fig. 2d show comparison graphs of the effects before and after the above iteration mode adopts the Retinex algorithm for processing. Fig. 2a shows a picture with sufficient illumination and the brightness enhancement effect is shown in fig. 2b, and fig. 2c shows a picture with weaker illumination and the brightness enhancement effect is shown in fig. 2 d.
Extracting a corrected RX color difference chart C from I' through formulas (2) to (4)m', using Otsu algorithm to Cm' performing a thresholding segmentation to obtain a binary image Cm"successively, erosion operation based on large-size structural elements (circular structural elements with a radius of 4 pixels may be used), dilation operation based on small-size structural elements (circular structural elements with a radius of 3 pixels may be used), and hole filling process Cm"from processed C by 8-connectivity criteriam"extracting potential fruit region and marking as Of. Among the above "large-size structural elements" and "small-size structural elements", large "and" small "are in the process Cm"the erosion operation and the dilation operation performed in time" are compared with the structural elements used. Fig. 3 shows the extraction effect of the litchi fruit region after the above treatment, fig. 3a is a diagram of the litchi fruit extraction effect of the image shown in fig. 2a, and fig. 3b is a diagram of the litchi fruit extraction effect of the image shown in fig. 2 c.
In order to ensure the accurate extraction of the sepia branch region, the invention adopts a local tone information reconstruction algorithm to regenerate the filtered tone component so as to eliminate the noise interference in the input image. The hue component H 'is extracted from I' by the RGB/HSI color space conversion model. In any m × n neighborhood of H', let the coordinates and hue values of the neighborhood center be (x)c,yc) And hcThe coordinates and hue values of the ith (i ═ 1,2, …, m × n) pixel are (x) respectivelyi,yi) And hiDetermining the ith pixel (x) according to equation (5)i,yi) From the centre of the neighbourhoodPositional relationship wpi。
wpi＝exp{-λpmax(dis((xi,yi),(xc,yc)))},(xi,yi)≠(xc,yc) (5)
Wherein λ ispIn this embodiment, λ is taken as the position amplification factorp＝2；dis((xi,yi),(xc,yc) Is a measurement point (x)i,yi) And (x)c,yc) Distance function between them, i.e. distance between the ith pixel and the center of the neighborhood, in this embodiment, the chessboard distance, i.e. dis ((x)i,yi),(xc,yc))＝max(|xi-xc|,|yi-yc|)。
In addition, another important factor influencing the estimation of the hue information of the m × n neighborhood center is the hue value of the neighborhood pixels, and in order to reduce the influence of the neighborhood center polluted by noise on the hue information reconstruction result, the invention measures the hue standard deviation sigma of the pixels except the center point in the neighborhoodcAnd calculating a domain median hmLocal neighborhood pixel (x) is scaled by equations (6) and (7)i,yi) And hmThe difference in hue information, the result marked wHi。
Wherein λ isHIn this example, λ is taken as an amplification factor of the hue differenceH＝1；diff(hi,hc) For the hue value h of the ith pixeliThe difference from the median hm of the neighborhood hue, diff (h) in this embodimenti,hc)＝||hi-hc||2，NcIs a division point (x)c,yc) The outer set of neighborhood pixels.
Since the hue value of the neighborhood center pixel will be affected by both the position and hue value of the mxn neighborhood pixels, the present invention considers w simultaneouslypiAnd wHiTo reconstruct the point (x)c,yc) The hue information of (a) is made closer to the actual hue value, and the neighborhood pixels (x) are formed by dot multiplication in the embodimenti,yi) Weight of (1), i.e. wci＝wpi×wHi. In particular, point (x)c,yc) The hue before reconstruction may have been disturbed by noise, so for (x)i,yi)＝(xc,yc) In case of hcAnd hmIf the deviation is large, set wciOtherwise, set w to 0ci1. Finally, according to the formula (8), calculating the weighted mean value of m multiplied by n neighborhood tone information as a point (x)c,yc) And (4) reconstructing the result.
When the above calculation is completed for all the pixels of H ', a reconstructed image H' of H 'can be obtained'c. Fig. 4 is a diagram showing a numerical operation result of tone value reconstruction of an m × n neighborhood center pixel, where fig. 4a is a tone value reconstruction result when a neighborhood center pixel is not polluted by noise, and fig. 4b is a tone value reconstruction result when a neighborhood center pixel is polluted by noise, where m is 5 and n is 5 in this embodiment. Each cell in fig. 4a and 4b represents a pixel, the first row of values in the cell represents the hue value of the pixel before reconstruction, the second row of values in parentheses represents the contribution weight of the hue information of the pixel during reconstruction, and the third row of values in the middle point of the neighborhood represents the hue value of the pixel after reconstruction.
Manually extracting litchi branch regions from training image data, converting the litchi branch regions into HSI color space, and counting average hue values h of the litchi branch regionsavgIs as pair H'cA division threshold value H 'subjected to binarization processing'cMiddle tone value of h or lessavgIs set as foreground, otherwise is set as background, and is set as background through 8-connectivity criterionExtracting potential branch region O from foreground images. Fig. 5 shows an effect diagram of litchi branch region extraction, fig. 5a is an embodiment diagram of litchi branch extraction effect of the image shown in fig. 2a, and fig. 5b is an embodiment diagram of litchi branch extraction effect of the image shown in fig. 2 c.
Processing O by using refinement algorithmsExtracting the skeleton O of the branch region of the litchis', detecting O by Harris algorithmsAll corner points in (let their number be k). In order to judge whether a certain angular point is positioned at a branch bifurcation point, all the angular points are traversed, and for the j (j is 1,2,.., k) th angular point, if the neighborhood pixels simultaneously meet the mode distribution rule shown by the formula (9), the coordinate of the angular point j is added to the key angular point set AcOtherwise, deleting the corner point j.
Wherein N is8(Pi) The number of pixels, P, with a value of 1 in the 8 neighborhood of corner j (excluding corner j) is represented2、P4、P6And P8Values are taken for the 4 neighborhood pixels of the corner j and they are located above, to the right, below and to the left of the corner j, respectively.
To OfPerforming a morphological dilation operation (which may employ a circular structuring element with a radius of 3 pixels) to obtain Of', mixing O withs' and Of' performing regional superposition, searching the connecting region A of the litchi fruit and the branchrAnd calculate ArMean value P of coordinates of all pixel points in the imager1Mean value P of sum column coordinatesr2. Then, according to the relative position and adhesion relation of the litchi fruit and the litchi branch region under the action of gravity, calculating AcMean value P of row coordinates of middle key angle pointsc1Mean value P of sum column coordinatesc2If A iscAs an empty set or Pr1＜Pc1From Os' Mizhou reject and Of' Branch region without connectivity, extract with O onlyfAnd the branch skeleton has connectivity. Finally, from AcMiddle searchThe corner point with the minimum line coordinate value is centered at the corner point and Pr2And Pc2The absolute difference between the two is standard difference, and the distribution interval of the picking points of the litchi fruits is determined. Fig. 6a and 6b are diagrams illustrating the positioning effect of the litchi fruit picking point, fig. 6a is a diagram illustrating an embodiment of the positioning effect of the litchi fruit picking point in the image shown in fig. 2a, and fig. 6b is a diagram illustrating an embodiment of the positioning effect of the litchi fruit picking point in the image shown in fig. 2 c.
The same or similar reference numerals correspond to the same or similar parts;
the positional relationships depicted in the drawings are for illustrative purposes only and are not to be construed as limiting the present patent;
it should be understood that the above-described embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Other variations and modifications will be apparent to persons skilled in the art in light of the above description. And are neither required nor exhaustive of all embodiments. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the claims of the present invention.
Claims (7)
1. A fruit harvesting area automatic positioning method facing to natural scenes is characterized by comprising the following steps:
(1) extracting weighted tricolor brightness component V of current RGB image I, and generating corrected RX color difference map C from I by MRXCM methodmSpecifically, according to the red hue or the green hue shown in the fruit region, the ratio of the single color of R to G or R to B is calculated as a scale factor, and the scale factor is weighted into the conventional red-green color difference diagram or the conventional red-blue color difference diagram to generate a corrected RX color difference diagram CmFrom CmExtracting the interested region of the local fruit region, and calculating the statistical characteristics of the brightness distribution of the interested region, specifically for CmPerforming threshold segmentation and corrosion operation to obtain a binary image, extracting local fruit regions RoIs as mask images, and calculating the average brightness value M of the region covered by the mask images in VRoIsAccording to statistical characteristics ofProcessing the V by adopting a Retinex algorithm in an iteration mode, reconstructing the I, and outputting an RGB image I' with enhanced brightness through a color space conversion model when the iteration process is ended; specifically, three primary color weighted brightness components of a plurality of training images are extracted, and the average brightness value M of the fruit region is countedtrainAnd executing the following steps:
s1, judging whether M is presentRoIs≥MtrainIf yes, go to step S2, otherwise go to step S3;
s2, taking the I as an RGB image I 'with enhanced brightness and outputting the I';
s3, obtaining an enhanced brightness component V ' by utilizing a Retinex algorithm, extracting hue H and color saturation S of the current RGB image through an RGB/HSI color space conversion model, fusing V ' to generate an RGB image I ' with enhanced brightness, and re-extracting the average M of IRoIs"judging whether M is presentRoIs＇≥MtrainIf not, continuing to execute the step S3, otherwise, continuing to execute the step S4;
s4, outputting I';
(2) extracting and correcting RX color difference chart C from the image by MRXCM methodm', for Cm' performing image-thresholding segmentation to obtain Cm"and processing C using mathematical morphological algorithmsm"from Cm"extracting the potentially mature or immature fruit region Of；
(3) Extracting hue component H 'of I', measuring the position relation between local neighborhood pixels and neighborhood centers, measuring the hue information difference between the local neighborhood pixels and the neighborhood centers, establishing a weight distribution model of local neighborhood to center pixel hue value, and reconstructing the hue component H by isolating neighborhood center pixels polluted by noise and in a weighted mean mannerc', for Hc' performing image thresholding segmentation to obtain Hc"from which potentially tan shoot region O is extracteds；
(4) Extraction of OsSkeleton Q ofs', detecting Qs'the angular point on the' is extracted, the key angular point related to the branch bifurcation property is extracted, and O is utilized under the action of gravityfAnd Qs' relative position constraint and space of key corner pointsDistribution characteristics, positioning the distribution interval of fruit picking points.
2. The method for automatically positioning fruit picking area facing natural scene as claimed in claim 1, wherein the step (1) extracts the weighted RGB luminance component V of the current RGB image I, specifically, the variables R, G and B are used to represent the pixel gray values of the red, green and blue color channels of I respectively, and the weighted RGB luminance component V is calculated according to the following formula: v ═ α R + β G + γ B,
α, β, and γ represent coefficients.
3. The method for automatically positioning fruit picking area facing natural scene as claimed in claim 1, wherein the step (3) is to measure the position relationship between local neighborhood pixels and neighborhood centers, specifically, the neighborhood center is (x)c,yc) In the m × n neighborhood of (c), the calculated coordinate is (x)i,yi) Is from the center of the neighborhood ((x)i,yi),(xc,yc) The position relation w between the ith pixel and the neighborhood center is obtained according to the following formulapi：
wpi＝exp{-λpmax(dis((xi,yi),(xc,yc)))}
In the formula, λpThe position amplification factor, i is 1,2, …, m × n,
m and n are the number of pixels in the row and column of the neighborhood respectively.
4. The method for automatically positioning fruit picking area facing natural scene as claimed in claim 1 or 3, wherein the step (3) is to measure the hue information difference between local neighborhood pixels and neighborhood centers, specifically, the coordinates of the neighborhood centers are (x)c,yc) In the m × n neighborhood of (c), the coordinates in the neighborhood are calculated as (x)i,yi) Tone value h of the ith pixel ofiAnd the median value h of the neighborhood tonemDifference of (h)i,hm) And the standard deviation sigma of the hue values of the neighborhood pixels excluding the center pointcCalculating the difference information w between the ith pixel and the neighborhood center according to the following formulaHi：
In the formula, λHI is 1,2, …, m × n, which is a hue difference amplification factor.
5. The method for automatically positioning fruit picking area facing natural scene as claimed in claim 4, wherein in step (3), the weight distribution model of local neighborhood to central pixel hue value is established, and hue component H 'is reconstructed in a weighted mean manner by isolating neighborhood central pixels polluted by noise'cSpecifically, by wpiAnd wHiThe dot product operation of (2) assigns a weight w of the ith pixelciIf the hue value h of the neighborhood centercAnd hmIf the deviation is more than a threshold value, setting the weight of the center of the neighborhood as 0, otherwise, setting the weight of the center of the neighborhood as 1, and calculating the weighted average value of the hue values of all the pixels in the neighborhood as the reconstructed hue value h 'of the pixel in the center of the neighborhood'cTraversing all the pixels of H 'based on the flow to obtain a reconstructed hue component H'c。
6. The natural scene oriented fruit picking area automatic positioning method as claimed in claim 1, wherein the step (4) of extracting OsSkeleton Q ofs', detecting Qs' the angular point on the surface of the branch is extracted, and the key angular point related to the branch bifurcation property is extracted, in particular, the refining algorithm is used for extracting OsSkeleton Q ofs' extracting Q by Harris angular point detection algorithms' all corner points on the surface are based on the bifurcation characteristics between different fruit branches under the condition of multi-fruit adhesion, and the mode distribution rule of the corner point neighborhood pixels at the bifurcation is utilizedAnd judging whether the angular point at the bifurcation belongs to a key angular point or not according to the connectivity of the skeleton and extracting the key angular point.
7. The natural scene oriented fruit picking area automatic positioning method as claimed in claim 1, wherein the step (4) utilizes O under gravityfAnd Qs' the relative position constraint and the spatial distribution characteristic of the key angular points position the distribution interval of fruit picking points, specifically, according to the characteristic that the gravity center position of the fruit is lower than the connecting branch thereof under the action of gravity, and the strong connectivity between the fruit stalk and the branch, the fruit stalk and branch connecting area A is calculatedrThe statistical characteristics of spatial coordinates of all pixels in the image are combined with the statistical characteristics of coordinates of key corner points to extract the statistical characteristics of the coordinates of the key corner points only with OfExtracting a key angular point far away from a fruit stalk on the skeleton as a fruit harvesting central point, and taking A asrAnd the discrete degree of the spatial column coordinates of the middle pixel and the key corner points is used as the regional standard deviation deviating from the picking point, and the distribution interval of the fruit picking point is positioned according to the picking central point and the regional standard deviation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811062603.9A CN109359531B (en) | 2018-09-12 | 2018-09-12 | Fruit harvesting area automatic positioning method facing natural scene |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811062603.9A CN109359531B (en) | 2018-09-12 | 2018-09-12 | Fruit harvesting area automatic positioning method facing natural scene |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109359531A CN109359531A (en) | 2019-02-19 |
| CN109359531B true CN109359531B (en) | 2021-12-14 |
Family
ID=65350966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811062603.9A Active CN109359531B (en) | 2018-09-12 | 2018-09-12 | Fruit harvesting area automatic positioning method facing natural scene |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109359531B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112541860B (en) * | 2019-09-23 | 2024-09-10 | 深圳开阳电子股份有限公司 | Skin color beautifying correction method and device |
| CN118058074B (en) * | 2024-02-20 | 2024-08-06 | 广东若铂智能机器人有限公司 | Method for judging burst interference in string-type fruit picking process |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101726251A (en) * | 2009-11-13 | 2010-06-09 | 江苏大学 | Automatic fruit identification method of apple picking robot on basis of support vector machine |
| CN103727877A (en) * | 2013-12-20 | 2014-04-16 | 北京农业信息技术研究中心 | Fruit identifying and locating method, device and system |
| CN104700404A (en) * | 2015-03-02 | 2015-06-10 | 中国农业大学 | Fruit location identification method |
| CN105373794A (en) * | 2015-12-14 | 2016-03-02 | 河北工业大学 | Vehicle license plate recognition method |
| CN105701829A (en) * | 2016-01-16 | 2016-06-22 | 常州大学 | Bagged green fruit image segmentation method |
| US9462749B1 (en) * | 2015-04-24 | 2016-10-11 | Harvest Moon Automation Inc. | Selectively harvesting fruits |
| CN106326894A (en) * | 2016-08-31 | 2017-01-11 | 西南交通大学 | Adverse state detection method of transverse pins of rotation double lugs of high-speed rail overhead contact line equipment |
| CN106327443A (en) * | 2016-08-24 | 2017-01-11 | 电子科技大学 | Night image enhancement method based on improved genetic algorithm |
| CN108319973A (en) * | 2018-01-18 | 2018-07-24 | 仲恺农业工程学院 | Detection method for citrus fruits on tree |
-
2018
- 2018-09-12 CN CN201811062603.9A patent/CN109359531B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101726251A (en) * | 2009-11-13 | 2010-06-09 | 江苏大学 | Automatic fruit identification method of apple picking robot on basis of support vector machine |
| CN103727877A (en) * | 2013-12-20 | 2014-04-16 | 北京农业信息技术研究中心 | Fruit identifying and locating method, device and system |
| CN104700404A (en) * | 2015-03-02 | 2015-06-10 | 中国农业大学 | Fruit location identification method |
| US9462749B1 (en) * | 2015-04-24 | 2016-10-11 | Harvest Moon Automation Inc. | Selectively harvesting fruits |
| CN105373794A (en) * | 2015-12-14 | 2016-03-02 | 河北工业大学 | Vehicle license plate recognition method |
| CN105701829A (en) * | 2016-01-16 | 2016-06-22 | 常州大学 | Bagged green fruit image segmentation method |
| CN106327443A (en) * | 2016-08-24 | 2017-01-11 | 电子科技大学 | Night image enhancement method based on improved genetic algorithm |
| CN106326894A (en) * | 2016-08-31 | 2017-01-11 | 西南交通大学 | Adverse state detection method of transverse pins of rotation double lugs of high-speed rail overhead contact line equipment |
| CN108319973A (en) * | 2018-01-18 | 2018-07-24 | 仲恺农业工程学院 | Detection method for citrus fruits on tree |
Non-Patent Citations (5)
| Title |
|---|
| A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot;Chenglin Wang等;《Optik》;20161128;第626-631页 * |
| 基于Retinex图像增强的不同光照条件下的成熟荔枝识别;熊俊涛 等;《农业工程学报》;20130630;第29卷(第12期);第170-178页 * |
| 尺度变化的Retinex红外图像增强;李毅 等;《液晶与显示》;20160131;第31卷(第1期);第104-111页 * |
| 自然环境下目标物的高速图像检测算法研究;刘阳;《中国博士学位论文全文数据库 信息科技辑》;20180115(第1期);I138-66 * |
| 苹果采摘机器人重叠果实快速动态识别及定位研究;沈甜;《中国优秀硕士学位论文全文数据库 信息科技辑》;20161115(第11期);I138-423 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109359531A (en) | 2019-02-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107578418B (en) | Indoor scene contour detection method fusing color and depth information | |
| CN107169475B (en) | A kind of face three-dimensional point cloud optimized treatment method based on kinect camera | |
| Liu et al. | A method of segmenting apples at night based on color and position information | |
| CN103927741B (en) | SAR image synthesis method for enhancing target characteristics | |
| CN105184779B (en) | One kind is based on the pyramidal vehicle multiscale tracing method of swift nature | |
| CN108319973A (en) | Detection method for citrus fruits on tree | |
| CN102867295B (en) | A kind of color correction method for color image | |
| CN106296725A (en) | Moving target detects and tracking and object detecting device in real time | |
| CN101976436B (en) | Pixel-level multi-focus image fusion method based on correction of differential image | |
| CN104751111B (en) | Identify the method and system of human body behavior in video | |
| CN109446978B (en) | Method for tracking moving target of airplane based on staring satellite complex scene | |
| CN109359531B (en) | Fruit harvesting area automatic positioning method facing natural scene | |
| CN104504722A (en) | Method for correcting image colors through gray points | |
| CN111080574A (en) | Fabric defect detection method based on information entropy and visual attention mechanism | |
| CN106886992A (en) | A kind of quality evaluating method of many exposure fused images of the colour based on saturation degree | |
| CN109754440A (en) | A kind of shadow region detection method based on full convolutional network and average drifting | |
| Yu et al. | Image and video dehazing using view-based cluster segmentation | |
| Changhui et al. | Overlapped fruit recognition for citrus harvesting robot in natural scenes | |
| CN115100240A (en) | Method and device for tracking object in video, electronic equipment and storage medium | |
| Xiang et al. | Measuring stem diameter of sorghum plants in the field using a high-throughput stereo vision system | |
| CN106558044A (en) | The resolution measuring method of image module | |
| Zheng et al. | UAV image haze removal based on saliency-guided parallel learning mechanism | |
| Setyawan et al. | Comparison of hsv and lab color spaces for hydroponic monitoring system | |
| Nursyahid et al. | Plant age identification system of outdoor hydroponic cultivation based on digital image processing | |
| CN113132693B (en) | Color correction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |