CN110599507B - Tomato identification and positioning method and system - Google Patents
Tomato identification and positioning method and system Download PDFInfo
- Publication number
- CN110599507B CN110599507B CN201810608651.7A CN201810608651A CN110599507B CN 110599507 B CN110599507 B CN 110599507B CN 201810608651 A CN201810608651 A CN 201810608651A CN 110599507 B CN110599507 B CN 110599507B
- Authority
- CN
- China
- Prior art keywords
- image
- tomato
- red
- tomatoes
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 240000003768 Solanum lycopersicum Species 0.000 title claims abstract description 360
- 235000007688 Lycopersicon esculentum Nutrition 0.000 title claims abstract description 357
- 238000000034 method Methods 0.000 title claims abstract description 53
- 238000003709 image segmentation Methods 0.000 claims abstract description 69
- 235000013399 edible fruits Nutrition 0.000 claims abstract description 54
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 38
- 230000000873 masking effect Effects 0.000 claims abstract description 33
- 238000005520 cutting process Methods 0.000 claims abstract description 18
- 238000003708 edge detection Methods 0.000 claims abstract description 17
- 238000012545 processing Methods 0.000 claims description 20
- 230000000877 morphologic effect Effects 0.000 claims description 11
- 238000001514 detection method Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000005516 engineering process Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 5
- 238000000926 separation method Methods 0.000 claims description 5
- 238000003860 storage Methods 0.000 claims description 5
- 230000011218 segmentation Effects 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 5
- 240000008042 Zea mays Species 0.000 description 3
- 235000005824 Zea mays ssp. parviglumis Nutrition 0.000 description 3
- 235000002017 Zea mays subsp mays Nutrition 0.000 description 3
- 235000005822 corn Nutrition 0.000 description 3
- 230000005070 ripening Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000003306 harvesting Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 235000013311 vegetables Nutrition 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/40—Image enhancement or restoration using histogram techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/90—Determination of colour characteristics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30181—Earth observation
- G06T2207/30188—Vegetation; Agriculture
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The embodiment of the invention provides a tomato identification and positioning method and a tomato identification and positioning system, wherein the tomato identification and positioning method comprises the following steps: acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a tomato image with a background removed; cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image; performing edge detection on the tomato image after image segmentation, and determining the coordinate position of the tomato in the tomato image after image segmentation; and acquiring point cloud data of the tomatoes according to the coordinate positions of the tomatoes in the tomato images after the image segmentation, and determining the actual positions of the tomatoes according to the point cloud data of the tomatoes. The method provided by the invention eliminates the influence of uneven light, the occlusion of the branches and the leaves to the fruits, the shadow caused by the mutual occlusion of the fruits and the like on the fruit identification.
Description
Technical Field
The embodiment of the invention relates to the technical field of image recognition, in particular to a tomato recognition and positioning method and system.
Background
The tomatoes are one of the most common vegetables cultivated in life, but the labor intensity of manual picking is high, the efficiency is low, the picking cost can be greatly reduced by developing automatic tomato harvesting equipment, and the current production situation is effectively improved. Due to the complex background of the tomato growth environment and the overlapping of fruit clusters, whether the fruit position and picking information can be accurately acquired or not becomes a big bottleneck restricting the picking of the tomato robot.
In the prior art, a lot of relevant research and application are carried out on tomato identification and positioning, wherein a common method is to use a binocular stereo vision technology for positioning, specifically, two CCD cameras are used for collecting images, and tomato position information is recovered from the images by using vision, so that the purposes of controlling the movement and positioning of a manipulator by using mechanical vision and picking tomato fruits are achieved. The other tomato identification method adopts a line structured light vision active detection means, in particular to a target area segmentation method based on a (2R-G-B) chromatic aberration model, highlights the color characteristic difference between mature fruits and a background under different sunlight conditions, establishes a depth image according to a gray scale distance mapping relation, and accordingly performs area segmentation and stereo measurement of overlapped fruits. In addition, in the document "corn stem thickness measuring method based on RGB-D camera", by Wengel et al, corn in a small flare stage is taken as an observation object, a color image and a depth image of the corn in the field are obtained by using the RGB-D camera, and crop phenotype parameters can be rapidly obtained by processing the image.
In the prior art, the tomato positioning and identifying method is too complex, the used camera equipment is expensive, meanwhile, the red ripe fruit cannot be effectively identified by adopting a line structure light visual active detection means, and the adhered parts of the fruit, the background and the fruit cannot be effectively segmented simultaneously in the aspect of image processing.
Disclosure of Invention
The embodiment of the invention provides a tomato identification and positioning method and system, which are used for solving the problems that in the prior art, a tomato identification and positioning method is too complex, camera equipment is expensive, a line structure light visual active detection means is adopted, red mature fruits cannot be effectively identified, and the adhesion parts of the fruits, a background and the fruits cannot be effectively segmented simultaneously in the aspect of image processing.
According to a first aspect of the present invention, there is provided a tomato identification and positioning method, comprising:
acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a background-removed tomato image;
cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image;
performing edge detection on the tomato image after the image segmentation, and determining the coordinate position of the tomato in the tomato image after the image segmentation;
and acquiring point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
The method comprises the steps of obtaining an image to be detected containing tomatoes, converting the image to be detected into an HSV image, integrating the HSV image into an RGB image after masking the HSV image, and masking the RGB image to obtain a background-removed tomato image, and specifically comprises the following steps:
converting the image to be detected into an HSV image, converting the HSV image into the HSV image, and separating an H channel histogram, an S channel histogram and a V channel histogram from the HSV image; respectively carrying out mask processing on the H channel histogram, the S channel histogram and the V channel histogram on a preset threshold value, and integrating the H channel histogram, the S channel histogram and the V channel histogram which are subjected to the mask processing into an RGB image; and (3) carrying out R, G, B masking treatment after the RGB image is closed and opened by a morphological technology, and further integrating to obtain the tomato image after the background is removed.
The background-removed red tomato image is cut by a watershed algorithm to obtain an image-segmented tomato image, and the method specifically comprises the following steps:
converting the tomato image after the background is removed into a gray image, and calculating the gradient amplitude of the gray image according to a Sobel operator to obtain a gradient image of the gray image; and cutting the gradient image through a watershed algorithm to obtain an image of the tomato after image segmentation.
The step of cutting the gradient image through the watershed algorithm to obtain the tomato image after image segmentation further comprises:
and performing morphological opening and closing reconstruction operation and watershed calculation on the tomato image after the image segmentation to obtain a tomato image calculated by a secondary watershed algorithm, wherein the tomato image is used as a new tomato image after the image segmentation.
Wherein, after the step of obtaining the tomato image after image segmentation, the method further comprises the following steps: and acquiring the number of the tomatoes in the tomato image after the image segmentation.
The edge detection is performed on the tomato image after the image segmentation, and the determination of the coordinate position of the tomato in the tomato image after the image segmentation specifically includes:
taking the number of the tomatoes as circle detection input parameters of Hough transform, and carrying out edge detection on the segmented tomato image to obtain foreground points of the segmented tomato image; converting the segmented tomato image into an a-b coordinate system from an x-y coordinate system, and obtaining coordinates of a point of a circle center corresponding to the tomato in the segmented tomato image and the radius of the tomato according to the coordinates of the foreground point; and obtaining the coordinate position of the tomato in the tomato image after the image segmentation according to the coordinate of the point of the circle center and the radius of the tomato.
The method comprises the steps of obtaining point cloud data of a tomato according to a coordinate position of the tomato in a tomato image after image segmentation, and determining an actual position of the tomato according to the point cloud data of the tomato, and specifically comprises the following steps:
acquiring a depth image corresponding to the image to be detected through a depth camera; obtaining point cloud data of the tomato through the coordinate position of the tomato in the tomato image after the image segmentation and the depth image; and determining the actual position of the tomato through the point cloud data of the tomato.
According to a second aspect of the present invention, there is provided a tomato identification and positioning system comprising:
the fruit extraction module is used for acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a tomato image with a background removed;
the fruit separation module is used for cutting the red tomato image after the background is removed through a watershed algorithm to obtain an image-segmented tomato image;
the coordinate determination module is used for carrying out edge detection on the tomato image after the image segmentation and determining the coordinate position of the tomato in the tomato image after the image segmentation;
and the positioning module is used for acquiring the point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
According to a third aspect of the present invention, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, carries out the steps of the method as provided in the first aspect above.
According to a fourth aspect of the present invention, there is provided a tomato identifying and positioning apparatus comprising:
at least one processor; and at least one memory coupled to the processor, wherein:
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the steps of the method as provided in the first aspect above.
The tomato identification and positioning method and the tomato identification and positioning equipment provided by the embodiment of the invention are based on the depth camera and the original RGB image, realize the spatial positioning of the tomatoes, lay a foundation for the quick and accurate picking of tomato fruits, eliminate the influence of uneven light, the occlusion of branches and leaves to the fruits, the shadow caused by the mutual occlusion between the fruits and the like on the fruit identification, realize the identification of the ripe red tomatoes, and reduce the tomato identification cost compared with the traditional binocular camera.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and those skilled in the art can also obtain other drawings according to the drawings without creative efforts.
Fig. 1 is a flowchart of a tomato identifying and positioning method according to an embodiment of the present invention;
fig. 2 is a schematic diagram of a flow of extracting HSV-based red tomato fruits in the method for identifying and positioning tomatoes according to an embodiment of the present invention;
fig. 3 is a schematic diagram of a tomato fruit separation process in the method for identifying and positioning tomatoes according to the embodiment of the present invention;
fig. 4 is a schematic flow chart illustrating calculation of a fruit center and a radius in a tomato identifying and positioning method according to an embodiment of the present invention;
fig. 5 is a schematic flow chart illustrating positioning of an actual position of a tomato in the method for identifying and positioning the tomato according to the embodiment of the present invention;
fig. 6 is a schematic structural diagram of a tomato identifying and positioning system according to an embodiment of the present invention;
fig. 7 is a schematic structural diagram of a tomato identifying and positioning device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1, fig. 1 is a flowchart of a tomato identifying and positioning method according to an embodiment of the present invention, where the method includes:
s1, obtaining an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking to obtain a tomato image with a background removed.
Specifically, after an image to be detected containing tomatoes is obtained, the image to be detected is converted into an HSV image, the converted HSV image is subjected to mask processing, specifically, the steps are that an H channel, an S channel and a V channel are separated from the HSV image, the H channel and the S channel are used as original color information to calculate the color information significance of the image, the V channel is used as original brightness information to calculate the brightness information significance, and a comprehensive significance map is obtained through the weighted color information significance and the brightness information significance. Masking is applied at H, S, V using the selected threshold, and the processed image is then reconciled into an RGB image and masked so that a background-removed tomato image can be obtained.
And S2, cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image.
In particular, during the ripening process of tomatoes, fruits often touch together, and many different image processing algorithms cannot detect two touched tomatoes. In this embodiment, a watershed segmentation method based on a graph form is adopted to perform a watershed algorithm, perform watershed segmentation on an image, retain important contour information, correctly separate connected fruits, realize image segmentation, and obtain a tomato image after image segmentation.
And S3, performing edge detection on the tomato image after image segmentation, and determining the coordinate position of the tomato in the tomato image after image segmentation.
Specifically, after the tomatoes in the tomato image are segmented, further center point and radius data of each tomato need to be obtained for the gripper to grab the tomatoes according to the numerical values, therefore, in this step, the center point and radius of the tomatoes in the image are detected through hough transform.
S4, acquiring point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
Specifically, after the coordinate position of each tomato in the image is obtained, the depth information of the tomato in the image cannot be determined, so that the depth information of the tomato in the actual position needs to be confirmed by a depth camera.
By the method, based on the depth camera and the original RGB image, the spatial positioning of the tomatoes is realized, a foundation is laid for the quick and accurate picking of the tomatoes, the influence of uneven light, the shielding of the fruits by branches and leaves, the shadow caused by the mutual shielding of the fruits and the like on the fruit recognition is eliminated, the recognition of the ripe red tomatoes is realized, and the tomato recognition cost is reduced compared with the traditional binocular camera.
On the basis of the above embodiment, the acquiring an image to be detected including a tomato, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a background-removed tomato image specifically includes:
converting the image to be detected into an HSV image, converting the HSV image into the HSV image, and separating an H channel histogram, an S channel histogram and a V channel histogram from the HSV image;
respectively carrying out mask processing on the H channel histogram, the S channel histogram and the V channel histogram on a preset threshold value, and integrating the H channel histogram, the S channel histogram and the V channel histogram which are subjected to the mask processing into an RGB image;
and (3) carrying out R, G, B masking treatment after the RGB image is closed and opened by a morphological technology, and further integrating to obtain the tomato image after the background is removed.
Specifically, in the background elimination step of the image to be detected, as shown in fig. 2, the HSV color space is composed of three components, namely Hue (Hue), Saturation (Saturation), and brightness (Value). The hue (H) represents different colors, and the value range is 0-360 degrees; the saturation (S) represents the shade of the color, and is usually 0-100%; the brightness (V) represents the brightness of the color, ranging from 0% (black) to 100% (white). Firstly, converting an image to be detected into an HSV image, separating an H channel, an S channel and a V channel from the HSV image, calculating the color information significance of the image by taking the H channel and the S channel as original color information, calculating the brightness information significance by taking the V channel as original brightness information, and obtaining a comprehensive significance map through the weighted color information significance and the brightness information significance. In this embodiment, a histogram is obtained, and on the basis of the histogram, H, S and a threshold value of the V component are selected, defining the region of interest as 0 to 1. In the hue histogram, the value 0 is red, in order to identify red tomatoes, a high hue H threshold value of 0.07 is selected, and the red color is detected by using an Otsu algorithm; similarly, in the saturation S histogram, 0.35 is selected as a saturation threshold; in the luminance V histogram, 0.4 is selected as the luminance threshold. The selected thresholds are then applied separately at H, S, V, masked and integrated into an RGB image. At this time, only red tomatoes remain in the image, then the opening and closing operations are carried out by using the morphological technology, and R, G, B masking treatment is carried out to integrate the operations, so that the red tomato image with the background removed is finally obtained.
On the basis of the above embodiment, the cutting the red tomato image with the background removed by using a watershed algorithm to obtain an image-segmented tomato image specifically includes:
converting the tomato image after the background is removed into a gray image, and calculating the gradient amplitude of the gray image according to a Sobel operator to obtain a gradient image of the gray image;
and cutting the gradient image through a watershed algorithm to obtain an image of the tomato after image segmentation.
In particular, during the ripening process of tomatoes, fruits often touch together, and many different image processing algorithms cannot detect two touched tomatoes. In this embodiment, a watershed algorithm is used to perform watershed algorithm calculation on the tomato image from which the background is removed. In specific implementation, firstly, filtering processing is carried out on a gradient image by utilizing morphological open-close reconstruction operation, and a local extreme value caused by irregular gray disturbance and noise in the image is eliminated; then, performing watershed segmentation on the image, and reserving important contour information; and finally, carrying out similarity measurement by combining an algorithm of normalization product correlation gray matching, and separating a target graph area from the natural background of the image by guiding area combination through a graph-based form template.
On the basis of the above embodiment, the step of obtaining the tomato image after image segmentation by cutting the gradient image through the watershed algorithm further includes:
and performing morphological opening and closing reconstruction operation and watershed calculation on the tomato image after the image segmentation to obtain a tomato image calculated by a secondary watershed algorithm, wherein the tomato image is used as a new tomato image after the image segmentation.
Wherein, after the step of obtaining the tomato image after image segmentation, the method further comprises the following steps: and acquiring the number of the tomatoes in the tomato image after the image segmentation.
Specifically, referring to fig. 3, in the step of separating fruits, a twice watershed algorithm may be adopted to improve the accuracy of fruit segmentation, in a specific implementation, a red tomato image with a background removed is converted into a gray image, Sobel operators are used to calculate gradient amplitudes, horizontal edges and vertical edges are detected, a first watershed algorithm is performed on the gradient image, then morphological open-close reconstruction operation is adopted to remove spots, so that the image is smooth, and in order to obtain a good foreground, a regional extremum of the image is modified to realize detection of a maximum region. To distinguish between the background and foreground of an image, an improved thresholding method is proposed, with background pixels marked black. After Euclidean matrix is used for distance conversion, a ridge line is obtained by using a second watershed algorithm, connected fruits are correctly separated on the basis of the first watershed, image segmentation is realized, and the actual number N of segmentation results is obtained and is used as an input parameter of fruit positioning based on Hough conversion in the next step.
On the basis of the foregoing embodiment, the performing edge detection on the image of the tomato after image segmentation to determine the coordinate position of the tomato in the image of the tomato after image segmentation specifically includes:
taking the number of the tomatoes as circle detection input parameters of Hough transform, and carrying out edge detection on the segmented tomato image to obtain foreground points of the segmented tomato image;
converting the segmented tomato image into an a-b coordinate system from an x-y coordinate system, and obtaining coordinates of a point of a circle center corresponding to the tomato in the segmented tomato image and the radius of the tomato according to the coordinates of the foreground point;
and obtaining the coordinate position of the tomato in the tomato image after the image segmentation according to the coordinate of the point of the circle center and the radius of the tomato.
Specifically, in the calculation process of the tomato coordinates, as shown in fig. 4, the number N of fruits obtained by separating red tomato fruits is used as a circle detection input parameter of hough transform. The principle of detecting the circle center and the radius by applying Hough transform is as follows: firstly, edge detection is carried out on an input image, and boundary points, namely foreground points, are obtained. If a circle is present in the image, its outline must belong to the foreground point. The general equation of the circle is expressed in a different way, and coordinate transformation is performed. From the x-y coordinate system to the a-b coordinate system, a point on the circular boundary in the x-y coordinate system corresponds to a circle in the a-b coordinate system. There are many points on a circular boundary in the x-y coordinate system, and there are many circles corresponding to the a-b coordinate system. Since the points in the original image are all on the same circle, then a, b must also satisfy the equations for all circles in the a-b coordinate system after transformation. The intuitive expression is that the circles corresponding to the plurality of points intersect at one point, and then the intersection point is likely to be the center (a, b). And counting the number of circles at the local intersection points, and taking each local maximum value to obtain the center coordinates (a, b) of the corresponding circle in the original image. Once a circle is detected under a certain r, the value of r is determined accordingly, and finally the coordinates (a, b, r) of the tomato in the image are obtained.
On the basis of the above embodiment, the acquiring point cloud data of the tomato according to the coordinate position of the tomato in the image-segmented tomato image, and determining the actual position of the tomato according to the point cloud data of the tomato specifically includes:
acquiring a depth image corresponding to the image to be detected through a depth camera;
obtaining point cloud data of the tomato through the coordinate position of the tomato in the tomato image after the image segmentation and the depth image;
and determining the actual position of the tomato through the point cloud data of the tomato.
Specifically, the step of obtaining the actual position of the tomato is shown in fig. 5, a Kinect 2.0 camera produced by microsoft corporation is selected to collect information of a tomato plant, a color RGB image of the tomato plant can be collected as an image to be detected, and meanwhile, depth information of the tomato plant can be collected; and matching the RGB image with the depth information, and acquiring the point cloud data of the corresponding red tomatoes according to the coordinates of the red tomato image (a, b, r) obtained by processing the color image. Because the point cloud data comprises the depth information of the image, namely the distance between the RGB-D camera and the fruit is actually separated, the tomato three-dimensional model can be reconstructed according to the acquired point cloud data, and positioning information is provided for tomato picking.
In summary, the tomato identification and positioning method based on the RGB-D camera eliminates the influence of uneven light, the shielding of the branches and the leaves on the fruits, the shadow caused by the mutual shielding of the fruits and the like on the fruit identification, realizes the identification of the mature red tomatoes, has certain significance on the protection of the fruits and the like in the picking process, and meanwhile, based on the depth camera with low price relative to a binocular camera, realizes the rapid identification of the red tomatoes with complex background and mutual adhesion, and lays a foundation for the rapid and accurate picking of the tomato fruits.
Referring to fig. 6, fig. 6 is a schematic structural diagram of a tomato identification and positioning system according to an embodiment of the present invention, where the system includes: a fruit extraction module 61, a fruit separation module 62, a coordinate determination module 63 and a positioning module 64.
The fruit extraction module 61 is configured to acquire an image to be detected including a tomato, convert the image to be detected into an HSV image, perform masking processing on the HSV image, integrate the HSV image into an RGB image, and perform masking processing to obtain a background-removed tomato image.
Specifically, after an image to be detected containing tomatoes is obtained, the image to be detected is converted into an HSV image, the converted HSV image is subjected to mask processing, specifically, the steps are that an H channel, an S channel and a V channel are separated from the HSV image, the H channel and the S channel are used as original color information to calculate the color information significance of the image, the V channel is used as original brightness information to calculate the brightness information significance, and a comprehensive significance map is obtained through the weighted color information significance and the brightness information significance. Masking is applied at H, S, V using the selected threshold, and the processed image is then reconciled into an RGB image and masked so that a background-removed tomato image can be obtained.
The fruit separation module 62 is configured to cut the background-removed red tomato image by using a watershed algorithm to obtain an image-segmented tomato image.
In particular, during the ripening process of tomatoes, fruits often touch together, and many different image processing algorithms cannot detect two touched tomatoes. In this embodiment, a watershed segmentation method based on a graph form is adopted to perform a watershed algorithm, perform watershed segmentation on an image, retain important contour information, correctly separate connected fruits, realize image segmentation, and obtain a tomato image after image segmentation.
The coordinate determination module 63 is configured to perform edge detection on the tomato image after the image segmentation, and determine a coordinate position of a tomato in the tomato image after the image segmentation;
specifically, after the tomatoes in the tomato image are segmented, further center point and radius data of each tomato need to be obtained for the gripper to grab the tomatoes according to the numerical values, therefore, in this step, the center point and radius of the tomatoes in the image are detected through hough transform.
The positioning module 64 is configured to obtain point cloud data of the tomato according to the coordinate position of the tomato in the image of the segmented tomato, and determine the actual position of the tomato according to the point cloud data of the tomato.
Specifically, after the coordinate position of each tomato in the image is obtained, the depth information of the tomato in the image cannot be determined, so that the depth information of the tomato in the actual position needs to be confirmed by a depth camera.
Through the system, based on the depth camera and the original RGB image, the positioning of the tomatoes in the space is realized, the foundation is laid for the quick and accurate picking of the tomatoes, the influences of the shading of the fruits by uneven light, branches and leaves, the shading of the fruits by the fruits and the shadow caused by the mutual shading between the fruits and the like on the fruit identification are eliminated, the identification of the ripe red tomatoes is realized, and the cost of the tomato identification is reduced compared with the traditional binocular camera.
Fig. 7 illustrates a schematic structural diagram of a tomato identification and positioning device, and as shown in fig. 7, the server may include: a processor (processor)710, a memory (memory)730, and a bus 740, wherein the processor 710 and the memory 730 communicate with each other via the bus 740. Processor 710 may call logic instructions in memory 730 to perform the following method: acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a background-removed tomato image; cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image; performing edge detection on the tomato image after the image segmentation, and determining the coordinate position of the tomato in the tomato image after the image segmentation; and acquiring point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
The present embodiments also provide a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions that, when executed by a computer, enable the computer to perform the methods provided by the above-described method embodiments, for example, including: acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a background-removed tomato image; cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image; performing edge detection on the tomato image after the image segmentation, and determining the coordinate position of the tomato in the tomato image after the image segmentation; and acquiring point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
The present embodiments provide a non-transitory computer-readable storage medium storing computer instructions that cause the computer to perform the methods provided by the above method embodiments, for example, including: acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a background-removed tomato image; cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image; performing edge detection on the tomato image after the image segmentation, and determining the coordinate position of the tomato in the tomato image after the image segmentation; and acquiring point cloud data of the tomato according to the coordinate position of the tomato in the tomato image after the image segmentation, and determining the actual position of the tomato according to the point cloud data of the tomato.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (4)
1. A tomato identification and positioning method is characterized by comprising the following steps:
acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a red tomato image with a background removed;
cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image, and acquiring the number of tomatoes in the image-segmented tomato image;
performing edge detection on the tomato image after the image segmentation, and determining the coordinate position of the red tomato in the tomato image after the image segmentation, specifically comprising: taking the number of the tomatoes as circle detection input parameters of Hough transform, and carrying out edge detection on the segmented tomato image to obtain foreground points of the segmented tomato image;
converting the segmented tomato image into an a-b coordinate system from an x-y coordinate system, and obtaining coordinates of a point of a circle center corresponding to the tomato in the segmented tomato image and the radius of the tomato according to the coordinates of the foreground point;
obtaining the coordinate position of the tomato in the tomato image after the image segmentation according to the coordinate of the point of the circle center and the radius of the tomato;
acquiring point cloud data of the red tomatoes according to the coordinate positions of the red tomatoes in the tomato images after the image segmentation, determining the actual positions of the red tomatoes according to the point cloud data of the red tomatoes,
the method comprises the steps of obtaining an image to be detected containing tomatoes, converting the image to be detected into an HSV image, integrating the HSV image into an RGB image after masking the HSV image, and masking the RGB image to obtain a red tomato image with a background removed, and specifically comprises the following steps:
converting the image to be detected into an HSV image, and separating an H channel histogram, an S channel histogram and a V channel histogram from the HSV image;
respectively carrying out mask processing on the H channel histogram, the S channel histogram and the V channel histogram on a preset threshold value, and integrating the H channel histogram, the S channel histogram and the V channel histogram which are subjected to the mask processing into an RGB image;
carrying out R, G, B masking treatment after the RGB image is closed and opened by a morphological technology, and further integrating to obtain a red tomato image with the background removed;
the method comprises the steps of obtaining point cloud data of the red tomatoes according to coordinate positions of the red tomatoes in the tomato images after image segmentation, and determining actual positions of the red tomatoes according to the point cloud data of the red tomatoes, and specifically comprises the following steps:
acquiring a depth image corresponding to the image to be detected through a depth camera;
obtaining point cloud data of the red tomatoes according to the coordinate positions of the red tomatoes in the tomato images after the image segmentation and the depth images;
determining the actual position of the red tomato through the point cloud data of the red tomato;
the background-removed red tomato image is cut by a watershed algorithm to obtain an image-segmented tomato image, and the method specifically comprises the following steps:
converting the red tomato image after the background is removed into a gray image, and calculating the gradient amplitude of the gray image according to a Sobel operator to obtain a gradient image of the gray image;
cutting the gradient image through a watershed algorithm to obtain an image of the tomato after image segmentation;
the step of cutting the gradient image through the watershed algorithm to obtain the tomato image after image segmentation further comprises:
and performing morphological opening and closing reconstruction operation and watershed calculation on the tomato image after the image segmentation to obtain a tomato image calculated by a secondary watershed algorithm, wherein the tomato image is used as a new tomato image after the image segmentation.
2. A tomato identification and positioning system, comprising:
the fruit extraction module is used for acquiring an image to be detected containing tomatoes, converting the image to be detected into an HSV image, after masking the HSV image, integrating the HSV image into an RGB image and masking the RGB image to obtain a red tomato image with a background removed;
the fruit separation module is used for cutting the red tomato image with the background removed through a watershed algorithm to obtain an image-segmented tomato image and acquiring the number of tomatoes in the image-segmented tomato image;
the coordinate determination module is configured to perform edge detection on the tomato image after the image segmentation, and determine a coordinate position of a red tomato in the tomato image after the image segmentation, and specifically includes: taking the number of the tomatoes as circle detection input parameters of Hough transform, and carrying out edge detection on the segmented tomato image to obtain foreground points of the segmented tomato image;
converting the segmented tomato image into an a-b coordinate system from an x-y coordinate system, and obtaining coordinates of a point of a circle center corresponding to the tomato in the segmented tomato image and the radius of the tomato according to the coordinates of the foreground point;
obtaining the coordinate position of the tomato in the tomato image after the image segmentation according to the coordinate of the point of the circle center and the radius of the tomato;
a positioning module for obtaining the point cloud data of the red tomato according to the coordinate position of the red tomato in the tomato image after the image segmentation, determining the actual position of the red tomato according to the point cloud data of the red tomato,
wherein, the fruit extraction module specifically comprises:
converting the image to be detected into an HSV image, and separating an H channel histogram, an S channel histogram and a V channel histogram from the HSV image;
respectively carrying out mask processing on the H channel histogram, the S channel histogram and the V channel histogram on a preset threshold value, and integrating the H channel histogram, the S channel histogram and the V channel histogram which are subjected to the mask processing into an RGB image;
carrying out R, G, B masking treatment after the RGB image is closed and opened by a morphological technology, and further integrating to obtain a red tomato image with the background removed;
the method comprises the steps of obtaining point cloud data of the red tomatoes according to coordinate positions of the red tomatoes in the tomato images after image segmentation, and determining actual positions of the red tomatoes according to the point cloud data of the red tomatoes, and specifically comprises the following steps:
acquiring a depth image corresponding to the image to be detected through a depth camera;
obtaining point cloud data of the red tomatoes according to the coordinate positions of the red tomatoes in the tomato images after the image segmentation and the depth images;
determining the actual position of the red tomato through the point cloud data of the red tomato;
the background-removed red tomato image is cut by a watershed algorithm to obtain an image-segmented tomato image, and the method specifically comprises the following steps:
converting the red tomato image after the background is removed into a gray image, and calculating the gradient amplitude of the gray image according to a Sobel operator to obtain a gradient image of the gray image;
cutting the gradient image through a watershed algorithm to obtain an image of the tomato after image segmentation; the step of cutting the gradient image through the watershed algorithm to obtain the tomato image after image segmentation further comprises:
and performing morphological opening and closing reconstruction operation and watershed calculation on the tomato image after the image segmentation to obtain a tomato image calculated by a secondary watershed algorithm, wherein the tomato image is used as a new tomato image after the image segmentation.
3. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method as claimed in claim 1.
4. Tomato identification and positioning equipment is characterized by comprising:
at least one processor; and at least one memory coupled to the processor, wherein:
the memory stores program instructions executable by the processor, the processor invoking the program instructions to perform the method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810608651.7A CN110599507B (en) | 2018-06-13 | 2018-06-13 | Tomato identification and positioning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810608651.7A CN110599507B (en) | 2018-06-13 | 2018-06-13 | Tomato identification and positioning method and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110599507A CN110599507A (en) | 2019-12-20 |
| CN110599507B true CN110599507B (en) | 2022-04-22 |
Family
ID=68849635
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810608651.7A Active CN110599507B (en) | 2018-06-13 | 2018-06-13 | Tomato identification and positioning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110599507B (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112021016A (en) * | 2020-09-07 | 2020-12-04 | 江苏农林职业技术学院 | Flower and fruit thinning system for strawberries |
| CN112132153B (en) * | 2020-09-22 | 2022-09-09 | 山东大学 | Tomato fruit identification method and system based on clustering and morphological processing |
| CN112529922A (en) * | 2020-11-18 | 2021-03-19 | 南京农业大学 | Method for accurately separating RGB image backgrounds of multicolor blades in open environment |
| CN112580646A (en) * | 2020-12-08 | 2021-03-30 | 北京农业智能装备技术研究中心 | Tomato fruit maturity dividing method and picking robot |
| CN114260895B (en) * | 2021-12-22 | 2023-08-22 | 江苏大学 | Method and system for determining grabbing obstacle avoidance direction of mechanical arm of picking robot |
| CN116778474A (en) * | 2023-06-02 | 2023-09-19 | 河南农业大学 | Intelligent phenotype analyzer for tomato fruits |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101106716A (en) * | 2007-08-21 | 2008-01-16 | 北京大学软件与微电子学院 | A shed image division processing method |
| CN102968782A (en) * | 2012-09-12 | 2013-03-13 | 苏州大学 | Automatic digging method for remarkable objects of color images |
| CN104700404A (en) * | 2015-03-02 | 2015-06-10 | 中国农业大学 | Fruit location identification method |
| CN106067181A (en) * | 2016-05-30 | 2016-11-02 | 湖南生物机电职业技术学院 | Citrusfruit color multi-fractal describes and stage division |
| CN107392963A (en) * | 2017-06-28 | 2017-11-24 | 北京航空航天大学 | A kind of imitative hawkeye moving target localization method for soft autonomous air refuelling |
-
2018
- 2018-06-13 CN CN201810608651.7A patent/CN110599507B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101106716A (en) * | 2007-08-21 | 2008-01-16 | 北京大学软件与微电子学院 | A shed image division processing method |
| CN102968782A (en) * | 2012-09-12 | 2013-03-13 | 苏州大学 | Automatic digging method for remarkable objects of color images |
| CN104700404A (en) * | 2015-03-02 | 2015-06-10 | 中国农业大学 | Fruit location identification method |
| CN106067181A (en) * | 2016-05-30 | 2016-11-02 | 湖南生物机电职业技术学院 | Citrusfruit color multi-fractal describes and stage division |
| CN107392963A (en) * | 2017-06-28 | 2017-11-24 | 北京航空航天大学 | A kind of imitative hawkeye moving target localization method for soft autonomous air refuelling |
Non-Patent Citations (2)
| Title |
|---|
| 图像算法：分水岭分割程序实现;第一江山临;《百度经验》;20171228;第1-8页 * |
| 温室绿熟番茄机器视觉检测方法;李寒等;《农业工程学报》;20170228;第33卷;第328-334页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110599507A (en) | 2019-12-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110599507B (en) | Tomato identification and positioning method and system | |
| Dias et al. | Multispecies fruit flower detection using a refined semantic segmentation network | |
| Dorj et al. | An yield estimation in citrus orchards via fruit detection and counting using image processing | |
| Liu et al. | A method of segmenting apples at night based on color and position information | |
| Xiong et al. | The recognition of litchi clusters and the calculation of picking point in a nocturnal natural environment | |
| Zhao et al. | On-tree fruit recognition using texture properties and color data | |
| Zhuang et al. | Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios | |
| Tong et al. | Skewness correction and quality evaluation of plug seedling images based on Canny operator and Hough transform | |
| Devi et al. | Image processing system for automatic segmentation and yield prediction of fruits using open CV | |
| Li et al. | An improved binocular localization method for apple based on fruit detection using deep learning | |
| Feng et al. | A separating method of adjacent apples based on machine vision and chain code information | |
| Revathi et al. | Homogenous segmentation based edge detection techniques for proficient identification of the cotton leaf spot diseases | |
| Raut et al. | Review on leaf disease detection using image processing techniques | |
| Changhui et al. | Overlapped fruit recognition for citrus harvesting robot in natural scenes | |
| Choudhary et al. | Image processing algorithm for fruit identification | |
| Tran et al. | Automatic dragon fruit counting using adaptive thresholds for image segmentation and shape analysis | |
| Sibi Chakkaravarthy et al. | Automatic leaf vein feature extraction for first degree veins | |
| CN107239761B (en) | Fruit tree branch pulling effect evaluation method based on skeleton angular point detection | |
| CN111369497B (en) | Walking type tree fruit continuous counting method and device | |
| Mohammed Amean et al. | Automatic plant branch segmentation and classification using vesselness measure | |
| Huang et al. | Mango surface defect detection based on HALCON | |
| Nejati et al. | Kiwifruit detection in challenging conditions | |
| Kanwal et al. | Overlapped apple fruit yield estimation using pixel classification and hough transform | |
| Yang et al. | Cherry recognition based on color channel transform | |
| Shao et al. | A novel tree trunk recognition approach for forestry harvesting robot |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |