CN118172414A - Fruit and vegetable stem positioning method and system - Google Patents
Fruit and vegetable stem positioning method and system Download PDFInfo
- Publication number
- CN118172414A CN118172414A CN202211574927.7A CN202211574927A CN118172414A CN 118172414 A CN118172414 A CN 118172414A CN 202211574927 A CN202211574927 A CN 202211574927A CN 118172414 A CN118172414 A CN 118172414A
- Authority
- CN
- China
- Prior art keywords
- fruit
- vegetable
- image
- stems
- stem
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 235000012055 fruits and vegetables Nutrition 0.000 title claims abstract description 299
- 238000000034 method Methods 0.000 title claims abstract description 66
- 238000001514 detection method Methods 0.000 claims abstract description 74
- 235000013399 edible fruits Nutrition 0.000 claims abstract description 73
- 238000012545 processing Methods 0.000 claims abstract description 42
- 238000004590 computer program Methods 0.000 claims description 15
- 230000011218 segmentation Effects 0.000 claims description 7
- 230000009466 transformation Effects 0.000 claims description 5
- 241000132456 Haplocarpha Species 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 16
- 240000003040 Solanum lycopersicum var. cerasiforme Species 0.000 description 14
- 235000003953 Solanum lycopersicum var cerasiforme Nutrition 0.000 description 13
- 238000013135 deep learning Methods 0.000 description 10
- 238000001914 filtration Methods 0.000 description 7
- 230000000694 effects Effects 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 6
- 238000012549 training Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 235000013311 vegetables Nutrition 0.000 description 4
- 238000005286 illumination Methods 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 230000006872 improvement Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012216 screening Methods 0.000 description 2
- 244000119012 Bequaertiodendron magalismontanum Species 0.000 description 1
- 241000167854 Bourreria succulenta Species 0.000 description 1
- 241000196324 Embryophyta Species 0.000 description 1
- 235000007688 Lycopersicon esculentum Nutrition 0.000 description 1
- 240000003768 Solanum lycopersicum Species 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 235000019693 cherries Nutrition 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000009191 jumping Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000002689 soil Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a fruit and vegetable stem positioning method and system, wherein the method comprises the following steps: inputting each first image into a first target detection model to obtain a first identification frame corresponding to the fruit and vegetable stems in each first image; obtaining a target area according to the first identification frames corresponding to the fruit stems and the identification frames of the fruits; performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems; and determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems. The system performs the method. The invention can effectively solve the problems of false detection and missing detection of the fruit and vegetable stems in a complex natural environment, thereby realizing the positioning of the fruit and vegetable stems, improving the accuracy of identifying fruit and vegetable picking points and further improving the fruit and vegetable picking efficiency.
Description
Technical Field
The invention relates to the technical field of image processing, in particular to a fruit and vegetable stem positioning method and system.
Background
The picking operation of fruits and vegetables (such as cherry tomatoes, cherry tomatoes and the like) with huge yield always faces the problems of labor shortage and the like, and the mechanized picking of the agricultural robot has important significance for large-scale picking. In order to ensure the high efficiency and accuracy of the hand-eye cooperative operation of the picking robot, the accuracy and speed of the identification and positioning of the fruit and vegetable stems are the premise and the basis, namely whether the fruit picking points are accurately and rapidly identified and positioned can directly influence the picking of the follow-up tomatoes.
The fruit and vegetable stem identifying and positioning method can be divided into two parts, wherein one part is deep learning detection, and the other part is image processing operation. The deep learning detection algorithm is mainly divided into two stages and a single stage, wherein the two stages refer to the steps of extracting features and detecting, namely firstly selecting an area and then classifying; and "single stage" is to combine the two steps into one step. Although the detection accuracy of the two-stage algorithm is slightly higher, the detection speed is insufficient, and the real-time operation requirement of the picking robot is not met, so that a target detection algorithm (for example, a YOLO series single-stage algorithm) with high speed and relatively high accuracy is adopted.
However, in a complex natural environment, particularly when branches, stems and leaves near fruits are numerous, the accuracy of identifying and positioning the stems of fruits and vegetables by deep learning or image processing is very low, and the phenomena of omission and false detection are easy to occur. The reduction of the fruit and vegetable stem identification positioning accuracy means that fruit and vegetable picking points are inaccurate, so that fruit and vegetable picking operation is greatly interfered, and the fruit and vegetable picking efficiency is low.
Disclosure of Invention
The fruit and vegetable stem positioning method and system provided by the invention are used for solving the problems that in the prior art, under the complex natural environment, particularly when the leaves, stems and leaves near fruits are greatly interfered, the fruit and vegetable stems cannot be accurately identified and positioned by using a deep learning algorithm or image processing, so that the fruit picking points are inaccurate and the fruit and vegetable picking efficiency is low, and can effectively solve the problems of false detection and omission detection of the fruit and vegetable stems under the complex natural environment, thereby realizing the positioning of the fruit and vegetable stems, improving the accuracy of the identification of the fruit and vegetable picking points and further improving the fruit and vegetable picking efficiency.
The invention provides a fruit and vegetable stem positioning method, which comprises the following steps:
Inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
Performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
And determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
According to the fruit and vegetable stem positioning method provided by the invention, the acquisition mode of the interested region of the fruit and vegetable stem comprises the following steps:
Inputting each second image into the second target detection model to obtain a second identification frame corresponding to the fruit and vegetable stems in each second image;
determining the average width and the average height of the fruit and vegetable stems according to the second identification frames of the fruit and vegetable stems;
And determining the interested region of the fruit and vegetable stems according to the identification frame of the fruit, the average width of the fruit and vegetable stems and the average height of the fruit and vegetable stems.
According to the fruit and vegetable stem positioning method provided by the invention, a target area is obtained according to the first identification frame and the fruit identification frame corresponding to each fruit and vegetable stem, and the method comprises the following steps:
under the condition that one fruit and vegetable stem is included in the first image, determining a target area according to a first identification frame of the fruit and vegetable stem;
And under the condition that the first image comprises a plurality of fruit and vegetable stems, determining the target area according to the distance between each first center point and each second center point, wherein the first center point is the center point of a first identification frame of the fruit and vegetable stems, and the second center point is the center point of the identification frame of the fruits.
According to the fruit and vegetable stem positioning method provided by the invention, when a plurality of fruit and vegetable stems are included in the first image, the determining the target area according to the distance between each first center point and each second center point includes:
determining a target center point with the shortest distance between the first center point and the second center point according to the distance between the first center point and the second center point;
And determining the target area according to a third identification frame of the fruit and vegetable stems corresponding to the target center point.
According to the fruit and vegetable stem positioning method provided by the invention, the image processing is carried out on the third image corresponding to the target area, and the mass center of the fruit and vegetable stem is obtained, which comprises the following steps:
gray processing is carried out on the third image, and a fourth image is obtained;
Processing the fourth image based on wavelet transformation to obtain a fifth image;
removing background pixel points in the fifth image to obtain a sixth image;
obtaining a threshold value of the sixth image based on a preset self-adaptive threshold segmentation algorithm;
Binarizing the sixth image based on the threshold value to obtain a seventh image;
removing noise points in the seventh image to obtain a target connected domain in the seventh image;
and determining the mass center of the fruit and vegetable stems according to the mass center of the target connected domain.
According to the fruit and vegetable stem positioning method provided by the invention, the positioning information of the fruit and vegetable stem on the second image is determined according to the mass center of the fruit and vegetable stem, and the method comprises the following steps:
Converting the two-dimensional coordinates of the barycenter of the fruit and vegetable stems in the third image into three-dimensional coordinates of a camera;
And determining the positioning information of the fruit and vegetable stems on the second image according to the three-dimensional coordinates of the camera.
The invention also provides a fruit and vegetable stem positioning system, which comprises: the device comprises a first acquisition module, a second acquisition module, a third acquisition module and a positioning module;
the first acquisition module is used for inputting each first image into a first target detection model to obtain a first identification frame corresponding to the fruit and vegetable stems in each first image, the first images are obtained by cutting out second images in a data set according to the interested areas of the fruit and vegetable stems, the data set comprises a plurality of second images, and the first target detection model is determined according to the trained second target detection model;
the second obtaining module is used for obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
The third acquisition module is used for performing image processing on a third image corresponding to the target area to acquire the mass center of the fruit and vegetable stems;
and the positioning module is used for determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
The invention also provides electronic equipment, which comprises a processor and a memory storing a computer program, wherein the processor realizes the fruit and vegetable stem positioning method according to any one of the above when executing the program.
The invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which when executed by a processor implements a fruit and vegetable stem positioning method as described in any of the above.
The invention also provides a computer program product comprising a computer program which when executed by a processor implements a method of positioning fruit and vegetable stems as described in any of the above.
The fruit and vegetable stem positioning method and system provided by the invention fully consider the interference of redundant branches and stems in the actual complex environment, and combine the image processing technology and the deep learning method, so that the problems of false detection and missing detection of the fruit and vegetable stems in the complex natural environment can be effectively solved, the positioning of the fruit and vegetable stems is realized, the accuracy of the recognition of fruit and vegetable picking points is improved, and the fruit and vegetable picking efficiency is further improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a fruit and vegetable stem positioning method provided by the invention;
FIG. 2 is a second flow chart of the fruit and vegetable stem positioning method according to the present invention;
FIG. 3 is an exemplary diagram of calculating a region of interest ROI of a fruit and vegetable stem of cherry tomato provided by the present invention;
fig. 4 is a third schematic flow chart of the fruit and vegetable stem positioning method provided by the invention;
FIG. 5 is an exemplary diagram of a third identification box provided by the present invention for identifying and cropping fruit and vegetable stems at a target center point;
FIG. 6 is an exemplary diagram of the present invention for achieving the shortest centroid distance constraint of fruit and vegetable stems and fruits;
FIG. 7 is a flow chart of a fruit and vegetable stem positioning method provided by the invention;
FIG. 8 is an exemplary diagram of image processing of a third image provided by the present invention;
fig. 9 is a schematic structural view of the fruit and vegetable stem positioning system provided by the invention;
Fig. 10 is a schematic diagram of the physical structure of the electronic device provided by the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
According to the fruit and vegetable stem positioning method provided by the invention, the fruits and the fruit and vegetable stems in the images in the camera shooting dataset are identified for the first time by utilizing YOLOv, the width and the height of an average stem identification frame are calculated, and based on the detected identification frame of the fruits and the average width and the average height of the fruit and vegetable stems, the images are cut to obtain the interested Region (ROI) of the corresponding stems of the fruits (namely the fruit and vegetable stems), and the ROI is detected for the second time YOLOv; filtering redundant interference stems by the constraint that the distances between the fruit stems and the barycenters of the fruits are shortest, and cutting out images to obtain a unique stem identification frame, namely a target area; and finally, performing image processing on a target area to obtain the mass center of the fruit and vegetable stems, converting the mass center of the fruit and vegetable stems into three-dimensional coordinates of a camera and displaying the three-dimensional coordinates on an original image in a data set to obtain positioning information of the fruit and vegetable stems, wherein the positioning information is specifically realized as follows:
Fig. 1 is a schematic flow chart of a fruit and vegetable stem positioning method provided by the invention, and as shown in fig. 1, the method comprises the following steps:
Step 110, inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Step 120, obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
130, performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
and 140, determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
It should be noted that, the execution subject of the above method may be a computer device.
Optionally, the first image may be specifically obtained by clipping a second image obtained by shooting a fruit and vegetable plant based on a depth camera according to a region of interest ROI of a fruit and vegetable stem, and a set formed by the second images is called a dataset, where the dataset may specifically include a plurality of second images. The fruit and vegetable stems can be particularly stems of cherry tomatoes or stems of cherries, and the invention is not particularly limited to the above.
The first recognition frame may be specifically a recognition frame of the fruit and vegetable stems in the first images, and more specifically may be obtained by inputting each first image into the first object detection model.
The first target detection model may specifically be a trained second target detection model obtained after training the second target detection model using the dataset, and the second target detection model may specifically be a YOLOv model, specifically:
Calibrating the first images in the data set, and dividing each first image in the data set into two major categories, namely fruits and fruit and vegetable stalks;
inputting the data set into a YOLOv model for training until the loss function of the YOLOv model converges (representing that the YOLOv model is trained), and obtaining weight parameters;
The YOLOv model which is trained is used as a first target detection model.
The YOLOv model is widely applied after being proposed in 2020, has a good detection effect, and has a large improvement space comprising an input/output part and a network structure. The image processing part processes the binary image of the fruit by using a color channel method, an OTSU method and the like to extract a detection target area. The image processing can be used for preprocessing the data set on one hand and carrying out post-processing and correction on the detection result on the other hand. For fruits under normal illumination images, the success rate of image processing is higher, but improvement is needed when illumination is uneven or light intensity is insufficient.
For example, the data set is marked as two major categories of fruits and fruit and vegetable stems, the data set is divided, the YOLOv model is adopted for iterative training, and the weight of the YOLOv model obtained in the last training is taken as the weight parameter of the first target detection model on the assumption that the loss function of the YOLOv model converges after 300 iterations.
The fruit identification frame may be specifically an identification frame of a fruit in the second image, and more specifically may be obtained by inputting each second image into the second object detection model.
And obtaining a target area according to the obtained first identification frames of the fruit and vegetable stems and the identification frames of the fruits in each first image, wherein the target area can specifically be an area corresponding to the identification frames comprising the unique fruit and vegetable stems.
And performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems, wherein the image processing can specifically comprise graying processing and depth filtering processing.
The image coordinates of the fruit and vegetable stems can be obtained according to the barycenter of the fruit and vegetable stems, and the image coordinates are converted into coordinates under a camera coordinate system, so that the positioning information of the fruit and vegetable stems on the second image can be obtained.
The fruit and vegetable stem positioning method provided by the invention fully considers the interference of redundant branches and stems in the actual complex environment, combines the image processing technology with the deep learning method, and can effectively solve the problems of false detection and missing detection of the fruit and vegetable stems in the complex natural environment, thereby realizing the positioning of the fruit and vegetable stems, improving the accuracy of the recognition of fruit and vegetable picking points and further improving the fruit and vegetable picking efficiency.
Further, in an embodiment, the method for obtaining the region of interest of the fruit and vegetable stalk may specifically include:
Inputting each second image into the second target detection model to obtain a second identification frame corresponding to the fruit and vegetable stems in each second image;
determining the average width and the average height of the fruit and vegetable stems according to the second identification frames of the fruit and vegetable stems;
And determining the interested region of the fruit and vegetable stems according to the identification frame of the fruit, the average width of the fruit and vegetable stems and the average height of the fruit and vegetable stems.
Alternatively, the second recognition frame may be specifically a recognition frame of the fruit and vegetable stems in the second image, and more specifically may be obtained by passing each second image to the second object detection model.
And counting the width and the height of the second identification frames of the fruit and vegetable stems in each second image, calculating the average width and the average height of the second identification frames of the fruit and vegetable stems, and taking the average width and the average height of the fruit and vegetable stems as the average width and the average height of the fruit and vegetable stems.
Obtaining a region of interest (ROI) of the fruit and vegetable stalks according to the identification frame of the fruit, the average width of the fruit and vegetable stalks and the average height of the fruit and vegetable stalks, which are obtained by inputting the second image into the second target detection model, specifically:
And when the difference between the left upper corner coordinate of the identification frame of the fruit and the average width of the fruit and vegetable stems is larger than 0, the difference between the right lower corner coordinate of the identification frame of the fruit and the average height of the fruit and vegetable stems is larger than 0, the sum of the left upper corner coordinate of the identification frame of the fruit and the average width of the fruit and vegetable stems is smaller than the width of the second image, and the sum of the right lower corner coordinate of the identification frame of the fruit and the average height of the fruit and vegetable stems is smaller than the height of the second image, obtaining the left upper corner coordinate of the region of interest ROI according to the left upper corner coordinate of the identification frame of the fruit, the average width of the fruit and vegetable stems and the average height of the fruit and vegetable stems, obtaining the right lower corner coordinate of the region of interest ROI according to the right lower corner coordinate of the right upper corner coordinate of the region of interest ROI and the average height of the fruit and vegetable stems, and taking the right lower corner coordinate of the region of interest ROI as the region of interest.
For example, fig. 2 is a second flow chart of the fruit and vegetable stem positioning method provided by the invention, as shown in fig. 2, including:
Step 1, acquiring an image to obtain a data set, and preprocessing the data set, wherein the preprocessing can specifically comprise image enhancement and expansion (namely, rotation, translation, color and other change processing on the image) of the image in the data set;
Step 2, calibrating and dividing a data set;
step 3, training YOLOv the model until the loss function converges to obtain weight parameters;
step 4, counting the width and the height of the second identification frame of the fruit and vegetable stems, and calculating the average width of the fruit and vegetable stems and the average height (avg_stem_w and avg_stem_h) of the fruit and vegetable stems;
Step 5, judging whether fruits (such as cherry tomatoes) are contained in the second image, if yes, carrying out step 6, and if no, ending;
Step 6, obtaining the left upper corner coordinate (x 1,y1) and the right lower corner coordinate (x 2,y2) of the identification frame of the fruit, and calculating the interested region of the fruit and vegetable stems;
Step 7, judging that x 1-avg_stem_w＞0,y1-avg_stem_h＞0,x2+avg_stem_w＜W,y2 +avg_stem_h is smaller than H, if not, replacing the left calculated amount with a boundary value, for example, replacing the left calculated amount x 1 -avg_stem_w with a boundary value 0, replacing the left calculated amount y 1 -avg_stem_h with a boundary value 0, replacing the left calculated amount x 2 +avg_stem_w with W, replacing the left calculated amount y 2 +avg_stem_h with H, and if step 8 is satisfied, wherein W represents the width of the second image and H represents the height of the second image;
Step 8, calculating to obtain the upper left corner coordinate of the interested region ROI of the fruit and vegetable stems as (x 1-avg_stem_w,y1 -avg_stem_h2), and the lower right corner coordinate of the interested region ROI of the fruit and vegetable stems as (x 2+avg_stem_w,y2 +avg_stem_h2);
and 9, cutting according to the region of interest ROI of the fruit and vegetable stems.
Fig. 3 is an exemplary diagram of calculating a region of interest ROI of a fruit and vegetable stem of cherry tomato, as shown in fig. 3, a represents inputting an image to be detected, namely, an image of a cherry tomato plant photographed by a depth camera, counting the width and the height of a second identification frame of all fruit and vegetable stems through YOLOv, and calculating an average width (avg_stem_w) and an average height (avg_stem_h) of the fruit and vegetable stems; b represents YOLOv the identified image, the coordinates of the left upper corner and the right lower corner of the identification frame of the cherry tomato are (x 1,y1),(x2,y2) respectively, and the interested area of the cherry tomato corresponding to the fruit and vegetable stems is calculated by combining the average width (avg_stem_w) and the average height (avg_stem_h) of the fruit and vegetable stems; c represents a determined example graph of the region of interest, and coordinates of the region of interest of the fruit and vegetable stems are calculated according to the distribution positions of the fruit and vegetable stems as follows: left upper corner coordinates (x 1-avg_stem_w,y1 -avg_stem_h2), right lower corner coordinates (x 2+avg_stem_w,y2 +avg_stem_h2).
The fruit and vegetable stem positioning method provided by the invention fully considers the interference of redundant branches and stems in the actual complex natural environment, combines the image processing technology with the YOLOv deep learning method to obtain the region of interest of the fruit and vegetable stems, and lays a foundation for obtaining the target region based on the region of interest and positioning the fruit stems.
Further, in an embodiment, the obtaining the target area according to the first identification frame corresponding to each fruit and vegetable stem and the identification frame of the fruit may specifically include:
under the condition that one fruit and vegetable stem is included in the first image, determining a target area according to a first identification frame of the fruit and vegetable stem;
And under the condition that the first image comprises a plurality of fruit and vegetable stems, determining the target area according to the distance between each first center point and each second center point, wherein the first center point is the center point of a first identification frame of the fruit and vegetable stems, and the second center point is the center point of the identification frame of the fruits.
Optionally, because the condition that one or more fruit and vegetable stems are included in the first image obtained by cutting the second image according to the region of interest ROI of the fruit and vegetable stems may exist in the first image, the existence of the redundant fruit and vegetable stems may interfere with the positioning of the final fruit and vegetable stems, and affect the positioning accuracy of the fruit and vegetable stems.
In the case where only one fruit and vegetable stem is included in the first image, since the fruit and vegetable stem in the first image is unique, the region corresponding to the first recognition frame of the fruit and vegetable stem may be taken as the target region.
For the case that the first image includes a plurality of fruit and vegetable stems, it is necessary to find the center point of the first recognition frame of each fruit and vegetable stem, that is, the first center point, and the center point of the recognition frame of the fruit, that is, the second center point, and calculate the distance between each first center point and the second center point, and find the corresponding target area according to the distance.
Further, in an embodiment, in a case where the first image includes a plurality of fruit and vegetable stems, the determining the target area according to the distance between each first center point and each second center point may specifically include:
determining a target center point with the shortest distance between the first center point and the second center point according to the distance between the first center point and the second center point;
And determining the target area according to a third identification frame of the fruit and vegetable stems corresponding to the target center point.
Optionally, a target center point with the shortest distance between the first center point and the second center point is found from the first center points, and the area corresponding to the third recognition frame of the fruit and vegetable stems corresponding to the target center point is taken as a target area.
For example, fig. 4 is a third flow chart of the fruit and vegetable stem positioning method provided by the present invention, as shown in fig. 4, including:
Step 11, obtaining a first image obtained after cutting a second image according to the region of interest (ROI);
step 12, identifying the fruit and vegetable stems in the first image by using the trained YOLOv;
Step 13, judging whether the interested region ROI contains fruit and vegetable stems, if not, ending; if yes, go to step 14;
Step 14, judging whether the fruit and vegetable stems in the region of interest ROI are unique (if the identification frames of the fruit and vegetable stems are unique, otherwise, the fruit and vegetable stems are not unique), if not, directly jumping to the step 16, and if yes, performing a step 15;
Step 15, calculating the distances between the center point of the first recognition frame of the fruit and vegetable stems (namely, the first center point, which is assumed to be (x_stem_center, y_stem_center)) and the center point of the fruit recognition frame (namely, the second center point), screening a target center point with the shortest distance to the fruit recognition frame, taking the region corresponding to the recognition frame (namely, the third recognition frame) of the fruit and vegetable stems corresponding to the target center point as a target region, taking the second center point as the mass center of the fruit, and converting the two-dimensional pixel coordinate corresponding to the mass center of the fruit into the three-dimensional camera coordinate under the camera coordinate system to obtain the positioning information of the mass center of the fruit in the second image;
and step 16, cutting the first image into an image with the size of a third identification frame of the fruit and vegetable stems corresponding to the target center point, namely a third image.
Fig. 5 is an exemplary diagram of a third recognition frame for recognizing and cropping a fruit and vegetable stem at a target center point, as shown in fig. 5, d represents a cropped region of interest ROI, where the length of the region of interest ROI is x 2-x1 +2×avg_stem_w, the width of the region of interest ROI is avg_stem_h, and recognition of YOLOv (inputting the first image into the first target detection model) is performed for the second time on the first image corresponding to the region of interest ROI; e represents YOLOv the identified image, the first identification frame of the identified fruit and vegetable stems is in the dashed frame, two fruit and vegetable stems are identified in the example diagram, so that the first identification frames of the two fruit and vegetable stems are available, and then screening is carried out by utilizing the shortest constraint of the distance between the first identification frames of the fruits of cherry tomatoes and the first identification frames of the fruits of cherry tomatoes; f represents an example diagram of a third identification frame of a unique fruit and vegetable stem corresponding to the screened and determined target center point, selects the fruit and vegetable stem with the shortest distance from the center of mass of the fruit, cuts the identification frame (namely the stem identification frame) of the fruit and vegetable stem, and obtains a third image corresponding to the target area.
FIG. 6 is an exemplary diagram for implementing the constraint condition that the barycenter distance between the fruit and vegetable stems is the shortest, as shown in FIG. 6, g represents the recognition result of the first object detection model (trained YOLOv); FIG. h is a schematic diagram showing the distance between the center of the fruit and vegetable stem identification frame and the center of the fruit identification frame after the center of the fruit and vegetable stem and the center of mass of the fruit are determined; i represents the identification frame (namely a third identification frame) of the unique fruit and vegetable stems after the shortest distance between the identification frame and the center of the fruit identification frame is calculated and screened, and a third image corresponding to the target area is obtained after cutting according to the third identification frame.
The fruit and vegetable stem positioning method provided by the invention fully considers the interference of redundant branches and stems in the actual complex natural environment, combines the image processing technology with the YOLOv deep learning method, utilizes the position information between the fruit and vegetable stems and the fruits to restrict, gradually reduces the interested area of the fruit and vegetable stems to obtain the target area, improves the detection precision of the fruit and vegetable stems, and obtains better fruit and vegetable stem positioning effect.
Further, in an embodiment, the image processing of the third image corresponding to the target area to obtain the centroid of the fruit and vegetable stalk may specifically include:
gray processing is carried out on the third image, and a fourth image is obtained;
Processing the fourth image based on wavelet transformation to obtain a fifth image;
removing background pixel points in the fifth image to obtain a sixth image;
obtaining a threshold value of the sixth image based on a preset self-adaptive threshold segmentation algorithm;
Binarizing the sixth image based on the threshold value to obtain a seventh image;
removing noise points in the seventh image to obtain a target connected domain in the seventh image;
and determining the mass center of the fruit and vegetable stems according to the mass center of the target connected domain.
Further, in an embodiment, the determining, according to the centroid of the fruit and vegetable stem, the positioning information of the fruit and vegetable stem on the second image may specifically include:
Converting the two-dimensional coordinates of the barycenter of the fruit and vegetable stems in the third image into three-dimensional coordinates of a camera;
And determining the positioning information of the fruit and vegetable stems on the second image according to the three-dimensional coordinates of the camera.
Optionally, performing image processing on a third image corresponding to the target area, specifically, using a 2G-R-B channel transform on the RGB color space to separate fruits in the soil from fruits in the background; removing interference point noise by wavelet transformation; filtering by using a depth value fed back by a depth camera to remove background interference; and performing picture threshold segmentation based on an OSTU binarization of a preset adaptive threshold segmentation algorithm.
And extracting the mass center of the obtained target connected domain (block), converting the two-dimensional pixel coordinates into camera three-dimensional coordinates by combining the mass center depth information, displaying the mass center of the fruit and vegetable stems on a second image through an Opencv function, and obtaining a fruit and vegetable stem identification positioning image, wherein the positioning image comprises positioning information of the fruit and vegetable stems.
Specifically, fig. 7 is a fourth flow chart of the fruit and vegetable stem positioning method provided by the invention, as shown in fig. 7, including:
Step 21, obtaining an image of a target area corresponding to a third identification frame of the cut unique stem fruit and vegetable stem to obtain a third image;
Step 22, graying the original image by using a 2R-G-B method to obtain a gray image, namely a fourth image;
step 23, filtering and denoising the fourth image by utilizing wavelet transformation, reducing the influence of uneven illumination and the like on color extraction, and obtaining a fifth image;
Step 24, filtering out background pixels with large depth (for example, the pixel depth value is greater than a preset threshold value) in the fifth image by combining the pixel depth value fed back by the depth camera, so as to remove the background pixels in the fifth image and obtain a sixth image;
step 25, a sixth image threshold is obtained by using a preset adaptive threshold segmentation algorithm OTSU algorithm;
Step 26, binarizing the sixth image by using a threshold value to obtain a binary image, namely a seventh image, and removing noise points of the small connected domain to obtain a target connected domain in the seventh image;
Step 27, calculating the mass center of the target connected domain, and taking the mass center as the mass center of the fruit and vegetable stems;
step 28, converting the two-dimensional pixel coordinates of the centroid of the fruit and vegetable stem into camera three-dimensional coordinates, and displaying the centroid of the fruit and vegetable stem on the second image.
For example, fig. 8 is an exemplary diagram of image processing performed on a third image, as shown in fig. 8, a series of image processing operations are performed on a third identification frame of a unique fruit and vegetable stem to obtain a centroid of the fruit and vegetable stem, j represents the inputted third identification frame of the unique fruit and vegetable stem, k is a graying effect diagram obtained through a 2R-G-B algorithm and a wavelet transform filtering denoising method, m is obtained after a threshold value is obtained through an OSTU method (m is one of two binarizing effect diagrams, n is the second of two binarizing effect diagrams), and background pixels with large depth are filtered out by combining a pixel depth value fed back by a depth camera to obtain o after depth filtering denoising. And p is the mass center of the block-shaped target connected domain, and the camera three-dimensional coordinates corresponding to the mass center of the transformed fruit and vegetable stems are displayed on the second image.
The feasibility of the fruit and vegetable stem positioning method provided by the invention is verified by using a depth camera. In the window of detection, can accurately discern cherry tomato and corresponding fruit vegetables stem, including the identification frame of ripe cherry tomato, the identification frame of fruit vegetables stem, the region of interest ROI and the barycenter of cherry tomato's fruit and the three-dimensional coordinates that the barycenter of fruit vegetables stem corresponds to the category and the confidence that mark out is fruit vegetables.
The fruit and vegetable stem positioning method combines the image processing technology and the YOLOv deep learning method, utilizes the position information and the depth information to restrict, gradually reduces the interested area of the fruit and vegetable stem and performs segmentation positioning and centroid extraction on the fruit and vegetable stem in the target area, and provides a more effective solution to the difficult problem of accurate identification and positioning of the fruit and vegetable stem in a complex natural environment, thereby improving the detection precision of the fruit and vegetable stem, obtaining a better fruit and vegetable stem positioning effect and laying a foundation for the efficient operation of a follow-up picking robot.
The fruit and vegetable stem positioning system provided by the invention is described below, and the fruit and vegetable stem positioning system described below and the fruit and vegetable stem positioning method described above can be referred to correspondingly.
Fig. 9 is a schematic structural diagram of a fruit and vegetable stem positioning system provided by the invention, as shown in fig. 9, including:
A first acquisition module 910, a second acquisition module 911, a third acquisition module 912, and a positioning module 913;
The first obtaining module 910 is configured to input each first image to a first target detection model, to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, where the first image is obtained by clipping a second image in a dataset according to a region of interest of the fruit and vegetable stem, and the dataset includes a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
The second obtaining module 911 is configured to obtain a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, where the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
the third obtaining module 912 is configured to perform image processing on a third image corresponding to the target area to obtain a centroid of the fruit and vegetable stem;
The positioning module 913 is configured to determine positioning information of the fruit and vegetable stems on the second image according to the centroid of the fruit and vegetable stems.
The fruit and vegetable stem positioning system provided by the invention fully considers the interference of redundant branches and stems in the actual complex environment, combines the image processing technology with the deep learning method, and can effectively solve the problems of false detection and missing detection of fruit and vegetable stems in the complex natural environment, thereby realizing the positioning of the fruit and vegetable stems, improving the accuracy of the recognition of fruit and vegetable picking points and further improving the fruit and vegetable picking efficiency.
Fig. 10 is a schematic physical structure of an electronic device according to the present invention, as shown in fig. 10, the electronic device may include: a processor 1010, a communication interface communication interface 1011, a memory 1012 and a bus 1013, wherein the processor 1010, the communication interface 1011 and the memory 1012 perform communication with each other through the bus 1013. The processor 1010 may call logic instructions in the memory 1012 to perform the following methods:
Inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
Performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
And determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
Further, the logic instructions in the memory described above may be implemented in the form of software functional units and stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer power supply screen (which may be a personal computer, a server, or a network power supply screen, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-only Memory (ROM), a random-access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Further, the present invention discloses a computer program product comprising a computer program stored on a non-transitory computer readable storage medium, the computer program comprising program instructions which, when executed by a computer, are capable of executing the fruit and vegetable stem positioning method provided by the above method embodiments, for example, comprising:
Inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
Performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
And determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
In another aspect, the present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor is implemented to perform the fruit and vegetable stem positioning method provided in the above embodiments, for example, including:
Inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
Performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
And determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
The system embodiments described above are merely illustrative, wherein the elements illustrated as separate elements may or may not be physically separate, and the elements shown as elements may or may not be physical elements, may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course may be implemented by means of hardware. Based on this understanding, the foregoing technical solution may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a ROM/RAM, a magnetic disk, an optical disk, etc., including several instructions for causing a computer power screen (which may be a personal computer, a server, or a network power screen, etc.) to perform the method described in the various embodiments or some parts of the embodiments.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The fruit and vegetable stem positioning method is characterized by comprising the following steps:
Inputting each first image into a first target detection model to obtain a first identification frame corresponding to a fruit and vegetable stem in each first image, wherein the first images are obtained by cutting a second image in a data set according to a region of interest of the fruit and vegetable stem, the data set comprises a plurality of second images, and the first target detection model is determined according to a trained second target detection model;
Obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
Performing image processing on a third image corresponding to the target area to obtain the mass center of the fruit and vegetable stems;
And determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
2. The fruit and vegetable stalk positioning method according to claim 1 wherein the method for obtaining the region of interest of the fruit and vegetable stalk comprises:
Inputting each second image into the second target detection model to obtain a second identification frame corresponding to the fruit and vegetable stems in each second image;
determining the average width and the average height of the fruit and vegetable stems according to the second identification frames of the fruit and vegetable stems;
And determining the interested region of the fruit and vegetable stems according to the identification frame of the fruit, the average width of the fruit and vegetable stems and the average height of the fruit and vegetable stems.
3. The method for positioning fruit and vegetable stems according to claim 1, wherein the obtaining the target area according to the first identification frame and the identification frame of the fruit corresponding to each fruit and vegetable stem comprises:
under the condition that one fruit and vegetable stem is included in the first image, determining a target area according to a first identification frame of the fruit and vegetable stem;
And under the condition that the first image comprises a plurality of fruit and vegetable stems, determining the target area according to the distance between each first center point and each second center point, wherein the first center point is the center point of a first identification frame of the fruit and vegetable stems, and the second center point is the center point of the identification frame of the fruits.
4. A fruit and vegetable stalk positioning method according to claim 3 wherein, in the case where a plurality of fruit and vegetable stalks are included in the first image, the determining the target area according to the distance between each first center point and the second center point includes:
determining a target center point with the shortest distance between the first center point and the second center point according to the distance between the first center point and the second center point;
And determining the target area according to a third identification frame of the fruit and vegetable stems corresponding to the target center point.
5. The method for positioning a fruit and vegetable stem according to claim 1, wherein the performing image processing on the third image corresponding to the target area to obtain a centroid of the fruit and vegetable stem comprises:
gray processing is carried out on the third image, and a fourth image is obtained;
Processing the fourth image based on wavelet transformation to obtain a fifth image;
removing background pixel points in the fifth image to obtain a sixth image;
obtaining a threshold value of the sixth image based on a preset self-adaptive threshold segmentation algorithm;
Binarizing the sixth image based on the threshold value to obtain a seventh image;
removing noise points in the seventh image to obtain a target connected domain in the seventh image;
and determining the mass center of the fruit and vegetable stems according to the mass center of the target connected domain.
6. The fruit and vegetable stalk positioning method according to claim 1 wherein said determining positioning information of said fruit and vegetable stalk on said second image based on the centroid of said fruit and vegetable stalk comprises:
Converting the two-dimensional coordinates of the barycenter of the fruit and vegetable stems in the third image into three-dimensional coordinates of a camera;
And determining the positioning information of the fruit and vegetable stems on the second image according to the three-dimensional coordinates of the camera.
7. A fruit and vegetable stem positioning system, comprising: the device comprises a first acquisition module, a second acquisition module, a third acquisition module and a positioning module;
the first acquisition module is used for inputting each first image into a first target detection model to obtain a first identification frame corresponding to the fruit and vegetable stems in each first image, the first images are obtained by cutting out second images in a data set according to the interested areas of the fruit and vegetable stems, the data set comprises a plurality of second images, and the first target detection model is determined according to the trained second target detection model;
the second obtaining module is used for obtaining a target area according to a first identification frame corresponding to each fruit and vegetable stem and an identification frame of a fruit, wherein the identification frame of the fruit is obtained by inputting each second image into the second target detection model;
The third acquisition module is used for performing image processing on a third image corresponding to the target area to acquire the mass center of the fruit and vegetable stems;
and the positioning module is used for determining the positioning information of the fruit and vegetable stems on the second image according to the mass centers of the fruit and vegetable stems.
8. An electronic device comprising a processor and a memory storing a computer program, characterized in that the processor implements the fruit and vegetable stem positioning method according to any one of claims 1 to 6 when executing the computer program.
9. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the fruit and vegetable stem positioning method according to any of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the fruit and vegetable stem positioning method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211574927.7A CN118172414A (en) | 2022-12-08 | 2022-12-08 | Fruit and vegetable stem positioning method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211574927.7A CN118172414A (en) | 2022-12-08 | 2022-12-08 | Fruit and vegetable stem positioning method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118172414A true CN118172414A (en) | 2024-06-11 |
Family
ID=91353241
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211574927.7A Pending CN118172414A (en) | 2022-12-08 | 2022-12-08 | Fruit and vegetable stem positioning method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118172414A (en) |
-
2022
- 2022-12-08 CN CN202211574927.7A patent/CN118172414A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Dias et al. | Multispecies fruit flower detection using a refined semantic segmentation network | |
| Dorj et al. | An yield estimation in citrus orchards via fruit detection and counting using image processing | |
| Zhuang et al. | Computer vision-based localisation of picking points for automatic litchi harvesting applications towards natural scenarios | |
| Hung et al. | Orchard fruit segmentation using multi-spectral feature learning | |
| CN109165538B (en) | Bar code detection method and device based on deep neural network | |
| CN105989601B (en) | Agricultural AGV corn inter-row navigation datum line extraction method based on machine vision | |
| CN112990103B (en) | String mining secondary positioning method based on machine vision | |
| CN110599507B (en) | Tomato identification and positioning method and system | |
| CN112380926B (en) | Weeding path planning system of field weeding robot | |
| CN109754423B (en) | Method and equipment for extracting coverage area of leaf scab | |
| Devi et al. | Image processing system for automatic segmentation and yield prediction of fruits using open CV | |
| CN111950391A (en) | Fruit tree bud recognition method and device | |
| CN111539293A (en) | Fruit tree disease diagnosis method and system | |
| CN111046782B (en) | Quick fruit identification method for apple picking robot | |
| Rahim et al. | Deep learning-based accurate grapevine inflorescence and flower quantification in unstructured vineyard images acquired using a mobile sensing platform | |
| CN110363103B (en) | Insect pest identification method and device, computer equipment and storage medium | |
| CN114596274A (en) | Natural background citrus greening disease detection method based on improved Cascade RCNN network | |
| CN111369497B (en) | Walking type tree fruit continuous counting method and device | |
| CN112418112A (en) | Orchard disease and pest monitoring and early warning method and system | |
| CN108967246B (en) | Shrimp larvae positioning method | |
| CN116385271A (en) | Remote sensing image cultivated land abandoned land recognition method | |
| CN113723833B (en) | Method, system, terminal equipment and storage medium for evaluating quality of forestation actual results | |
| CN111401121A (en) | Method for realizing citrus segmentation based on super-pixel feature extraction | |
| CN118172414A (en) | Fruit and vegetable stem positioning method and system | |
| CN114998274A (en) | Object positioning method and device, electronic equipment and readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |