CN109711325B - Mango picking point identification method - Google Patents
Mango picking point identification method Download PDFInfo
- Publication number
- CN109711325B CN109711325B CN201811587011.9A CN201811587011A CN109711325B CN 109711325 B CN109711325 B CN 109711325B CN 201811587011 A CN201811587011 A CN 201811587011A CN 109711325 B CN109711325 B CN 109711325B
- Authority
- CN
- China
- Prior art keywords
- fruit
- mango
- fruits
- picking
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 235000014826 Mangifera indica Nutrition 0.000 title claims abstract description 84
- 235000004936 Bromus mango Nutrition 0.000 title claims abstract description 83
- 235000009184 Spondias indica Nutrition 0.000 title claims abstract description 83
- 238000000034 method Methods 0.000 title claims abstract description 41
- 240000007228 Mangifera indica Species 0.000 title description 4
- 235000013399 edible fruits Nutrition 0.000 claims abstract description 239
- 241001093152 Mangifera Species 0.000 claims abstract description 80
- 230000011218 segmentation Effects 0.000 claims abstract description 61
- 238000001514 detection method Methods 0.000 claims abstract description 7
- 238000012360 testing method Methods 0.000 claims description 9
- 238000012549 training Methods 0.000 claims description 9
- 238000002372 labelling Methods 0.000 claims description 6
- 230000009466 transformation Effects 0.000 claims description 6
- 238000013139 quantization Methods 0.000 claims description 5
- 238000005286 illumination Methods 0.000 claims description 4
- 238000012795 verification Methods 0.000 claims description 4
- 238000009499 grossing Methods 0.000 claims description 3
- 230000009191 jumping Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000005070 sampling Methods 0.000 claims description 3
- 230000001629 suppression Effects 0.000 claims description 3
- 239000002420 orchard Substances 0.000 abstract description 6
- 230000008859 change Effects 0.000 abstract description 3
- 238000013527 convolutional neural network Methods 0.000 description 23
- 238000010586 diagram Methods 0.000 description 9
- 241000220225 Malus Species 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 241000207199 Citrus Species 0.000 description 2
- 241000196324 Embryophyta Species 0.000 description 2
- 235000021016 apples Nutrition 0.000 description 2
- 235000020971 citrus fruits Nutrition 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 238000003709 image segmentation Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 235000011389 fruit/vegetable juice Nutrition 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a mango picking point identification method, which comprises the following steps: collecting mango images, and establishing a mango picking image library in a natural scene; establishing a mango fruit segmentation model based on a Mask R-CNN network; calculating the major axis, the minor axis and the mass center of each fruit; judging whether clustering is carried out by using a bottom-up hierarchical clustering method; if the mango fruits are clustered, identifying clustered fruit parent branches and positioning picking points on the parent branches; if the mango is a single fruit, dividing and identifying the fruit stalks of the fruit, and determining picking points on the fruit stalks. According to the invention, the mango fruit segmentation model based on the Mask R-CNN network is utilized to segment the fruit instance, so that the detection segmentation problem caused by light change, shielding and overlapping in a natural orchard scene is solved, and the method has the advantages of accurate segmentation and multiple applicable scenes.
Description
Technical Field
The invention belongs to the technical field of fruit picking point identification, and particularly relates to a mango fruit segmentation, fruit stem identification and picking point positioning method based on Mask R-CNN.
Background
Mango in China is one of the original production places, and has an important role in the development of fruit industry in China for producing large countries with mango. Current mango picking still relies on a lot of manpower. With the continuous expansion of mango production areas in China and the increasing shortage of agricultural labor force, the level of mechanization, automation and intelligence of mango picking is needed to be improved. And the effective identification of mango fruit picking points is a precondition of intelligent mango picking.
In fruit segmentation and picking point identification, traditional computer vision and currently popular Deep Convolutional Neural Networks (DCNN) are mainly adopted at present. Publication number CN101085442P discloses a method for image processing classification of citrus based on RGB composite model. And (3) carrying out fruit threshold segmentation and edge extraction on the R channel image, and carrying out fruit segmentation by calculating the maximum fruit diameter of the citrus and the color level value of the target area. Publication number CN104036494B discloses a rapid matching calculation method for fruit images, which realizes rapid matching of fruit images by extraction of effective matching areas and improvement of SIFT method. Publication number CN106124511a discloses a fruit surface defect detection method based on adaptive brightness regression correction. Converting the RGB color image of the fruit into an initial gray image, dividing the background and the target, performing binarization processing to obtain a gray image of the fruit, calculating to obtain a brightness correction image, performing binarization and completing hole filling and median filtering to obtain a defect image of the surface of the fruit. Publication number CN104636722a discloses a method for quickly tracking and identifying overlapped fruits of a picking robot. 10 frames of overlapping apple images continuously collected by a camera are used for dividing the collected first frame of images, and the background is removed; determining the position of the circle center of the overlapped apples by calculating the maximum value from the point in the circle to the minimum distance of the contour edge, fitting and pre-judging the motion path of the robot according to the circle center of each frame of image, determining the position of the overlapped apples in the next frame of image by combining the radius and the pre-judging path, and intercepting the overlapped apple region; and finally, carrying out matching identification by adopting a rapid normalized cross-correlation matching algorithm.
However, such methods for visually identifying fruit on crowns using computer vision have disadvantages, mainly using color, shape and texture features. When the color of the fruit is similar to the background color, the fruit is difficult to divide; shape features are also susceptible to more occluded or overlapping natural orchard scenes. The application of the deep convolutional neural network in agriculture in recent years realizes a great breakthrough. However, the fruit instance segmentation research based on the deep convolutional neural network is very few at present, and the fruit instance segmentation is an important step of picking point identification.
Therefore, there is a need to provide a mango picking point identification method based on Mask R-CNN.
Disclosure of Invention
The invention mainly aims to overcome the defects and shortcomings of the prior art and provides a mango picking point identification method, which utilizes a mango fruit segmentation model based on a Mask R-CNN network to segment fruit examples, solves the detection segmentation problem caused by light change, shielding and overlapping in a natural orchard scene and has the advantages of accurate segmentation and multiple applicable scenes.
The aim of the invention is achieved by the following technical scheme: a mango picking point identification method comprises the following steps:
s1, acquiring mango images, and establishing a mango picking image library in a natural scene;
s2, establishing a mango fruit segmentation model based on a Mask R-CNN network;
s3, calculating the major axis, the minor axis and the mass center of each fruit;
s4, judging whether clustering is carried out: clustering fruits according to a spatial relationship by using a bottom-up hierarchical clustering method, grouping the fruits which are adhered and overlapped in space into a group, grouping single fruits into a group, and judging whether the fruits of each group are clustered according to the number of the fruits in each group;
s5, picking point identification: if the mango fruits are clustered, identifying clustered fruit parent branches and positioning picking points on the parent branches; if the mango is a single fruit, dividing and identifying the fruit stalks of the fruit, and determining picking points on the fruit stalks.
Preferably, the specific steps of the step S1 are as follows:
s11, data acquisition: collecting immature mango color images under different illumination and different angles;
s12, constructing a database: the collected data are arranged, the fruit images are adjusted to be uniform standard format sizes, and an image training set, a verification set and a test set are established;
s13, data marking: performing instance segmentation labeling on the data;
s14, data enhancement: the original image is subjected to contrast enhancement, sharpness enhancement and other operations to expand the data set.
Further, the fruit image format size is 1008×756 in the step S12.
Preferably, in the step S2, a Mask R-CNN network for mango fruit segmentation is established, a simplified residual basic network is established, and a feature pyramid structure (FPN) is constructed in a backbone network; introducing soft non-maximum suppression (softNMS) into the Mask R-CNN network to obtain a mango fruit segmentation model.
Further, the specific steps of the step S2 are as follows:
s21, constructing a Mask R-CNN-based basic network, wherein the basic convolution layer comprises 5 residual modules, namely residual modules 1, 2, 3, 4 and 5; removing part of residual modules, simplifying the convolutional layer to 38 layers, and changing the network output category number to 2;
s22, up-sampling the feature images output by the residual error module 5 by 2 times in sequence;
s23, adding the characteristic graphs output by the residual modules 4, 3 and 2 with the characteristic graphs generated in the step S22 after 1×1 convolution transformation, and performing 3×3 convolution transformation again;
s24, replacing NMS with softNMS to reduce missing inspection and segmentation of overlapped fruits;
s25, pretraining the Mask R-CNN network by using the COCO data set, and fine-tuning the Mask R-CNN network by using the mango segmentation training set to obtain a mango fruit segmentation model.
Preferably, the specific steps of the step S4 are as follows:
let fruit real number in the image be N, calculate N fruit centroid distance matrix, hierarchical clustering's basic algorithm is as follows:
s41, initializing: taking each fruit as a cluster, wherein the distance (similarity) between clusters is the Euclidean space distance between the fruits;
s42, finding the nearest (most similar)Similar) two clusters C i And C j The distance between the two clusters is calculated using a single connection method (shortest distance method):
wherein a and b are each C i And C j Element (fruit) of (a) a plant;
s43, if
Combining the two clusters to form a new cluster and jumping to step S42; otherwise, executing S44; wherein l a And l b Short axis lengths of a and b, respectively; />
S44, ending, and outputting a clustering result: in a certain cluster, if the number of fruits is 1, the fruits are single fruits, otherwise, the fruits are clustered.
Preferably, in the step S5, when the mango is a single fruit, the steps of dividing and identifying the fruit stalks of the single fruit, and determining the picking points on the fruit stalks are as follows:
s511, pushing the fruit detection frame upwards by a distance of 2/5 of the length of the long axis of the fruit to obtain an interested region where the fruit stalks are located;
s512, in the region of interest of the fruit stalks, performing branch segmentation by using an unsupervised color texture region segmentation algorithm JSEG (J-image Segmentation);
s513, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying the branches by using a minimum distance classifier;
s514, closing the branch segmentation result by using 5X 5 circular structural elements, and smoothing the branch segmentation result;
s515, judging the peduncles of the fruits; finding out the connection point of the fruit and the branch, solving the connection point and the mass center connection line of the fruit, solving the included angle between the connection line and the vertical line, wherein the branch with the smallest included angle is the fruit stalk of the fruit, and selecting the highest position of the fruit stalk in the interested area as the picking point of the fruit.
Further, in the step S512, a color quantization threshold T c =120, scale number 3, region merging threshold T r ＝0.4。
Preferably, in the step S5, when the mango fruits are clustered, the specific steps of identifying the clustered fruit parent branches and positioning picking points on the parent branches are as follows:
s521, taking all fruits in the cluster as a whole, and taking an external rectangular frame; because the fruit stalks and the parent branches are often above the fruits, the circumscribed rectangular frame is pushed upwards, the pushed height is 2/5 of the length of the major axis of the largest fruit in the cluster, and the pushed height is used as an interested area of the fruit stalks/the parent branches;
s522, in the region of interest of the fruit stalks/parent branches, performing branch segmentation by using JSEG;
s523, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying fruit stem/parent branch regions by utilizing a minimum distance classifier;
s524, labeling the communicating body of the branch area; and for each communicating body, taking the highest position of the cluster fruits as an initial position, scanning upwards, calculating the number of intersection points of a scanning line and branches, searching the coordinate with the last intersection point reduced to 1 as the highest bifurcation point of the parent branch, and selecting the highest position of the parent branch above the highest bifurcation point as a picking point.
Further, in the step S522, a color quantization threshold T c =120, scale number 3, region merging threshold T r ＝0.4。
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. according to the invention, the mango fruit segmentation model based on the Mask R-CNN network is utilized to segment the fruit instance, so that the detection segmentation problem caused by light change, shielding and overlapping in a natural orchard scene is solved, and the method has the advantages of accurate segmentation and multiple applicable scenes.
2. The invention judges whether fruits are clustered by using a hierarchical clustering method, adopts different picking point identification methods according to whether the fruits are clustered, is better suitable for mango picking characteristics, and ensures the integrity of the fruits on the premise of accurate picking.
Drawings
Fig. 1 is a flowchart of an overall framework of a mango picking point identification method according to an embodiment of the invention.
Fig. 2 is a segmentation result of fruit instance segmentation using a Mask R-CNN network-based mango fruit segmentation model according to an embodiment of the present invention.
Fig. 3 is a schematic flow chart of a picture processing effect of a mango picking point identification method according to an embodiment of the present invention, (a) is a schematic diagram of Mask R-CNN segmentation of a fruit instance; (b) clustering and identifying schematic diagrams for hierarchical clustering fruits; (c) a schematic diagram of the regions of interest of fruit stalks and parent branches; (d) The method is characterized by schematically identifying the fruit stalk/parent branch JSEG segmentation and picking points.
Fig. 4 is a schematic diagram of identifying single fruit picking points by using a mango picking point identification method according to an embodiment of the present invention, (a) is a schematic diagram of division of a fruit instance Mask R-CNN; (b) is a schematic diagram of the regions of interest of fruit stalks and parent branches; (c) The method is characterized by schematically identifying the fruit stalk/parent branch JSEG segmentation and picking points.
Fig. 5 is a schematic diagram of identifying clustered fruit picking points by using a mango picking point identification method according to an embodiment of the invention; (a) is a schematic representation of clustered fruits; (b) is a schematic diagram of the regions of interest of fruit stalks and parent branches; (c) The method is characterized by schematically identifying the fruit stalk/parent branch JSEG segmentation and picking points.
Fig. 6 is a schematic diagram of the number of intersections of the row scan lines with the limbs during the identification of the clustered fruit picking points of fig. 5.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
Examples
A mango picking point identification method comprises the following steps:
s1, acquiring mango images, and establishing a mango picking image library in a natural scene;
s11, data acquisition: collecting immature mango color images under different illumination and different angles by using high-definition digital camera equipment;
s12, constructing a database: the collected data are collated, fruit images are adjusted to 1008×756, and an image training set, a verification set and a test set are established;
s13, data marking: performing instance segmentation labeling on the data by using open source Labelme software;
s14, data enhancement: the original image is subjected to contrast enhancement, sharpness enhancement and other operations to expand the data set.
S2, constructing a simplified residual basic network, and constructing a characteristic pyramid structure (FPN) in a backbone network; introducing soft non-maximum suppression (softNMS) into a Mask R-CNN network to obtain a mango fruit segmentation model based on the Mask R-CNN network;
s21, constructing a Mask R-CNN-based basic network, wherein the basic convolution layer comprises 5 residual modules, namely residual modules 1, 2, 3, 4 and 5; removing part of residual modules, simplifying the convolutional layer to 38 layers, and changing the network output class number 80 to 2;
s22, up-sampling the feature images output by the residual error module 5 by 2 times in sequence;
s23, adding the characteristic graphs output by the residual modules 4, 3 and 2 with the characteristic graphs generated in the step S22 after 1×1 convolution transformation, and performing 3×3 convolution transformation again;
s24, replacing NMS with softNMS to reduce missing inspection and segmentation of overlapped fruits;
s25, pretraining the Mask R-CNN network by using the COCO data set, and fine-tuning the Mask R-CNN network by using the mango segmentation training set to obtain a mango fruit segmentation model.
S3, calculating the major axis, the minor axis and the mass center of each fruit;
s4, judging whether the clusters are formed or not; one branch of mango fruits is single fruit, one branch of mango fruits bears two or more clustered fruits, and two picking point judging modes are different. Therefore, whether fruits are clustered is firstly distinguished, and picking points are respectively identified according to different conditions. Clustering fruits according to a spatial relationship by using a bottom-up hierarchical clustering method, grouping the fruits which are adhered and overlapped in space into a group, grouping single fruits into a group, and judging whether the fruits of each group are clustered according to the number of the fruits in each group;
let fruit real number in the image be N, calculate N fruit centroid distance matrix, hierarchical clustering's basic algorithm is as follows:
s41, initializing: taking each fruit as a cluster, wherein the distance (similarity) between clusters is the Euclidean space distance between the fruits;
s42, finding two clusters C which are closest (most similar) i And C j The distance between the two clusters is calculated using a single connection method (shortest distance method):
wherein a and b are each C i And C j Element (fruit) of (a) a plant;
s43, if
Combining the two clusters to form a new cluster and jumping to step S42; otherwise, executing S44; wherein l a And l b Short axis lengths of a and b, respectively;
s44, ending, and outputting a clustering result: in a certain cluster, if the number of fruits is 1, the fruits are single fruits, otherwise, the fruits are clustered.
S5, identifying picking points; if the mango fruits are clustered, identifying clustered fruit parent branches and positioning picking points on the parent branches; if the mango is a single fruit, dividing and identifying the fruit stalks of the fruit, and determining picking points on the fruit stalks. The method has the advantages that the black spots of mangoes caused by juice secreted by the fruit stalks in too short time are avoided, 2-10 cm of the fruit stalks are reserved as much as possible when the mangoes are picked, and the length of the mangoes is 5-10 cm and the width of the mangoes is 3-4.5 cm, so that the picking points can be positioned by utilizing the length information of the mangoes. Single fruits and clustered fruits have different growth states, and the single fruits often have long and drooping fruit stems; two or more clustered fruits grow on the same parent branch, the stems of the fruits in the cluster are often short and are easily blocked by other fruits in the cluster, and the clustered fruits are often used as a whole, so that picking points need to be positioned on the parent branch. Thus, individual fruit and clustered fruit stalks and picking points are identified, respectively.
When the mango is a single fruit, dividing and identifying the fruit stalks of the fruit, and determining picking points on the fruit stalks comprises the following specific steps:
s511, pushing the fruit detection frame upwards by a distance of 2/5 of the length of the long axis of the fruit to obtain an interested region where the fruit stalks are located;
s512, in the region of interest of the fruit stalks, performing branch segmentation by using an unsupervised color texture region segmentation algorithm JSEG (J-image Segmentation), and performing a color quantization threshold T c =120, scale number 3, region merging threshold T r ＝0.4；
S513, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying the branches by using a minimum distance classifier;
s514, closing the branch segmentation result by using 5X 5 circular structural elements, and smoothing the branch segmentation result;
s515, judging the peduncles of the fruits; finding out the connection point of the fruit and the branch, solving the connection point and the mass center connection line of the fruit, solving the included angle between the connection line and the vertical line, wherein the branch with the smallest included angle is the fruit stalk of the fruit, and selecting the highest position of the fruit stalk in the interested area as the picking point of the fruit.
When mango fruits are clustered, the specific steps of identifying clustered fruit parent branches and positioning picking points on the parent branches are as follows:
s521, taking all fruits in the cluster as a whole, and taking an external rectangular frame; because the fruit stalks and the parent branches are often above the fruits, the circumscribed rectangular frame is pushed upwards, the pushed height is 2/5 of the length of the major axis of the largest fruit in the cluster, and the pushed height is used as an interested area of the fruit stalks/the parent branches;
s522, in the region of interest of fruit stalks/parent branches, performing branch segmentation by using JSEG, and quantifying a threshold T of color c =120, scale number 3, region merging threshold T r ＝0.4；
S523, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying fruit stem/parent branch regions by utilizing a minimum distance classifier;
s524, labeling the communicating body of the branch area; and for each communicating body, taking the highest position of the cluster fruits as an initial position, scanning upwards, calculating the number of intersection points of a scanning line and branches, searching the coordinate with the last intersection point reduced to 1 as the highest bifurcation point of the parent branch, and selecting the highest position of the parent branch above the highest bifurcation point as a picking point.
The test data acquisition orchard is located in Guangdong cloud floating city and Zhaoqing city mango orchard, and 100 mango trees are probably used. A Canon EOS800D single lens and a Hua mobile phone are used, the distance between the crown and the mobile phone is 1-2 meters, and the total of 1408 gray-green mango images during mango picking are collected according to 4 different directions of southeast and northwest. The weather during image acquisition comprises sunny days, cloudy days and cloudy days, the acquisition period is 8:00-18:00, and the possible illumination conditions such as forward light, reverse light, side light and the like are covered. 1008 pieces of raw data are selected as raw training images, and the remaining 400 pieces are used as test images.
Table 1 mango image dataset
| Data set | Raw data (sheet) | Data expansion (Zhang) | Target(s) for effectiveness |
| Training set | 1008 | 5040 | 29712 |
| Test set | 400 | 0 | 2107 |
The test adopts 32GB memory, a GPU of Nvidia GTX Titan X model,
The hardware platform of the Xeon (R) CPU E3-1245 v3@3.40GHz processor and the Ubuntu16.04 operating system. On a Caffe2 deep learning framework, adopting Python as a programming language to realize the improved Mask R-CNN algorithm; and (5) realizing fruit stalk segmentation and picking point identification by Matlab2018 b.
After the training set data are amplified, a Mask R-CNN network for dividing mango fruit examples is trained, the dividing result is shown in fig. 2 on a test set, the dividing statistical result is shown in table 2, the dividing precision AP75 is up to 91.80%, and the speed is 0.275 seconds per frame.
After the fruits are segmented, whether the fruits are clustered or not is judged through hierarchical clustering. The individual fruits and clustered fruits were subjected to fruit stem/parent branch segmentation and picking point identification, respectively, see fig. 3.
For a single fruit, after acquiring a fruit stem region of interest, identifying the branches by using a JSEG segmentation algorithm and a minimum distance classifier; judging the stem of the fruit by utilizing the position relation between the branches and the fruit. As shown in fig. 4, there are two branches, the connection points of the fruit are A1 and A2, the straight lines of the centroid and A1 and A2 are calculated respectively, the included angles theta 1 and theta 2 with the vertical line are calculated (see fig. 4 c), the branch with the small included angle is selected as the fruit stalk of the fruit, and the highest point of the fruit stalk is selected as the picking point.
For clustered fruits, after obtaining a fruit stalk region of interest, identifying branches by using a JSEG segmentation algorithm and a minimum distance classifier; the line scanning line passes through the branches for times to judge the positions of the parent branches (see figure 6), and the highest end of the parent branches in the region of interest of the parent branches is selected as a picking point.
And selecting 100 picking points randomly from samples, which are not shielded by single fruit stalks and clustered fruit parent branches, on the test set to perform identification accuracy verification, wherein the identification accuracy reaches 87%.
TABLE 2 segmentation statistics
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (8)
1. The mango picking point identification method is characterized by comprising the following steps of:
s1, acquiring mango images, and establishing a mango picking image library in a natural scene;
s2, establishing a mango fruit segmentation model based on a Mask R-CNN network;
s3, calculating the major axis, the minor axis and the mass center of each fruit;
s4, judging whether clustering is carried out: clustering fruits according to a spatial relationship by using a bottom-up hierarchical clustering method, grouping the fruits which are adhered and overlapped in space into a group, grouping single fruits into a group, and judging whether the fruits of each group are clustered according to the number of the fruits in each group;
s5, picking point identification: if the mango fruits are clustered, identifying clustered fruit parent branches and positioning picking points on the parent branches; if the mango is a single fruit, dividing and identifying the fruit stalks of the fruit, and determining picking points on the fruit stalks;
step S2, establishing a Mask R-CNN network for mango fruit segmentation, constructing a simplified residual basic network, and constructing a characteristic pyramid structure in a backbone network; introducing soft non-maximum suppression into a Mask R-CNN network to obtain a mango fruit segmentation model;
the specific steps of the step S2 are as follows:
s21, constructing a Mask R-CNN-based basic network, wherein the basic convolution layer comprises 5 residual modules, namely residual modules 1, 2, 3, 4 and 5; removing part of residual modules, simplifying the convolutional layer to 38 layers, and changing the network output category number to 2;
s22, up-sampling the feature images output by the residual error module 5 by 2 times in sequence;
s23, adding the characteristic graphs output by the residual modules 4, 3 and 2 with the characteristic graphs generated in the step S22 after 1×1 convolution transformation, and performing 3×3 convolution transformation again;
s24, replacing the NMS by softNMS;
s25, pretraining the Mask R-CNN network by using the COCO data set, and fine-tuning the Mask R-CNN network by using the mango segmentation training set to obtain a mango fruit segmentation model.
2. The mango picking point identification method according to claim 1, wherein the specific steps of the step S1 are:
s11, data acquisition: collecting immature mango color images under different illumination and different angles;
s12, constructing a database: the collected data are arranged, the fruit images are adjusted to be uniform standard format sizes, and an image training set, a verification set and a test set are established;
s13, data marking: performing instance segmentation labeling on the data;
s14, data enhancement: the data set is extended by performing contrast enhancement and sharpness enhancement operations on the original image.
3. The mango picking point identification method according to claim 2, wherein the fruit image format size is 1008 x 756 in step S12.
4. The mango picking point identification method according to claim 1, wherein the specific steps of the step S4 are:
let fruit real number in the image be N, calculate N fruit centroid distance matrix, hierarchical clustering's basic algorithm is as follows:
s41, initializing: taking each fruit as a cluster, wherein the distance between clusters is the Euclidean space distance between the fruits;
s42, finding two nearest clusters C i And C j The distance between the two clusters is calculated using a single connection method:
wherein a and b are each C i And C j Is an element of (2);
s43, if
Combining the two clusters to form a new cluster and jumping to step S42; otherwise, executing S44; wherein l a And l b Short axis lengths of a and b, respectively;
s44, ending, and outputting a clustering result: in a certain cluster, if the number of fruits is 1, the fruits are single fruits, otherwise, the fruits are clustered.
5. The method for identifying mango picking point according to claim 1, wherein when the mango is a single fruit in the step S5, dividing and identifying the stem of the fruit, and determining the picking point on the stem comprises the following specific steps:
s511, pushing the fruit detection frame upwards by a distance of 2/5 of the length of the long axis of the fruit to obtain an interested region where the fruit stalks are located;
s512, in the region of interest of the fruit stalks, performing branch segmentation by using an unsupervised color texture region segmentation algorithm JSEG;
s513, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying the branches by using a minimum distance classifier;
s514, closing the branch segmentation result by using 5X 5 circular structural elements, and smoothing the branch segmentation result;
s515, judging the peduncles of the fruits; finding out the connection point of the fruit and the branch, solving the connection point and the mass center connection line of the fruit, solving the included angle between the connection line and the vertical line, wherein the branch with the smallest included angle is the fruit stalk of the fruit, and selecting the highest position of the fruit stalk in the interested area as the picking point of the fruit.
6. The mango picking point identification method according to claim 5, wherein in the step S512, the color quantization threshold T is c =120, scale number 3, region merging threshold T r ＝0.4。
7. The method for identifying mango picking points according to claim 1, wherein in the step S5, when the mango fruits are clustered, the specific steps of identifying clustered fruit parent branches and positioning picking points on the parent branches are as follows:
s521, taking all fruits in the cluster as a whole, and taking an external rectangular frame; the circumscribed rectangle frame is extrapolated upwards, the extrapolated height is 2/5 of the length of the major axis of the largest fruit in the cluster, and the extrapolated height is used as an interested area of the fruit stalks/the parent branches;
s522, in the region of interest of the fruit stalks/parent branches, performing branch segmentation by using JSEG;
s523, calculating first, second and third-order color moments of the 4 color channels, namely R, G and B channels of the RGB color space and H channels of the HSV color space, of each region respectively, and identifying fruit stem/parent branch regions by utilizing a minimum distance classifier;
s524, labeling the communicating body of the branch area; and for each communicating body, taking the highest position of the cluster fruits as an initial position, scanning upwards, calculating the number of intersection points of a scanning line and branches, searching the coordinate with the last intersection point reduced to 1 as the highest bifurcation point of the parent branch, and selecting the highest position of the parent branch above the highest bifurcation point as a picking point.
8. The mango picking point identification method according to claim 7, wherein in said step S522, the color quantization threshold T is c =120, scale number 3, region merging threshold T r ＝0.4。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811587011.9A CN109711325B (en) | 2018-12-25 | 2018-12-25 | Mango picking point identification method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811587011.9A CN109711325B (en) | 2018-12-25 | 2018-12-25 | Mango picking point identification method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109711325A CN109711325A (en) | 2019-05-03 |
| CN109711325B true CN109711325B (en) | 2023-05-23 |
Family
ID=66256132
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811587011.9A Active CN109711325B (en) | 2018-12-25 | 2018-12-25 | Mango picking point identification method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109711325B (en) |
Families Citing this family (28)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110223349A (en) * | 2019-05-05 | 2019-09-10 | 华南农业大学 | A kind of picking independent positioning method |
| CN110309301B (en) * | 2019-05-09 | 2021-03-19 | 上海泰豪迈能能源科技有限公司 | Enterprise category classification method and device and intelligent terminal |
| CN110197129B (en) * | 2019-05-09 | 2023-04-18 | 常州大学 | Visual identification method for growth form of fruits and vegetables in natural environment |
| CN110026987B (en) * | 2019-05-28 | 2022-04-19 | 广东工业大学 | Method, device and equipment for generating grabbing track of mechanical arm and storage medium |
| CN110619632B (en) * | 2019-09-18 | 2022-01-11 | 华南农业大学 | Mango example confrontation segmentation method based on Mask R-CNN |
| CN110659694B (en) * | 2019-09-27 | 2022-10-11 | 华中农业大学 | Method for detecting citrus fruit base based on machine learning |
| CN111046730A (en) * | 2019-11-08 | 2020-04-21 | 北京海益同展信息科技有限公司 | Plant data processing method and device, computer equipment and storage medium |
| CN111738271B (en) * | 2020-03-04 | 2023-05-02 | 沈阳工业大学 | Method for identifying blocked fruits in natural environment |
| CN111476153A (en) * | 2020-04-07 | 2020-07-31 | 中科汇智(重庆)信息技术合伙企业(有限合伙) | Method for calculating fruit maturity |
| CN111507967B (en) * | 2020-04-17 | 2023-06-02 | 华南农业大学 | Mango high-precision detection method in natural orchard scene |
| CN111666883B (en) * | 2020-06-08 | 2023-04-07 | 佛山科学技术学院 | Grape picking robot target identification and fruit stalk clamping and cutting point positioning method |
| CN111738151B (en) * | 2020-06-22 | 2023-10-10 | 佛山科学技术学院 | Grape fruit stem accurate identification method based on deep learning post-optimization |
| CN112132780A (en) * | 2020-08-17 | 2020-12-25 | 珠海市卓轩科技有限公司 | Reinforcing steel bar quantity detection method and system based on deep neural network |
| CN112115885B (en) * | 2020-09-22 | 2023-08-11 | 中国农业科学院农业信息研究所 | Fruit tree fruiting branch shearing point positioning method based on deep convolutional neural network |
| CN112544235B (en) * | 2020-12-04 | 2022-07-12 | 江苏省农业科学院 | Intelligent fruit picking robot |
| CN112580646A (en) * | 2020-12-08 | 2021-03-30 | 北京农业智能装备技术研究中心 | Tomato fruit maturity dividing method and picking robot |
| CN112818781B (en) * | 2021-01-21 | 2021-09-14 | 重庆文理学院 | Method for judging interference type of serial fruit mother branches based on visual scaling method |
| CN112784735B (en) * | 2021-01-21 | 2021-10-01 | 重庆文理学院 | Method for identifying string-type fruit mother branches based on monocular camera and binocular camera |
| CN113177947B (en) * | 2021-04-06 | 2024-04-26 | 广东省科学院智能制造研究所 | Multi-module convolutional neural network-based complex environment target segmentation method and device |
| CN112990103B (en) * | 2021-04-16 | 2023-03-24 | 苏州大学 | String mining secondary positioning method based on machine vision |
| CN113192129B (en) * | 2021-05-25 | 2022-03-25 | 广东技术师范大学 | Method for positioning adhered citrus based on deep convolutional neural network model |
| GB2607579A (en) * | 2021-06-01 | 2022-12-14 | Geovisual Tech Inc | Tool for counting and sizing plants in a field |
| CN113808194B (en) * | 2021-11-17 | 2022-03-08 | 季华实验室 | Method and device for acquiring picking angle of cluster tomatoes, electronic equipment and storage medium |
| CN114258781B (en) * | 2022-01-06 | 2023-07-21 | 重庆邮电大学 | Morphology and color space-based strawberry stem picking point positioning method |
| CN114387520B (en) * | 2022-01-14 | 2024-05-14 | 华南农业大学 | Method and system for accurately detecting compact Li Zijing for robot picking |
| CN114693658A (en) * | 2022-04-01 | 2022-07-01 | 西南交通大学 | Grape fruit stem identification method based on combination of deep learning and image processing |
| CN116616045B (en) * | 2023-06-07 | 2023-11-24 | 山东农业工程学院 | Picking method and picking system based on plant growth |
| CN116935070B (en) * | 2023-09-19 | 2023-12-26 | 北京市农林科学院智能装备技术研究中心 | Modeling method for picking target of fruit cluster picking robot |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529592A (en) * | 2016-11-07 | 2017-03-22 | 湖南源信光电科技有限公司 | License plate recognition method based on mixed feature and gray projection |
| CN107301401A (en) * | 2017-06-21 | 2017-10-27 | 西北农林科技大学 | A kind of multiple target kiwifruit fruit recognition methods and image acquiring device |
| CN108898610A (en) * | 2018-07-20 | 2018-11-27 | 电子科技大学 | A kind of object contour extraction method based on mask-RCNN |
| CN108932471A (en) * | 2018-05-23 | 2018-12-04 | 浙江科技学院 | A kind of vehicle checking method |
-
2018
- 2018-12-25 CN CN201811587011.9A patent/CN109711325B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106529592A (en) * | 2016-11-07 | 2017-03-22 | 湖南源信光电科技有限公司 | License plate recognition method based on mixed feature and gray projection |
| CN107301401A (en) * | 2017-06-21 | 2017-10-27 | 西北农林科技大学 | A kind of multiple target kiwifruit fruit recognition methods and image acquiring device |
| CN108932471A (en) * | 2018-05-23 | 2018-12-04 | 浙江科技学院 | A kind of vehicle checking method |
| CN108898610A (en) * | 2018-07-20 | 2018-11-27 | 电子科技大学 | A kind of object contour extraction method based on mask-RCNN |
Non-Patent Citations (1)
| Title |
|---|
| 自然环境下葡萄采摘机器人采摘点的自动定位;罗陆锋 等;《农业工程学报》;20150131;第31卷(第2期);第1-8页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109711325A (en) | 2019-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109711325B (en) | Mango picking point identification method | |
| Wu et al. | Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments | |
| Li et al. | Detection of fruit-bearing branches and localization of litchi clusters for vision-based harvesting robots | |
| Song et al. | Kiwifruit detection in field images using Faster R-CNN with VGG16 | |
| CN105718945B (en) | Apple picking robot night image recognition method based on watershed and neural network | |
| CN107016405B (en) | A kind of pest image classification method based on classification prediction convolutional neural networks | |
| CN110472575B (en) | Method for detecting ripeness of tomatoes stringing together based on deep learning and computer vision | |
| CN109344883A (en) | Fruit tree diseases and pests recognition methods under a kind of complex background based on empty convolution | |
| CN110569786B (en) | Fruit tree identification and quantity monitoring method and system based on unmanned aerial vehicle data acquisition | |
| Lv et al. | A visual identification method for the apple growth forms in the orchard | |
| CN111652326A (en) | Improved fruit maturity identification method and identification system based on MobileNet v2 network | |
| CN110765916A (en) | Farmland seedling ridge identification method and system based on semantics and example segmentation | |
| Keller et al. | Soybean leaf coverage estimation with machine learning and thresholding algorithms for field phenotyping | |
| CN109919088B (en) | Automatic extraction method for identifying individual plants of pitaya in karst region | |
| Gao et al. | Recognition and Detection of Greenhouse Tomatoes in Complex Environment. | |
| CN115984698A (en) | Litchi fruit growing period identification method based on improved YOLOv5 | |
| CN115862004A (en) | Corn ear surface defect detection method and device | |
| Sun et al. | GHFormer-Net: Towards more accurate small green apple/begonia fruit detection in the nighttime | |
| CN117456358A (en) | Method for detecting plant diseases and insect pests based on YOLOv5 neural network | |
| CN118097372B (en) | Crop growth visual prediction method based on artificial intelligence | |
| CN111950349A (en) | Semantic segmentation based field navigation line extraction method | |
| Liu et al. | “Is this blueberry ripe?”: a blueberry ripeness detection algorithm for use on picking robots | |
| CN118053154A (en) | Oyster mushroom growth monitoring method and device, electronic equipment and storage medium | |
| Khokher et al. | Early yield estimation in viticulture based on grapevine inflorescence detection and counting in videos | |
| CN114462485A (en) | Red date jujube witches broom initial-stage control method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |