CN114973006A - Method, device and system for picking Chinese prickly ash and storage medium - Google Patents
Method, device and system for picking Chinese prickly ash and storage medium Download PDFInfo
- Publication number
- CN114973006A CN114973006A CN202210919966.XA CN202210919966A CN114973006A CN 114973006 A CN114973006 A CN 114973006A CN 202210919966 A CN202210919966 A CN 202210919966A CN 114973006 A CN114973006 A CN 114973006A
- Authority
- CN
- China
- Prior art keywords
- pepper
- picking
- cluster
- point cloud
- prickly ash
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Robotics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a pepper picking method, a device, a system and a storage medium, wherein the method comprises the steps of acquiring pepper point cloud data of a target pepper tree acquired by a three-dimensional laser scanner; performing prickly ash cluster identification and segmentation on the prickly ash point cloud data to obtain a plurality of prickly ash cluster point clouds; determining a point cloud center coordinate of each prickly ash cluster based on a plurality of prickly ash cluster point clouds; and sending the point cloud central coordinates to pepper picking equipment so that the pepper picking equipment drives a pepper picking part to sequentially move to picking positions of a plurality of pepper clusters and execute picking actions. According to the invention, the processing step of long-range pepper identification is added before short-range pepper identification, and a plurality of pepper cluster point clouds obtained by pepper cluster identification and division are utilized to drive pepper picking equipment to pick each pepper cluster in sequence, so that the overall efficiency of automatic pepper picking is improved.
Description
Technical Field
The invention relates to the technical field of automatic control, in particular to a method, a device and a system for picking peppers and a storage medium.
Background
The pepper is widely planted in China as an economic crop, and typical planting places include Shanxi Hancheng, Heyang, Gansu Longnan, Linxia, Sichuan Hanyuan, Henan Funiu mountain, Taihang mountain and the like. The wild pepper is usually planted in hills, hills and in front and back of houses, and is difficult to pick due to the fact that branches are long, have skin pricks and small fruits, so that the wild pepper is basically picked manually at present, and the development of the wild pepper industry is severely restricted.
With the rapid development of computers and automation technologies and the application and popularization of agricultural high and new technologies, the robot technology gradually enters the field of agricultural production, and the modern agriculture is promoted to move to a road for mechanization and intelligent production. The appearance and the application of the agricultural robot change the traditional agricultural labor mode, greatly improve the modern agricultural operation environment, promote the development of the modern agriculture and simultaneously reduce the labor intensity of farmers.
The picking robot is an automatic mechanical harvesting device with a perception system aiming at fruit or vegetable harvesting operation, is a cross edge science integrating multiple disciplines such as machinery, electronic information, computer science, artificial intelligence, agriculture and life science and relates to the multi-disciplinary field knowledge such as mechanical structure, sensing technology, visual image processing, forward and reverse kinematics and dynamics of the robot, control driving technology and information processing. In contrast to industrial robots that operate in a structural environment, picking robots are developed in such a way that many factors such as the characteristics of the robot work object itself and the external growth environment need to be taken into full consideration.
At present, the vision identification technology of the pepper picking robot mainly takes near-view pepper fruit identification as a main part, and extracts and positions a near-view pepper fruit target through identification algorithms such as a threshold segmentation method, a fuzzy C mean algorithm, a K-means clustering algorithm and the like. However, the single close-range identification scheme enables the pepper picking equipment to execute the picking action in the current picking area, so that the position of the next picking area cannot be quickly known, and the automatic picking efficiency of pepper is reduced.
Disclosure of Invention
The invention mainly aims to provide a pepper picking method, device, system and storage medium, and aims to solve the technical problem that the automatic pepper picking efficiency in the whole area is low by the existing single close-range identification scheme.
In order to achieve the purpose, the invention provides a pepper picking method, which comprises the following steps:
acquiring pepper point cloud data of a target pepper tree acquired by a three-dimensional laser scanner;
performing prickly ash cluster identification and segmentation on the prickly ash point cloud data to obtain a plurality of prickly ash cluster point clouds;
determining a point cloud center coordinate of each prickly ash cluster based on a plurality of prickly ash cluster point clouds;
and sending the point cloud central coordinates to pepper picking equipment so that the pepper picking equipment drives a pepper picking part to sequentially move to picking positions of a plurality of pepper clusters and execute picking actions.
Optionally, before the step of obtaining the pepper point cloud data of the target pepper tree acquired by the three-dimensional laser scanner, the method further includes:
constructing a first coordinate system corresponding to the three-dimensional laser scanner and a second coordinate system corresponding to the pepper picking part;
acquiring a first calibration plate corner point cloud of a calibration plate under the first coordinate system and a second calibration plate corner point cloud under the second coordinate system;
and determining a hand-eye relation matrix based on the first calibration plate angular point cloud and the second calibration plate angular point cloud, and executing hand-eye calibration based on the hand-eye relation matrix.
Optionally, after the step of obtaining the pepper point cloud data of the target pepper tree collected by the three-dimensional laser scanner, the method further comprises the following steps:
performing data preprocessing on the prickly ash point cloud data;
wherein the pre-processing comprises one or more of data cropping, data denoising, or data downsampling.
Optionally, the step of performing pepper cluster identification and segmentation on the pepper point cloud data to obtain a plurality of pepper cluster point clouds specifically includes:
performing pepper cluster identification on the pepper point cloud data by using a pepper cluster identification model to obtain pepper cluster point clouds;
and performing pricklyash cluster segmentation on the pricklyash cluster point cloud group by utilizing a pricklyash cluster segmentation algorithm to obtain a plurality of pricklyash cluster point clouds.
Optionally, the step of determining the point cloud center coordinate of each zanthoxylum clusters based on a plurality of zanthoxylum clusters point clouds specifically comprises:
determining a characteristic vector corresponding to the top point of the surface of each zanthoxylum bungeanum cluster based on a plurality of zanthoxylum bungeanum cluster point clouds;
constructing a bounding box coordinate system by using the feature vectors;
and determining the point cloud center coordinate of each prickly ash cluster according to the three-dimensional coordinate information of each prickly ash cluster in the bounding box coordinate system.
Optionally, after the step of determining the point cloud center coordinate of each zanthoxylum clusters, the method further includes: and generating a picking sequence of the pepper clusters corresponding to the target pepper tree according to the point cloud central coordinate of each pepper cluster and the position coordinate of the current pepper picking part.
Optionally, the picking action executing step specifically includes:
when the pepper picking part moves to a picking position of a target pepper cluster, acquiring a pepper cluster depth image;
determining the accurate position coordinates of the target zanthoxylum clusters based on the zanthoxylum cluster depth images;
and sending the accurate position coordinates to pepper picking equipment so that the pepper picking equipment generates a picking instruction set, and driving a pepper picking part to carry out picking action on a target pepper cluster according to the picking instruction set.
In addition, in order to achieve the above object, the present invention also provides a pepper picking apparatus, comprising:
the point cloud data acquisition module is used for acquiring pepper point cloud data of a target pepper tree acquired by the three-dimensional laser scanner;
the identification and segmentation module is used for identifying and segmenting the pepper clusters of the pepper point cloud data to obtain a plurality of pepper cluster point clouds;
the determining module is used for determining the point cloud center coordinates of each prickly ash cluster based on a plurality of prickly ash cluster point clouds;
and the picking module is used for sending the point cloud center coordinates to the pepper picking equipment so that the pepper picking equipment drives the pepper picking parts to sequentially move to picking positions of a plurality of pepper clusters and executes picking actions.
In addition, in order to achieve the above object, the present invention also provides a pepper picking system, comprising:
a three-dimensional laser scanner;
picking equipment for Chinese prickly ash; and
with three-dimensional laser scanner with equipment communication connection's control end is picked to prickly ash, the control end includes: the pepper picking device comprises a memory, a processor and a pepper picking method program which is stored on the memory and can run on the processor, wherein the steps of the pepper picking method are realized when the pepper picking method program is executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a storage medium having stored thereon a pepper picking method program, which when executed by a processor, implements the steps of the pepper picking method as described above.
The embodiment of the invention provides a pepper picking method, a device, a system and a storage medium, wherein the method comprises the steps of acquiring pepper point cloud data of a target pepper tree acquired by a three-dimensional laser scanner; performing prickly ash cluster identification and segmentation on the prickly ash point cloud data to obtain a plurality of prickly ash cluster point clouds; determining a point cloud center coordinate of each prickly ash cluster based on a plurality of prickly ash cluster point clouds; and sending the point cloud central coordinates to pepper picking equipment so that the pepper picking equipment drives a pepper picking part to sequentially move to picking positions of a plurality of pepper clusters and execute picking actions. According to the invention, the processing step of long-range pepper identification is added before short-range pepper identification, and a plurality of pepper cluster point clouds obtained by pepper cluster identification and division are utilized to drive pepper picking equipment to pick each pepper cluster in sequence, so that the overall efficiency of automatic pepper picking is improved.
Drawings
Fig. 1 is a schematic structural diagram of a pepper picking system in the embodiment of the invention.
Fig. 2 is a schematic structural diagram of a control end according to an embodiment of the present invention.
Fig. 3 is a schematic flow chart of a pepper picking method in the embodiment of the invention.
Fig. 4 is a schematic diagram of a calculation result of a hand-eye relationship matrix in the embodiment of the present invention.
FIG. 5 is a diagram illustrating the results of Euclidean clustering segmentation and localization in the embodiment of the present invention.
Fig. 6 is a structural block diagram of a pepper picking device in the embodiment of the invention.
The implementation, functional features and advantages of the present invention will be further described with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
At present, in the related technical field, the automatic picking efficiency of the pepper in the whole area by the existing single close-range identification scheme is not high.
In order to solve this problem, various embodiments of the pepper picking method of the present invention are proposed. According to the pepper picking method provided by the invention, the processing step of pepper long-range view recognition is added before pepper short-range view recognition, and a plurality of pepper cluster point clouds obtained by pepper cluster recognition and segmentation are utilized to drive pepper picking equipment to pick each pepper cluster in sequence, so that the overall efficiency of pepper automatic picking is improved.
Referring to fig. 1, fig. 1 is a schematic structural diagram of a pepper picking system according to an embodiment of the present invention.
In this embodiment, the pepper picking system comprises a three-dimensional laser scanner 100, a pepper picking device 200 and a control terminal 300 in communication connection with the three-dimensional laser scanner 100 and the pepper picking device 200.
Specifically, the three-dimensional laser scanner 100 is configured to collect pepper point cloud data of a target pepper tree, the pepper picking apparatus 200 is configured to perform actions such as position movement, pepper cluster depth image acquisition, pepper picking and the like according to an instruction of the control terminal, the control terminal 300 is configured to generate a coarse positioning (i.e., a point cloud center coordinate) of a pepper cluster according to the pepper point cloud data and generate an accurate position coordinate according to the pepper cluster depth image acquired by the pepper picking apparatus 200, and then send a corresponding driving instruction to the pepper picking apparatus 200 to complete control of the whole pepper picking process.
As can be readily appreciated, the pepper picking apparatus employs a robot that can automatically perform pepper picking, which can move to a target pepper tree to pick clusters of peppers on the target pepper tree.
In a preferred embodiment, the pepper picking device is provided with a pepper picking part for picking pepper clusters, the pepper picking part can be a mechanical arm, a mechanical claw and a corresponding driving control component, and the pepper picking part can rotate with multiple degrees of freedom and stretch in multiple directions so as to pick pepper clusters at different positions on a target pepper tree.
Those skilled in the art will appreciate that the configuration shown in fig. 1 does not constitute a limitation of a pepper picking system and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.
Referring to fig. 2, fig. 2 is a schematic structural diagram of a control end according to an embodiment of the present invention.
The control end may be a User Equipment (UE) such as a Mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a Personal Digital Assistant (PDA), a tablet computer (PAD), a handheld device, a vehicle-mounted device, a wearable device, a computing device or other processing device connected to a wireless modem, a Mobile Station (MS), or the like. The device may be referred to as a user terminal, portable terminal, desktop terminal, etc.
Preferably, the control end can also be configured on the pepper picking equipment, and the rough positioning and the accurate position coordinate generated by the control end can be directly transmitted to the pepper picking equipment so as to execute the pepper picking step.
Generally, the control terminal includes: at least one processor 301, a memory 302, and a pepper picking method program stored on said memory and executable on said processor, said pepper picking method program being configured to implement the steps of a pepper picking method as described previously.
The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and so on. The processor 301 may be implemented in at least one hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor, where the main processor is a processor for processing data in an awake state, and is also called a Central Processing Unit (CPU); a coprocessor is a low power processor for processing data in a standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the display screen. Processor 301 may also include an AI (Artificial Intelligence) processor for processing relevant pepper picking method operations so that pepper picking method models can be trained and learned autonomously, improving efficiency and accuracy.
In some embodiments, the control terminal may further include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected by a bus or signal lines. Various peripheral devices may be connected to communication interface 303 via a bus, signal line, or circuit board. Specifically, the peripheral device includes: at least one of radio frequency circuitry 304, a display screen 305, and a power source 306.
The communication interface 303 may be used to connect at least one peripheral device related to I/O (Input/Output) to the processor 301 and the memory 302. The communication interface 303 is used for receiving the movement tracks of the plurality of mobile terminals uploaded by the user and other data through the peripheral device. In some embodiments, processor 301, memory 302, and communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302 and the communication interface 303 may be implemented on a single chip or circuit board, which is not limited in this embodiment.
The Radio Frequency circuit 304 is used for receiving and transmitting RF (Radio Frequency) signals, also called electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals, so as to obtain the movement tracks and other data of a plurality of mobile terminals. The rf circuit 304 converts an electrical signal into an electromagnetic signal to transmit, or converts a received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 304 comprises: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and so forth. The radio frequency circuitry 304 may communicate with other terminals via at least one wireless communication protocol. The wireless communication protocols include, but are not limited to: metropolitan area networks, various generation mobile communication networks (2G, 3G, 4G, and 5G), Wireless local area networks, and/or WiFi (Wireless Fidelity) networks. In some embodiments, the rf circuit 304 may further include NFC (Near Field Communication) related circuits, which are not limited in this application.
The display screen 305 is used to display a UI (User Interface). The UI may include graphics, text, icons, video, and any combination thereof. When the display screen 305 is a touch display screen, the display screen 305 also has the ability to capture touch signals on or over the surface of the display screen 305. The touch signal may be input to the processor 301 as a control signal for processing. At this point, the display screen 305 may also be used to provide virtual buttons and/or a virtual keyboard, also referred to as soft buttons and/or a soft keyboard. In some embodiments, the display screen 305 may be one, the front panel of the electronic device; in other embodiments, the display screens 305 may be at least two, respectively disposed on different surfaces of the electronic device or in a folded design; in still other embodiments, the display screen 305 may be a flexible display screen disposed on a curved surface or a folded surface of the electronic device. Even further, the display screen 305 may be arranged in a non-rectangular irregular figure, i.e. a shaped screen. The Display screen 305 may be made of LCD (liquid crystal Display), OLED (Organic Light-Emitting Diode), and the like.
The power supply 306 is used to power various components in the electronic device. The power source 306 may be alternating current, direct current, disposable or rechargeable. When the power source 306 includes a rechargeable battery, the rechargeable battery may support wired or wireless charging. The rechargeable battery can also be used to support fast charge technology.
Those skilled in the art will appreciate that the configuration shown in fig. 2 does not constitute a limitation of the control end and may include more or fewer components than shown, or some components may be combined, or a different arrangement of components.
The embodiment of the invention provides a pepper picking method, and referring to fig. 3, fig. 3 is a flow schematic diagram of the pepper picking method embodiment of the invention.
In this embodiment, the pepper picking method comprises the following steps:
and S100, acquiring pepper point cloud data of the target pepper tree, which is acquired by a three-dimensional laser scanner.
Specifically, the control end acquires the pepper point cloud data of the target pepper tree, and can be connected with the three-dimensional laser scanner to acquire the pepper point cloud data in advance by the three-dimensional laser scanner and download the pepper point cloud data corresponding to the target pepper tree from the point cloud database constructed according to the pepper point cloud data.
It should be noted that after the three-dimensional laser scanner collects the positional information of the zanthoxylum bungeanum clusters, since the coordinate system of the point cloud data and the coordinate system of the zanthoxylum bungeanum picking part are not the same coordinate system, in order to enable the zanthoxylum bungeanum picking part to know the positional information of the zanthoxylum bungeanum clusters, the zanthoxylum bungeanum picking task is accurately executed, the positional information of the zanthoxylum bungeanum clusters cannot be directly utilized, and the hand-eye calibration of the three-dimensional laser scanner and the zanthoxylum bungeanum picking part needs to be executed before the zanthoxylum bungeanum point cloud data of the target zanthoxylum bungeanum tree collected by the three-dimensional laser scanner is acquired.
Specifically, a first coordinate system corresponding to a three-dimensional laser scanner and a second coordinate system corresponding to a pepper picking part are constructed, then a first calibration plate angular point cloud of a calibration plate under the first coordinate system and a second calibration plate angular point cloud under the second coordinate system are obtained, finally, a hand-eye relation matrix is determined based on the first calibration plate angular point cloud and the second calibration plate angular point cloud, and hand-eye calibration is executed based on the hand-eye relation matrix.
In a specific application, the present embodiment may use PCL (Point Cloud Library) to call a transformational optimization svd class to implement the calculation of the hand-eye relationship matrix. Firstly, a rigid body transformation object and a transformation relation matrix are created, then point clouds of calibration board corner points under a coordinate system of a scanner and a pepper picking part are obtained according to a member function estimarigidtranformation () in a transformation optimization SVD class, and a hand-eye relation matrix is calculated through the function.
The pseudo code for executing the hand-eye calibration specifically comprises:
inputting: and point cloud data under the coordinate systems of the three-dimensional laser scanner and the pepper picking part.
And (3) outputting: a hand-eye relationship matrix.
1. Creating a corner point coordinate point cloud pointer CA in a coordinate system of the pepper picking part;
2. creating a corner point coordinate point cloud pointer CB under a three-dimensional laser scanner coordinate system;
3. creating an SVD transformation matrix object TESVD;
4. creating a hand-eye relationship matrix transformation;
5. calculate the hand-eye relationship matrix tesvd.
The calculation result of the hand-eye relationship matrix obtained by performing the hand-eye calibration is shown in fig. 4.
In a preferred embodiment, after the step of acquiring the pepper point cloud data of the target pepper tree acquired by the three-dimensional laser scanner, data preprocessing is also required to be performed on the pepper point cloud data.
Specifically, the data preprocessing comprises one or more of data cutting, data denoising or data downsampling, and the accuracy of processing the pepper point cloud data is improved by performing data preprocessing on the acquired pepper point cloud data.
And S200, performing pricklyash cluster identification and segmentation on the pricklyash point cloud data to obtain a plurality of pricklyash cluster point clouds.
After data preprocessing, pricklyash cluster identification and segmentation processing can be performed on the pricklyash point cloud data, and finally a plurality of pricklyash cluster point clouds are obtained.
Specifically, the prickly ash cluster identification model is utilized to perform prickly ash cluster identification on the prickly ash point cloud data to obtain prickly ash cluster point cloud clusters, and a prickly ash cluster segmentation algorithm is utilized to perform prickly ash cluster segmentation on the prickly ash cluster point cloud clusters to obtain a plurality of prickly ash cluster point cloud clusters.
It should be noted that the step of identifying and segmenting the zanthoxylum clusters can be realized based on the semantic segmentation result of the zanthoxylum clusters by the PointNet + + network. The method comprises the steps of performing semantic segmentation on the Chinese prickly ash trees, wherein the identification and segmentation of the Chinese prickly ash clusters are realized on the basis of deep learning semantic segmentation, and the identified Chinese prickly ash clusters are all Chinese prickly ash cluster point clouds on the Chinese prickly ash trees; after that, because a certain distance exists between the zanthoxylum clusters and the zanthoxylum clusters in the identified zanthoxylum cluster point cloud group, the point cloud group needs to be considered to be divided to obtain a single zanthoxylum cluster point cloud.
In this embodiment, an euclidean clustering method is adopted to segment the pepper cluster point cloud cluster. By setting the optimal parameters after debugging, not only can a single pepper cluster point cloud group be obtained, but also a single picking area formed by pepper cluster point cloud groups with short distances can be obtained, and the division of the picking area is realized by setting reasonable European-style clustering parameters.
And step S300, determining the point cloud center coordinates of each prickly ash cluster based on a plurality of prickly ash cluster point clouds.
After a plurality of pepper cluster point clouds are obtained, the point cloud center coordinates of each pepper cluster can be determined based on the pepper cluster point clouds so as to realize the coarse positioning of the pepper clusters.
Specifically, firstly, determining a characteristic vector corresponding to the top point of the surface of each prickly ash cluster based on a plurality of prickly ash cluster point clouds, then constructing a bounding box coordinate system by using the characteristic vector, and finally determining the point cloud center coordinate of each prickly ash cluster according to the three-dimensional coordinate information of each prickly ash cluster in the bounding box coordinate system.
It should be noted that the point cloud center coordinate determination of each zanthoxylum clusters can be performed by a bounding box algorithm. Wherein there is a bounding box (Ori)entered Bounding Box, OBB) has an arbitrary property with respect to the coordinate Axis direction, which is higher than the tightness of the Axis Aligned Bounding Box (AABB). Therefore, in this embodiment, the bounding box algorithm uses a directed bounding box, and the center point of the directed bounding box of the OBB is assumed to be
Half side length of
The direction vector is
Thus the OBB bounding box region is defined as:
in practical application, the principle of implementing the OBB bounding box in the PCL is as follows: and obtaining a characteristic vector through PCA (principal component analysis) according to the vertex of the surface of the object, wherein a coordinate system formed by the characteristic vector is the coordinate system of the OBB bounding box.
Specifically, when PCL is used for determining the point cloud center coordinates of each zanthoxylum bungeanum cluster, an Euclidean clustering object can be created through an Euclidean Cluster extraction class, then a search radius, a minimum clustering point and a maximum clustering point are respectively set through member functions setCluster Tolerance (), setMinclusterSize (), setMaxMaxMexclusitSize (), finally Euclidean partitioning is carried out through an extract () function, and a partitioning result is stored in a for-loop mode. After single pepper cluster point cloud is obtained through segmentation, a rotational inertia estimation object is created through momentOfInertiaEstimation, then the segmented pepper cluster point cloud is obtained through a member function setInputCloud (), and finally the pepper cluster point cloud center coordinate can be obtained through a member function getMassCenter ().
The pseudo code for performing the Euclidean clustering segmentation and positioning specifically comprises the following steps:
inputting: deeply learning the zanthoxylum bungeanum cluster point cloud cluster with semantic segmentation.
And (3) outputting: center coordinates of a single prickly ash cluster.
1. Creating a European cluster object seg;
2. setting a search radius setclusterTolerance ();
3. setting a minimum clustering point setMinclusterSize ();
4. setting a maximum clustering point setMaxClustSize ();
5. euclidean segmentation extract ();
6. save each segmentation result
for (it = cluster_indices.begin(); it != cluster _indices .end(); ++it)
{
for (pit = cluster_indices.begin(); pit != cluster _indices .end(); ++pit)
cloud_cluster->push_back();
} ；
7. Setting a single pepper cluster point cloud pointer;
8. creating a moment of inertia estimation object moment _ of _ inertia;
9. acquiring a pepper cluster point cloud setInputCloud (redpepper);
10. calculating a point cloud center coordinate getMassCenter () of the zanthoxylum bungeanum clusters;
11. the output center coordinates outfile < < mass _ center (0) < "< < mass _ center (1) <" \\ n ").
The results of performing euclidean segmentation and localization are shown in fig. 5.
In a preferred embodiment, after the step of determining the point cloud center coordinates of each zanthoxylum bungeanum cluster, the picking sequence of the zanthoxylum bungeanum clusters corresponding to the target zanthoxylum bungeanum tree can be generated according to the point cloud center coordinates of each zanthoxylum bungeanum cluster and the position coordinates of the current zanthoxylum bungeanum picking part.
In this embodiment, when sorting the picking sequence of the zanthoxylum bungeanum clusters on the target zanthoxylum bungeanum tree, the required position coordinates can be selected in the point cloud centers of the obtained several zanthoxylum bungeanum clusters based on different principles of shortest time consumption, shortest moving distance or only picking zanthoxylum bungeanum meeting the preset requirements, and the picking sequence of the zanthoxylum bungeanum clusters can be generated according to the required principles.
On the basis, the growth state (such as maturity and insect pest) of each pepper cluster can be collected (such as image collection and identification) before sequencing, so that the preset requirement for selecting and picking pepper can be set according to the obtained growth state of the pepper clusters.
And S400, sending the point cloud center coordinates to pepper picking equipment, so that the pepper picking equipment drives a pepper picking part to sequentially move to picking positions of a plurality of pepper clusters, and executing picking actions.
After the point cloud central coordinate of each pepper cluster is obtained, the point cloud central coordinate can be sent to pepper picking equipment, so that the pepper picking equipment drives the pepper picking parts to sequentially move to a plurality of picking positions of the pepper clusters, and the rough positioning of the pepper clusters and the rough positioning movement of the pepper picking parts are completed.
After that, the picking action is executed, namely the pepper picking part moves to the picking position of the target pepper cluster, and then the precise pepper cluster picking action is executed.
Specifically, when the pepper picking part moves to the picking position of a target pepper cluster, firstly, a pepper cluster depth image is obtained, then, an accurate position coordinate of the target pepper cluster is determined based on the pepper cluster depth image, and finally, the accurate position coordinate is sent to pepper picking equipment, so that the pepper picking equipment generates a picking instruction set, and the pepper picking part is driven to carry out picking action on the target pepper cluster according to the picking instruction set.
It should be noted that, in this embodiment, the communication between the control end and the three-dimensional laser scanner is mainly for controlling the working state thereof and downloading the point cloud data in the internal memory card thereof. The communication between the control end and the pepper picking equipment is used for sending the identified pepper cluster position information to the pepper picking equipment so that the pepper cluster position information is moved to the close-range pepper cluster end, and a close-range pepper cluster identification visual system can conveniently work.
Therefore, the communication between the control end and the three-dimensional laser scanner can be realized by using a come card SDK tool bag, the SDK tool bag is internally packaged with the functions of connecting with the scanner, controlling the start and stop of equipment and downloading data, and the communication requirement between the control end and the three-dimensional laser scanner can be realized. The communication between the control end and the pepper picking equipment can adopt a TCP/IP protocol which can better meet the requirements of the application scene on distance and transmission efficiency.
The embodiment provides a pepper picking method, which is characterized in that a processing step of pepper long-range view recognition is added before pepper short-range view recognition, and pepper cluster point clouds obtained by pepper cluster recognition and segmentation are utilized to drive pepper picking equipment to pick each pepper cluster in sequence, so that the overall efficiency of pepper automatic picking is improved.
Referring to fig. 6, fig. 6 is a structural block diagram of an embodiment of a pepper picking device of the present invention.
As shown in fig. 6, the pepper picking device provided by the embodiment of the invention comprises:
the point cloud data acquisition module 10 is used for acquiring pepper point cloud data of a target pepper tree acquired by a three-dimensional laser scanner;
the identification and segmentation module 20 is used for identifying and segmenting the pepper clusters of the pepper point cloud data to obtain a plurality of pepper cluster point clouds;
the determining module 30 is used for determining the point cloud center coordinates of each prickly ash cluster based on a plurality of prickly ash cluster point clouds;
and the picking module 40 is used for sending the point cloud center coordinates to the pepper picking equipment so that the pepper picking equipment drives the pepper picking parts to sequentially move to picking positions of a plurality of pepper clusters and executes picking actions.
Other embodiments or specific implementation manners of the pepper picking device can refer to the above method embodiments, and are not described herein again.
In addition, the embodiment of the invention also provides a storage medium, wherein the storage medium stores a pepper picking method program, and the pepper picking method program realizes the steps of the pepper picking method as described above when being executed by a processor. Therefore, a detailed description thereof will be omitted. In addition, the beneficial effects of the same method are not described in detail. For technical details not disclosed in embodiments of the computer-readable storage medium referred to in the present application, reference is made to the description of embodiments of the method of the present application. It is determined that, by way of example, the program instructions may be deployed to be executed on one computing device or on multiple computing devices at one site or distributed across multiple sites and interconnected by a communication network.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
It should be noted that the above-described embodiments of the apparatus are merely schematic, where the units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. In addition, in the drawings of the embodiment of the apparatus provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, and may be specifically implemented as one or more communication buses or signal lines. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that the present invention may be implemented by software plus necessary general hardware, and may also be implemented by special purpose hardware including special purpose integrated circuits, special purpose CPUs, special purpose memories, special purpose components and the like. Generally, functions performed by computer programs can be easily implemented by corresponding hardware, and specific hardware structures for implementing the same functions may be various, such as analog circuits, digital circuits, or dedicated circuits. However, the implementation of a software program is a more preferable embodiment for the present invention. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which is stored in a readable storage medium, such as a floppy disk, a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk of a computer, and includes instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
Claims (10)
1. A pepper picking method is characterized by comprising the following steps:
acquiring pepper point cloud data of a target pepper tree acquired by a three-dimensional laser scanner;
performing prickly ash cluster identification and segmentation on the prickly ash point cloud data to obtain a plurality of prickly ash cluster point clouds;
determining a point cloud center coordinate of each prickly ash cluster based on a plurality of prickly ash cluster point clouds;
and sending the point cloud central coordinates to pepper picking equipment so that the pepper picking equipment drives a pepper picking part to sequentially move to picking positions of a plurality of pepper clusters and execute picking actions.
2. The pepper picking method according to claim 1, wherein before the step of obtaining the pepper point cloud data of the target pepper tree collected by the three-dimensional laser scanner, the method further comprises:
constructing a first coordinate system corresponding to the three-dimensional laser scanner and a second coordinate system corresponding to the pepper picking part;
acquiring a first calibration plate angular point cloud of a calibration plate under the first coordinate system and a second calibration plate angular point cloud under the second coordinate system;
and determining a hand-eye relation matrix based on the first calibration plate angular point cloud and the second calibration plate angular point cloud, and executing hand-eye calibration based on the hand-eye relation matrix.
3. The pepper picking method according to claim 1, wherein after the step of obtaining the pepper point cloud data of the target pepper tree collected by the three-dimensional laser scanner, the method further comprises:
performing data preprocessing on the prickly ash point cloud data;
wherein the pre-processing comprises one or more of data cropping, data denoising, or data downsampling.
4. The pepper picking method according to claim 1, wherein the step of performing pepper cluster recognition and segmentation on the pepper point cloud data to obtain a plurality of pepper cluster point clouds specifically comprises:
performing pepper cluster identification on the pepper point cloud data by using a pepper cluster identification model to obtain pepper cluster point clouds;
and performing pricklyash cluster segmentation on the pricklyash cluster point cloud group by utilizing a pricklyash cluster segmentation algorithm to obtain a plurality of pricklyash cluster point clouds.
5. The pepper picking method according to claim 1, wherein the step of determining the point cloud center coordinates of each pepper cluster based on a plurality of the pepper cluster point clouds specifically comprises:
determining a characteristic vector corresponding to the top point of the surface of each zanthoxylum bungeanum cluster based on a plurality of zanthoxylum bungeanum cluster point clouds;
constructing a bounding box coordinate system by using the feature vectors;
and determining the point cloud center coordinate of each prickly ash cluster according to the three-dimensional coordinate information of each prickly ash cluster in the bounding box coordinate system.
6. The pepper picking method according to claim 1, wherein after the step of determining the point cloud center coordinates of each pepper cluster, the method further comprises: and generating a picking sequence of the pepper clusters corresponding to the target pepper tree according to the point cloud central coordinate of each pepper cluster and the position coordinate of the current pepper picking part.
7. The pepper picking method according to claim 1, wherein the step of performing picking actions specifically comprises:
when the pepper picking part moves to the picking position of the target pepper cluster, acquiring a pepper cluster depth image;
determining the accurate position coordinates of the target zanthoxylum clusters based on the zanthoxylum cluster depth images;
and sending the accurate position coordinates to pepper picking equipment so that the pepper picking equipment generates a picking instruction set, and driving a pepper picking part to carry out picking action on a target pepper cluster according to the picking instruction set.
8. A pepper picking device, comprising:
the point cloud data acquisition module is used for acquiring pepper point cloud data of a target pepper tree acquired by the three-dimensional laser scanner;
the identification and segmentation module is used for identifying and segmenting the pepper clusters of the pepper point cloud data to obtain a plurality of pepper cluster point clouds;
the determining module is used for determining the point cloud center coordinates of each prickly ash cluster based on a plurality of prickly ash cluster point clouds;
and the picking module is used for sending the point cloud center coordinates to the pepper picking equipment so that the pepper picking equipment drives the pepper picking parts to sequentially move to picking positions of a plurality of pepper clusters and execute picking actions.
9. A pepper picking system, comprising:
a three-dimensional laser scanner;
picking equipment for Chinese prickly ash; and
with three-dimensional laser scanner with equipment communication connection's control end is picked to prickly ash, the control end includes: a memory, a processor and a pepper picking method program stored on said memory and executable on said processor, said pepper picking method program when executed by said processor implementing the steps of the pepper picking method according to any one of claims 1-7.
10. A storage medium, characterized in that the storage medium stores thereon a zanthoxylum bungeanum picking method program, which when executed by a processor implements the steps of the zanthoxylum bungeanum picking method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210919966.XA CN114973006B (en) | 2022-08-02 | 2022-08-02 | Method, device and system for picking Chinese prickly ash and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210919966.XA CN114973006B (en) | 2022-08-02 | 2022-08-02 | Method, device and system for picking Chinese prickly ash and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114973006A true CN114973006A (en) | 2022-08-30 |
| CN114973006B CN114973006B (en) | 2022-10-18 |
Family
ID=82969986
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210919966.XA Active CN114973006B (en) | 2022-08-02 | 2022-08-02 | Method, device and system for picking Chinese prickly ash and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114973006B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116652951A (en) * | 2023-06-08 | 2023-08-29 | 广东技术师范大学 | Method and device for positioning robot vision in unstructured large working space |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109410238A (en) * | 2018-09-20 | 2019-03-01 | 中国科学院合肥物质科学研究院 | A kind of fructus lycii identification method of counting based on PointNet++ network |
| CN110226413A (en) * | 2019-06-26 | 2019-09-13 | 佛山科学技术学院 | A kind of picking robot goes here and there the method for stacking grape more |
| US20200117898A1 (en) * | 2018-10-10 | 2020-04-16 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| CN111758395A (en) * | 2020-07-21 | 2020-10-13 | 四川省机械研究设计院 | Prickly ash picking device and prickly ash vision picking robot that constitutes thereof |
| CN111882610A (en) * | 2020-07-15 | 2020-11-03 | 中国科学院自动化研究所 | Method for grabbing target object by service robot based on elliptical cone artificial potential field |
| CN112509145A (en) * | 2020-12-22 | 2021-03-16 | 珠海格力智能装备有限公司 | Material sorting method and device based on three-dimensional vision |
| CN112802199A (en) * | 2021-01-25 | 2021-05-14 | 夏永霞 | High-precision mapping point cloud data processing method and system based on artificial intelligence |
| CN112907735A (en) * | 2021-03-10 | 2021-06-04 | 南京理工大学 | Flexible cable identification and three-dimensional reconstruction method based on point cloud |
| WO2021168708A1 (en) * | 2020-02-26 | 2021-09-02 | 深圳市大疆创新科技有限公司 | Control method for movable platform, movable platform, device and storage medium |
| US20220024486A1 (en) * | 2018-12-20 | 2022-01-27 | Augean Robotics, Inc. | Collaborative autonomous ground vehicle |
| CN114067309A (en) * | 2021-10-25 | 2022-02-18 | 大连理工大学 | Sweet pepper identification and picking sequence determination method based on multi-view three-dimensional reconstruction |
| CN114119574A (en) * | 2021-11-30 | 2022-03-01 | 安徽农业大学 | Picking point detection model construction method and picking point positioning method based on machine vision |
| CN114119998A (en) * | 2021-12-01 | 2022-03-01 | 成都理工大学 | Vehicle-mounted point cloud ground point extraction method and storage medium |
| CN114260895A (en) * | 2021-12-22 | 2022-04-01 | 江苏大学 | Method and system for determining grabbing obstacle avoidance direction of mechanical arm of picking machine |
| CN114387343A (en) * | 2021-12-30 | 2022-04-22 | 中南大学 | Method for detecting picking position of mushroom stem of clustered oyster mushroom |
| CN114677674A (en) * | 2022-04-15 | 2022-06-28 | 河北农业大学 | Apple rapid identification and positioning method based on binocular point cloud |
| WO2022156175A1 (en) * | 2021-01-20 | 2022-07-28 | 上海西井信息科技有限公司 | Detection method, system, and device based on fusion of image and point cloud information, and storage medium |
-
2022
- 2022-08-02 CN CN202210919966.XA patent/CN114973006B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109410238A (en) * | 2018-09-20 | 2019-03-01 | 中国科学院合肥物质科学研究院 | A kind of fructus lycii identification method of counting based on PointNet++ network |
| US20200117898A1 (en) * | 2018-10-10 | 2020-04-16 | Midea Group Co., Ltd. | Method and system for providing remote robotic control |
| US20220024486A1 (en) * | 2018-12-20 | 2022-01-27 | Augean Robotics, Inc. | Collaborative autonomous ground vehicle |
| CN110226413A (en) * | 2019-06-26 | 2019-09-13 | 佛山科学技术学院 | A kind of picking robot goes here and there the method for stacking grape more |
| WO2021168708A1 (en) * | 2020-02-26 | 2021-09-02 | 深圳市大疆创新科技有限公司 | Control method for movable platform, movable platform, device and storage medium |
| CN111882610A (en) * | 2020-07-15 | 2020-11-03 | 中国科学院自动化研究所 | Method for grabbing target object by service robot based on elliptical cone artificial potential field |
| CN111758395A (en) * | 2020-07-21 | 2020-10-13 | 四川省机械研究设计院 | Prickly ash picking device and prickly ash vision picking robot that constitutes thereof |
| CN112509145A (en) * | 2020-12-22 | 2021-03-16 | 珠海格力智能装备有限公司 | Material sorting method and device based on three-dimensional vision |
| WO2022156175A1 (en) * | 2021-01-20 | 2022-07-28 | 上海西井信息科技有限公司 | Detection method, system, and device based on fusion of image and point cloud information, and storage medium |
| CN112802199A (en) * | 2021-01-25 | 2021-05-14 | 夏永霞 | High-precision mapping point cloud data processing method and system based on artificial intelligence |
| CN112907735A (en) * | 2021-03-10 | 2021-06-04 | 南京理工大学 | Flexible cable identification and three-dimensional reconstruction method based on point cloud |
| CN114067309A (en) * | 2021-10-25 | 2022-02-18 | 大连理工大学 | Sweet pepper identification and picking sequence determination method based on multi-view three-dimensional reconstruction |
| CN114119574A (en) * | 2021-11-30 | 2022-03-01 | 安徽农业大学 | Picking point detection model construction method and picking point positioning method based on machine vision |
| CN114119998A (en) * | 2021-12-01 | 2022-03-01 | 成都理工大学 | Vehicle-mounted point cloud ground point extraction method and storage medium |
| CN114260895A (en) * | 2021-12-22 | 2022-04-01 | 江苏大学 | Method and system for determining grabbing obstacle avoidance direction of mechanical arm of picking machine |
| CN114387343A (en) * | 2021-12-30 | 2022-04-22 | 中南大学 | Method for detecting picking position of mushroom stem of clustered oyster mushroom |
| CN114677674A (en) * | 2022-04-15 | 2022-06-28 | 河北农业大学 | Apple rapid identification and positioning method based on binocular point cloud |
Non-Patent Citations (3)
| Title |
|---|
| HANWEN KANG 等: "Geometry-Aware Fruit Grasping Estimation for Robotic Harvesting in Orchards", 《ARXIV:2112.04363V1》 * |
| 张志会 等: "基于深度学习的花椒采摘器视觉识别", 《机械》 * |
| 王青 等: "增强的OHTA颜色空间融合FPFH识别点云中的甜椒果实", 《重庆师范大学学报（自然科学版）》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116652951A (en) * | 2023-06-08 | 2023-08-29 | 广东技术师范大学 | Method and device for positioning robot vision in unstructured large working space |
| CN116652951B (en) * | 2023-06-08 | 2024-04-05 | 广州鑫帅机电设备有限公司 | Method and device for positioning robot vision in unstructured large working space |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114973006B (en) | 2022-10-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109241820B (en) | Unmanned aerial vehicle autonomous shooting method based on space exploration | |
| CN110495819B (en) | Robot control method, robot, terminal, server and control system | |
| CN110276344A (en) | A kind of method of image segmentation, the method for image recognition and relevant apparatus | |
| CN109325967A (en) | Method for tracking target, device, medium and equipment | |
| CN109325456B (en) | Target identification method, target identification device, target identification equipment and storage medium | |
| CN105158253A (en) | Identification method of crop maturity | |
| CN110942479B (en) | Virtual object control method, storage medium and electronic device | |
| US11850747B2 (en) | Action imitation method and robot and computer readable medium using the same | |
| CN107907124A (en) | The localization method known again based on scene, electronic equipment, storage medium, system | |
| CN101683037A (en) | Controller for fruit picking robot | |
| CN114973006B (en) | Method, device and system for picking Chinese prickly ash and storage medium | |
| CN114683290B (en) | Method and device for optimizing pose of foot robot and storage medium | |
| CN113761255B (en) | Robot indoor positioning method, device, equipment and storage medium | |
| CN104535575A (en) | Crop maturity identification platform based on unmanned aerial vehicle detection | |
| CN112990101B (en) | Facial organ positioning method based on machine vision and related equipment | |
| CN114693865A (en) | Data processing method and related device | |
| CN112016447A (en) | Intelligent forehead temperature measurement method based on Yolo neural network and application thereof | |
| CN115670308A (en) | Sweeping method, device, equipment and storage medium of sweeping robot | |
| CN115081643A (en) | Countermeasure sample generation method, related device and storage medium | |
| CN114967495A (en) | Orchard virtual simulation inspection system and method based on Internet of things cloud control platform | |
| Strecker et al. | MR Object Identification and Interaction: Fusing Object Situation Information from Heterogeneous Sources | |
| US10849264B1 (en) | Determining activity swath from machine-collected worked data | |
| CN112862814A (en) | Alignment method, alignment device, underwater vehicle and storage medium | |
| CN113221729A (en) | Unmanned aerial vehicle cluster control method and system based on gesture human-computer interaction | |
| CN117340914B (en) | Humanoid robot human body feeling control method and control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |