CN114830915B - Litchi vision picking robot based on laser radar navigation and implementation method thereof - Google Patents
Litchi vision picking robot based on laser radar navigation and implementation method thereof Download PDFInfo
- Publication number
- CN114830915B CN114830915B CN202210384198.2A CN202210384198A CN114830915B CN 114830915 B CN114830915 B CN 114830915B CN 202210384198 A CN202210384198 A CN 202210384198A CN 114830915 B CN114830915 B CN 114830915B
- Authority
- CN
- China
- Prior art keywords
- litchi
- camera
- picking
- end effector
- binocular
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J11/00—Manipulators not otherwise provided for
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
- G01C11/04—Interpretation of pictures
- G01C11/30—Interpretation of pictures by triangulation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Electromagnetism (AREA)
- Computer Networks & Wireless Communication (AREA)
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Multimedia (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a litchi vision picking robot based on laser radar navigation and a realization method thereof, wherein the litchi vision picking robot comprises a navigation mechanism, a carrying mechanism, a lifting mechanism, a power supply, a mechanical arm, a binocular vision mechanism, an end execution mechanism, a control cabinet and a fruit collecting device; the navigation mechanism is arranged at the front end of the vehicle body bottom plate of the carrying mechanism; the lifting mechanism is arranged on a vehicle body bottom plate of the carrying mechanism; the mechanical arm, the power supply and the control cabinet are arranged on a working platform of the lifting mechanism; the binocular vision mechanism is arranged on the end effector fixing piece; the tail end executing mechanism is arranged at the tail end of the mechanical arm; the fruit collecting device is arranged on the side surface of the lifting mechanism. The litchi stem picking machine can accurately identify litchi stems, automatically pick the litchi stems, has high picking efficiency, and has a simple integral structure and high automation and intelligent degrees.
Description
Technical Field
The invention belongs to the field of agricultural machinery, and particularly relates to a litchi vision picking robot based on laser radar navigation and an implementation method thereof.
Background
China is the most important litchi production country worldwide, but picking and harvesting of litchi requires a lot of labor force, the litchi maturity period is very short, and meanwhile, the weather in the south of Ling is hot and rainy, so that serious economic loss can be caused if picking cannot be performed in time. Currently, some litchi picking machines, such as patent CN109566097a, disclose a litchi picking machine, in which when picking claws perform picking actions, a plurality of claw fingers which are arranged on claw finger brackets in parallel with each other can clamp branches without fruits in the picking process, so that the picking efficiency is low, autonomous navigation cannot be realized, and no vision module is carried, so that automatic picking cannot be realized.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a litchi vision picking robot based on laser radar navigation and an implementation method thereof, which can accurately identify litchi stalks and automatically pick the litchi stalks, and has the advantages of high picking efficiency, simple overall structure and high automation and intelligent degree.
The aim of the invention is achieved by the following technical scheme:
the litchi vision picking robot based on laser radar navigation comprises a navigation mechanism 1, a carrying mechanism 2, a lifting mechanism 3, a power supply 4, a mechanical arm 5, a binocular vision mechanism 6, an end execution mechanism 7, a control cabinet 8 and a fruit collecting device 9; the navigation mechanism 1 is arranged at the front end of a vehicle body bottom plate of the carrying mechanism 2; the lifting mechanism 3 is arranged on the vehicle body bottom plate of the carrying mechanism 2; the mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on a working platform of the lifting mechanism 3; the binocular vision mechanism 6 is mounted on the end effector mount 34; the end actuating mechanism 7 is arranged at the end of the mechanical arm; the fruit collecting device 9 is mounted on the side of the lifting mechanism.
The navigation mechanism 1 comprises a laser radar and an encoder, wherein the laser radar is arranged at the front end of a vehicle body bottom plate of the carrying mechanism, and the laser radar and the encoder are used for acquiring data information of a road.
The lifting mechanism 3 comprises a base 11, a crossed lifting assembly 12, a hydraulic driving assembly 10 and a working platform 13; the hydraulic driving assembly 10 is fixed on the base 11, and the hydraulic driving assembly 10 is respectively connected with the cross type lifting assembly 12 and the control cabinet 8 and is used for driving the cross type lifting assembly 12 to do lifting motion; the working platform 13 is horizontally arranged on the crossed lifting assembly 12; the fruit collecting device 9 is arranged on the side surface of the working platform of the lifting mechanism.
The mechanical arm 5 comprises a first screw 14, a second screw 15, a third screw 16, a fourth screw 17, a fifth screw 18, a sixth screw 19, a first push rod 20, a second push rod 21, a third push rod 22, a fourth push rod 23, a first connecting piece 24 and a second connecting piece 25; the mechanical arms 5 are two mechanical arms with the same structure and are respectively arranged at the diagonal positions of the lifting mechanism working platform 13; one set of mechanical arm is characterized in that a first screw rod 14 and a second screw rod 15 are vertically fixed on a lifting mechanism working platform 13 along the advancing direction of a carrying mechanism, two ends of a third screw rod 16 are respectively and horizontally fixed on sliding tables of the first screw rod 14 and the second screw rod 15, a first push rod 20 and a second push rod 21 are symmetrically and vertically fixed on the third screw rod 16, an end effector is connected to the tail end of the first push rod 20 through an end effector fixing piece 34, and then the end effector fixing piece is connected with the second push rod 21 through a first connecting piece 24; the other set of mechanical arms which are diagonally arranged is of the same structure.
The end effector 7 comprises an end effector fixing piece 34 and an end effector; the end effector comprises a motor 29, an engagement blade 30, a rubber clamping plate 31, a clamping plate fixing piece 32, a structure fixing piece 33, a screw nut 35 and a push rod 36; the engagement blade 30 is rigidly connected to the clamp plate holder 32, the rubber clamp plate 31 is mounted under the engagement blade 30 by the clamp plate holder 32, the motor 29 is connected to the structural holder 33, the spindle nut 35 is connected to the spindle of the motor 29, the push rod 36 is fixed to the spindle nut 35, and the engagement blade 30 is connected to the end of the push rod 36. When picking operation is carried out, the meshing blade and the clamping part, namely the rubber clamping plate 31 are relatively static, when the motor works, the screw nut 35 realizes the mutual conversion between the rotation motion and the linear motion by the rotation of the screw of the motor 29, and the push rod 36 is driven to provide power for the meshing blade 30 under the reciprocating linear motion of the screw nut 35 so as to realize the shearing motion; the rubber clamping plate 31 is engaged with the engaging blade 30 and simultaneously clamps the litchi fruit stalks. The engaging blade 30 is designed through the serrated structure of the bionic teeth, the rubber clamping plate 31 is designed to be similar to the shape of the biological lips, and the clamping and shearing integrated structure of the bionic mouth structure has the advantages of realizing processability, being favorable for positioning litchi peduncles and preventing the peduncles from sliding during engagement and shearing. The end effector designed by the invention can pick the litchi fruits and branches and leaves on the premise of not damaging the litchi fruits and branches and leaves, ensures the integrity of the litchi fruits and branches and leaves, can be better suitable for litchi stalks with different thicknesses, has a compact structure, can be prevented from being influenced by other branches during clamping and shearing, and can stably clamp litchi strings without falling.
The binocular vision mechanism 6 includes a left camera 27, a right camera 28, a camera mount 26 on which the left and right cameras are mounted, the camera mount 26 being mounted on an end effector mount 34.
A method for realizing litchi vision picking robot based on laser radar navigation comprises the following steps:
(1) Laser radar navigation: the laser radar and the encoder are used for acquiring data information of the road, positioning, map construction and path planning of the robot are carried out through the acquired data information, and the carrying mechanism accurately reaches a designated position after receiving the instruction;
(2) Lifting mechanism goes up and down: aiming at different picking environments and litchi varieties, the lifting mechanism is controlled to reach a proper height to perform picking operation;
(3) And (3) data set preparation: firstly, a large number of litchi images are collected by a camera, and then the collected images are marked by a marking tool, namely, litchi stems to be picked are selected by a frame, so that a litchi stem data set is obtained;
(4) Deep learning model: training the litchi fruit stalk data set by using an improved YOLOV4 network to obtain a network model for identifying litchi fruit stalks;
(5) Monocular and binocular calibration of binocular cameras and hand-eye calibration: firstly, monocular calibration is carried out on two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, binocular stereoscopic vision calibration is carried out on the two cameras to obtain a re-projection matrix for binocular correction, and meanwhile, the conversion relation between the real object distance and the camera pixel distance under the world coordinate system is obtained; performing binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration is used for determining the coordinate system conversion relation between the robot and the camera;
(6) Three-dimensional reconstruction: obtaining a color image of a litchi image through a binocular camera, obtaining a depth image through stereo matching, inputting the color image into the trained network model in the step (4), obtaining the position coordinates of picking points of the litchi stems, and obtaining three-dimensional point clouds of the litchi stems according to a triangle ranging principle;
(7) Planning a motion trail: transmitting the three-dimensional point cloud information of the litchi stems to a control cabinet, and planning the motion track of the mechanical arm by the control cabinet through analyzing the spatial three-dimensional information contained in the point cloud information and adopting an obstacle avoidance algorithm to enable the motion track to be close to a picking target;
(8) Posture estimation: because the fruit stalk direction of the litchi is not vertical and downward, the fruit stalk axis direction vector n and the direction vector m vertical to the n are required to be determined through the obtained three-dimensional point cloud of the litchi fruit stalk;
(9) Picking and collecting: the end effector clamps and shears the picking points from the direction of the vector m by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit cluster is put into the fruit collecting device.
In the step (1), the laser radar navigation means that a carrying mechanism utilizes a sensor of the carrying mechanism to identify the position of the carrying mechanism in the environment and construct an environment-based map, and meanwhile, a positioning navigation function is realized based on the map, and the realization of the function is that a map technology is simultaneously positioned and constructed.
In the step (4), the modified YOLOV4 network refers to a characteristic diagram of different receptive fields obtained by convolution, and the kernel size is set to be 1×1,3×3 and 5×5 respectively. The original YOLOV4 network adopts the operation of maximum pooling in a pyramid pooling structure to obtain the characteristic diagrams of different receptive fields, but the maximum pooling can not acquire multi-scale object information and can lose a lot of detail information about image boundaries, so the invention adopts convolution to realize the characteristic diagrams of different receptive fields, after improvement, the problem that the size of an input image is required to be fixed by the convolution neural network is solved, and the receptive fields are enlarged in a phase change way by introducing different void ratios, and the output results are fused.
In the step (5), camera calibration refers to a process of solving parameters of a camera model. Firstly, monocular calibration is carried out to respectively obtain an outer parameter matrix, an inner parameter matrix and a distortion matrix of a left camera and a right camera, double-target calibration is carried out on the basis of monocular calibration to obtain matrix parameters such as a double-target calibration re-projection matrix, a mapping table and the like, and finally, the correlation between the three-dimensional geometric position of a certain point on the surface of a space object and a corresponding point in an image is obtained. The purpose of solving the conversion relation between the camera coordinate system and the robot coordinate system is to: and converting the three-dimensional coordinates of the litchi based on the camera coordinate system into the three-dimensional coordinates of the litchi based on the robot coordinate system, calculating the motion postures of the lead screw and the push rod according to the coordinates, and controlling the end effector to reach the designated position.
In the step (6), the three-dimensional reconstruction based on binocular vision is an important component of image processing technology and machine vision. Because the image shot by the monocular camera is two-dimensional, a binocular camera is required to simulate a human visual system, the distance between the position of a certain point on the surface of a space object and the optical center of two lenses is calculated according to the principle of similar triangles and then the three-dimensional coordinates of the point cloud on the surface of the object are obtained according to the parallax of the two image matching point pairs shot by the left camera and the right camera. The three-dimensional reconstruction is a process of reconstructing the surface of the target object by utilizing the three-dimensional coordinates of the point cloud on the surface of the object, so that the picking speed of the robot can be increased, the efficiency is improved, and the robot is more intelligent.
Compared with the prior art, the invention has the following advantages and effects:
(1) The invention uses an improved YOLOV4 network, and adopts convolution to replace the operation of maximum pooling in the original network space pyramid pooling structure to obtain the characteristic diagrams of different receptive fields. Because the multi-scale object information can not be acquired through the maximum pooling and a lot of detail information about the image boundary can be lost, the invention is realized by convolution, the kernel size of the convolution is respectively set to be 1 multiplied by 1,3 multiplied by 3 and 5 multiplied by 5, the problem that the size of the input image of the convolution neural network is fixed is solved after improvement, the receptive field is enlarged by phase change through introducing different void ratios, and the output result is fused. In order to solve the problem of missed detection of the small target litchi by the YOLOV4, a method for sampling and training only the small target litchi and then performing mixed training with pictures shot under other large fields is provided, so that the recognition rate of the small target litchi by the YOLOV4 is improved.
(2) The end effector used in the invention has the advantages that the saw tooth structure of the bionic teeth is used for designing the clamping and shearing integrated structure that the meshing blades are meshed and cut off the litchi peduncles and the rubber material is used for clamping the litchi peduncles, so that the processing performance can be realized. The end effector with the bionic mouth structure can pick the litchi fruit and the branches and leaves under the premise of ensuring that the litchi fruit and the branches and leaves are not damaged, the integrity of the litchi fruit and the branches and leaves is ensured, the litchi fruit stalks with different thicknesses can be better adapted, and the jagged structure and the rubber material clamp can stably clamp the litchi strings and cannot fall. The simulated milling machine type mechanical arm designed by the invention has the advantages of simple structure, good symmetrical load and force balance of the mechanical arms at two sides and low use cost, avoids the defects of large working space, high cost, increased accumulated error, poor rigidity and the like required by the mechanical arm connected in series with 6 shafts, can realize the simultaneous picking of fruits at two sides and improves the picking efficiency.
(3) When the end effector is used for clamping and shearing picking, as the fruit stalk direction of the litchi is not vertical and downward, the axis direction vector n and the direction vector m vertical to the axis direction vector n of the fruit stalk are required to be determined through the obtained three-dimensional point cloud of the litchi fruit stalk, and the picking angle of the end effector is innovatively changed by controlling the expansion and contraction amount of the push rod, so that the end clamping and shearing mechanism is used for picking the vertical direction of the axis direction of the obtained litchi fruit stalk.
Drawings
Fig. 1 is a schematic diagram of the overall structure of the litchi vision picking robot.
Fig. 2 is a front view of the litchi vision picking robot of the present invention.
Fig. 3 is a side view of the litchi vision picking robot of the present invention.
Fig. 4 is a top view of the litchi vision picking robot of the present invention.
Fig. 5 is a schematic structural view of a mechanical arm according to the present invention.
FIG. 6 is a schematic diagram of an end effector of the present invention.
Fig. 7 is a schematic view of the structure of the end effector of the present invention.
Fig. 8 is a laser radar navigation flow chart of the present invention.
Fig. 9 is a flow chart of a visual recognition algorithm of the present invention.
Fig. 10 is a modified YOLOV4 network SPP architecture of the present invention.
The device comprises a 1-navigation mechanism, a 2-carrying mechanism, a 3-lifting mechanism, a 4-power supply, a 5-mechanical arm, a 6-binocular vision mechanism, a 7-end actuating mechanism, an 8-control cabinet, a 9-fruit collecting device, a 10-hydraulic driving assembly, a 11-base, a 12-crossed lifting assembly, a 13-working platform, a 14-first screw rod, a 15-second screw rod, a 16-third screw rod, a 17-fourth screw rod, a 18-fifth screw rod, a 19-sixth screw rod, a 20-first push rod, a 21-second push rod, a 22-third push rod, a 23-fourth push rod, a 24-first connecting piece, a 25-second connecting piece, a 26-camera fixing frame, a 27-left camera, a 28-right camera, a 29-motor, a 30-meshing blade, a 31-rubber clamping plate, a 32-clamping plate fixing piece, a 33-structure fixing piece, a 34-end effector fixing piece, a 35-screw rod nut and a 36-push rod.
Detailed Description
In order that the invention may be readily understood, a detailed description of the invention will be provided below with reference to specific examples. The following examples will assist those skilled in the art in further understanding the present invention, but are not intended to limit the invention in any way. It should be noted that the present invention can be modified and improved by those skilled in the art without departing from the spirit of the present invention, which falls within the scope of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
Examples
As shown in fig. 1 to 4, the litchi automatic picking robot based on laser radar navigation comprises a navigation mechanism 1, a lifting mechanism 3, a binocular vision mechanism 6, a carrying mechanism 2, a mechanical arm 5, an end execution mechanism 7, a power supply 4, a control cabinet 8 and a fruit collecting device 9; the laser radar for navigation is arranged at the front end of the bottom plate of the vehicle body of the carrying mechanism 2; the lifting mechanism 3 is arranged on the bottom plate of the vehicle body of the carrying mechanism 2; the mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on a working platform of the lifting mechanism 3; the binocular vision mechanism 6 is mounted on the end effector mount 34; the end actuating mechanism 7 is arranged at the tail end of the mechanical arm; the fruit collecting device 9 is arranged on the side surface of the lifting platform. As shown in fig. 1, the laser radar is installed at the front end of the vehicle body bottom plate of the carrying mechanism, the laser radar and the encoder are responsible for acquiring data information of a road, and the carrying mechanism is responsible for executing a motion instruction. The robot performs functions such as positioning, map construction, and path planning of the robot by using the obtained data. The elevating system installs on the transport mechanism automobile body bottom plate, elevating system 3 includes base 11, crossing elevating system 12, hydraulic drive subassembly 10 and work platform 13, hydraulic drive subassembly 10 is fixed on base 11, and hydraulic drive subassembly 10 is connected with crossing elevating system 12, switch board 8 respectively for drive elevating system 12 and do elevating movement, work platform 13 level sets up on crossing elevating system 12, and fruit collection device 9 installs in elevating system work platform side. The mechanical arm 5, the power supply 4 and the control cabinet 8 are arranged on the working platform 13. As shown in fig. 5, the mechanical arm 5 includes a first screw 14, a second screw 15, a third screw 16, a fourth screw 17, a fifth screw 18, a sixth screw 19, a first push rod 20, a second push rod 21, a third push rod 22, a fourth push rod 23, a first connection member 24, a second connection member 25, an end effector fixing member 34, and an end effector. Two mechanical arms with the same structure are assembled together by the parts and are respectively arranged at the diagonal positions of the working platform 13 of the lifting mechanism. In recent years, agricultural planting in China is more and more prone to standardized planting, especially in fruit and vegetable planting, so that the mechanical arm can pick fruits at two sides simultaneously, and the picking efficiency is greatly improved. One set of mechanical arm is characterized in that a first screw rod 14 and a second screw rod 15 are vertically fixed on a lifting mechanism working platform 13 along the advancing direction of a carrying mechanism, two ends of a third screw rod 16 are respectively and horizontally fixed on sliding tables of the first screw rod 14 and the second screw rod 15, a first push rod 20 and a second push rod 21 are symmetrically and vertically fixed with the third screw rod 16, an end effector is connected to the tail end of the first push rod 20 through an end effector fixing piece 34, then the end effector fixing piece is connected with the second push rod 21 through a first connecting piece 24, and the mechanical arm which is obliquely and diagonally placed with the end effector is of the same structure. As shown in fig. 6, the end effector 7 comprises an end effector fixing part 34 and an end effector, as shown in fig. 7, the end effector is composed of a motor 29, an engagement blade 30, a rubber clamping plate 31, a clamping plate fixing part 32, a structure fixing part 33, a screw nut 35 and a push rod 36, the engagement blade 30 is fixed on the structure fixing part 33 through bolts, the motor 29 rotates to drive the engagement blade 30 to open or close, the rubber clamping plate 31 is arranged below the engagement blade 30 through the clamping plate fixing part 32, the motor 29 is connected with the structure fixing part 33, the screw nut 35 is connected with a screw of the motor 29, the push rod 36 is fixed on the screw nut 35, the engagement blade 30 is connected with the tail end of the push rod 36, and the rubber clamping plate 31 is used for cutting while clamping litchi stems. The binocular vision mechanism 6 includes a left camera 27, a right camera 28, a camera mount 26 on which the left and right cameras are mounted, the camera mount 26 being mounted on an end effector mount 34. The end effector clamps and shears the picking points from the direction perpendicular to the axis of the litchi peduncles by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit cluster is put into the collecting device 9.
The litchi vision picking robot based on laser radar navigation and the implementation method thereof comprise the following steps:
(1) Laser radar navigation: the laser radar and the encoder are responsible for acquiring data information of a road, in a robot software system, the robot utilizes the acquired data to perform functions of positioning, map construction, path planning and the like of the robot, and the carrying mechanism can accurately reach a designated position after receiving an instruction; as shown in fig. 8;
(2) Lifting a picking platform: due to the influence of different environments and litchi varieties, the lifting mechanism 3 needs to be controlled to reach a proper height to perform picking operation;
(3) And (3) data set preparation: firstly, a large number of litchi images are collected by a camera, and then the collected images are marked by a marking tool, namely litchi stems to be picked are selected by a frame;
(4) Deep learning model: training the data set in the step (3) by using an improved YOLOV4 network to obtain a better network model for identifying fruit stalks; as shown in fig. 10;
(5) Monocular and binocular calibration of binocular cameras and hand-eye calibration: firstly, monocular calibration is carried out on two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, binocular stereoscopic vision calibration is carried out on the two cameras to obtain a re-projection matrix for binocular correction, and meanwhile, the conversion relation between the real object distance and the camera pixel distance under the world coordinate system is obtained; performing binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration can be used for determining the coordinate system conversion relation between the robot and the camera;
(6) Three-dimensional reconstruction: obtaining a color image of the litchi image through a binocular camera, obtaining a depth image through stereo matching, inputting the color image into the trained network model in the step (4), obtaining the position coordinates of picking points of the litchi stems, and obtaining the three-dimensional point cloud of the litchi stems according to the principle of triangular ranging; as shown in fig. 9;
(7) Planning a motion trail: transmitting three-dimensional point cloud information of litchi stems to a control cabinet 8, wherein the control cabinet 8 adopts an obstacle avoidance algorithm to plan a movement track of the mechanical arm 5 by analyzing the spatial three-dimensional information contained in the point cloud data so as to enable the movement track to be close to a picking target;
(8) Posture estimation: because the fruit stalk direction of the litchi is not vertical and downward, the fruit stalk axis direction vector n and the direction vector m vertical to the n are required to be determined through the obtained three-dimensional point cloud of the litchi fruit stalk;
(9) Picking and collecting: the end effector clamps and shears the picking points from the direction of vector m by controlling the extension of the first push rod 20 and the second push rod 21, and the picked litchi fruit cluster is put into the collecting device 9.
The working principle of the invention is as follows: automatic driving of the picking robot in an orchard is achieved through a laser radar, litchi stems are identified through an improved YOLOV4 network frame according to the height of a lifting platform controlled by the height of a tree, color images of litchi images are obtained through a binocular camera, a depth map is obtained through three-dimensional matching, three-dimensional point clouds of the litchi stems are obtained according to a triangular ranging principle, position coordinates of picking points of the litchi stems are obtained, an end actuating mechanism is controlled to clamp and shear the picking points in a direction perpendicular to the axis of the litchi stems according to a detected target, and picked litchi fruit strings are placed in a collecting device.
The foregoing is illustrative of the present invention, and the present invention is not limited to the above embodiments, but is capable of other modifications, adaptations, alternatives, combinations, and simplifications without departing from the spirit and principles of the invention.
Claims (4)
1. Litchi vision picking robot based on laser radar navigation, its characterized in that: comprises a navigation mechanism, a carrying mechanism, a lifting mechanism, a power supply, a mechanical arm, a binocular vision mechanism, an end execution mechanism, a control cabinet and a fruit collecting device; the navigation mechanism is arranged at the front end of a vehicle body bottom plate of the carrying mechanism and comprises a laser radar and an encoder, and the laser radar and the encoder are used for acquiring data information of a road; the lifting mechanism is arranged on a vehicle body bottom plate of the carrying mechanism; the mechanical arm, the power supply and the control cabinet are arranged on a working platform of the lifting mechanism; the mechanical arm is characterized in that a first screw rod and a second screw rod are vertically fixed on a lifting mechanism working platform along the advancing direction of a carrying mechanism, two ends of a third screw rod are respectively and horizontally fixed on sliding tables of the first screw rod and the second screw rod, and a first push rod and a second push rod are symmetrically and vertically fixed on the upper side and the lower side of the sliding table of the third screw rod; the binocular vision mechanism comprises a left camera, a right camera and a camera fixing frame, wherein the left camera and the right camera are arranged on the camera fixing frame; the end effector comprises an end effector fixing piece and an end effector, the end effector is connected to the tail end of the first push rod through the end effector fixing piece, then the end effector fixing piece is connected with the second push rod through the first connecting piece, and the camera fixing frame is arranged on the end effector fixing piece; the end effector comprises a motor, a meshing blade, a rubber clamping plate, a clamping plate fixing piece, a structure fixing piece, a screw rod nut and a push rod, wherein the meshing blade is rigidly connected with the clamping plate fixing piece; the meshing blade is of a saw-tooth structure of the bionic teeth, and the rubber splint is in the shape of the bionic lips; the fruit collecting device is arranged on the side surface of the lifting mechanism;
the three-dimensional point cloud of the litchi fruit stalks is obtained through a binocular camera to determine the axial direction vector n of the fruit stalks and the direction vector m perpendicular to the axial direction vector n, and the end effector clamps and shears picking points from the direction of the vector m by controlling the extension of the first push rod and the second push rod.
2. The laser radar navigation-based litchi vision picking robot as claimed in claim 1, wherein: the lifting mechanism comprises a base, a crossed lifting assembly, a hydraulic driving assembly and a working platform; the hydraulic driving assembly is fixed on the base and is respectively connected with the cross type lifting assembly and the control cabinet and used for driving the cross type lifting assembly to do lifting motion; the working platform is horizontally arranged on the cross lifting assembly.
3. The laser radar navigation-based litchi vision picking robot as claimed in claim 1, wherein: the mechanical arms are two mechanical arms with the same structure and are respectively arranged at the diagonal positions of the working platform of the lifting mechanism.
4. A method for implementing the litchi vision picking robot based on laser radar navigation according to any one of claims 1-3, which is characterized by comprising the following steps:
(1) Laser radar navigation: the laser radar and the encoder are used for acquiring data information of the road, positioning, map construction and path planning of the robot are carried out through the acquired data information, and the carrying mechanism accurately reaches a designated position after receiving the instruction;
(2) Lifting mechanism goes up and down: aiming at different picking environments and litchi varieties, the lifting mechanism is controlled to reach a proper height to perform picking operation;
(3) And (3) data set preparation: firstly, a large number of litchi images are collected by a camera, and then the collected images are marked by a marking tool, namely, litchi stems to be picked are selected by a frame, so that a litchi stem data set is obtained;
(4) Deep learning model: training the litchi fruit stalk data set by using an improved YOLOV4 network to obtain a network model for identifying litchi fruit stalks; the improved YOLOV4 network is a characteristic diagram of different receptive fields obtained by adopting convolution, wherein the kernel size is respectively set to be 1 multiplied by 1,3 multiplied by 3 and 5 multiplied by 5;
(5) Monocular and binocular calibration of binocular cameras and hand-eye calibration: firstly, monocular calibration is carried out on two cameras respectively to obtain an internal reference matrix and a distortion matrix of each camera; then, binocular stereoscopic vision calibration is carried out on the two cameras to obtain a re-projection matrix for binocular correction, and meanwhile, the conversion relation between the real object distance and the camera pixel distance under the world coordinate system is obtained; performing binocular correction on the litchi image shot by the binocular camera to obtain a binocular corrected image; the hand-eye calibration is used for determining the coordinate system conversion relation between the robot and the camera;
(6) Three-dimensional reconstruction: obtaining a color image of a litchi image through a binocular camera, obtaining a depth image through stereo matching, inputting the color image into the trained network model in the step (4), obtaining the position coordinates of picking points of the litchi stems, and obtaining three-dimensional point clouds of the litchi stems according to a triangle ranging principle;
(7) Planning a motion trail: transmitting the three-dimensional point cloud information of the litchi stems to a control cabinet, and planning the motion track of the mechanical arm by the control cabinet through analyzing the spatial three-dimensional information contained in the point cloud information and adopting an obstacle avoidance algorithm to enable the motion track to be close to a picking target;
(8) Posture estimation: because the fruit stalk direction of the litchi is not vertical and downward, the fruit stalk axis direction vector n and the direction vector m vertical to the n are required to be determined through the obtained three-dimensional point cloud of the litchi fruit stalk;
(9) Picking and collecting: the end effector clamps and shears the picking points from the direction of the vector m by controlling the extension of the first push rod and the second push rod, and the picked litchi fruit cluster is put into the fruit collecting device.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210384198.2A CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210384198.2A CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114830915A CN114830915A (en) | 2022-08-02 |
| CN114830915B true CN114830915B (en) | 2023-09-26 |
Family
ID=82563866
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210384198.2A Active CN114830915B (en) | 2022-04-13 | 2022-04-13 | Litchi vision picking robot based on laser radar navigation and implementation method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114830915B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115316129B (en) * | 2022-08-12 | 2024-03-12 | 华南农业大学 | Self-adaptive bionic picking device based on binocular vision recognition and fruit stringing picking method |
| CN115589849B (en) * | 2022-10-18 | 2023-06-30 | 大连理工大学 | Spherical fruit and vegetable picker |
Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004180554A (en) * | 2002-12-02 | 2004-07-02 | National Agriculture & Bio-Oriented Research Organization | Method and apparatus for selectively harvesting fruit vegetables |
| CN107094429A (en) * | 2017-07-03 | 2017-08-29 | 彭紫薇 | A kind of multiaxis fruit picking robot |
| CN207678305U (en) * | 2017-12-18 | 2018-08-03 | 艾力·艾散 | A kind of mechanical arm for walnut picking |
| CN108557143A (en) * | 2018-05-01 | 2018-09-21 | 浙江工业大学 | A kind of fruit picking packing machine |
| CN108575320A (en) * | 2018-05-23 | 2018-09-28 | 芮杰 | Strawberry picking robot |
| CN208047324U (en) * | 2018-04-17 | 2018-11-06 | 石河子大学 | A kind of safflower picking end effector |
| CN208079866U (en) * | 2018-04-16 | 2018-11-13 | 苏州大学 | A kind of fruit automatic picking apparatus |
| CN108834576A (en) * | 2018-06-07 | 2018-11-20 | 华南农业大学 | A kind of citrus picking robot and its implementation based on binocular vision |
| CN109328643A (en) * | 2018-12-17 | 2019-02-15 | 青岛科技大学 | A kind of self-balancing type apple-picking vehicle |
| JP2019037214A (en) * | 2017-08-29 | 2019-03-14 | 株式会社デンソー | End effector |
| CN109743966A (en) * | 2019-02-18 | 2019-05-14 | 德施普科技发展温州有限公司 | A kind of fruit picking |
| CN209047050U (en) * | 2018-08-17 | 2019-07-02 | 长沙理工大学 | A kind of Portable electric fruit picker |
| CN110832991A (en) * | 2019-10-24 | 2020-02-25 | 广州大学 | Picking machine based on image recognition and control method thereof |
| CN211020054U (en) * | 2019-09-16 | 2020-07-17 | 中国农业科学院农业信息研究所 | Fruit and vegetable picking device |
| CN211415245U (en) * | 2019-12-26 | 2020-09-04 | 青岛悟牛智能科技有限公司 | Picking robot |
| CN111673755A (en) * | 2020-06-24 | 2020-09-18 | 西北农林科技大学 | Picking robot control system and method based on visual servo |
| CN111709489A (en) * | 2020-06-24 | 2020-09-25 | 广西师范大学 | Citrus identification method based on improved YOLOv4 |
| CN111742709A (en) * | 2020-04-23 | 2020-10-09 | 华南农业大学 | Banana picking device and method based on depth vision |
| CN112425373A (en) * | 2020-12-02 | 2021-03-02 | 陕西中建建乐智能机器人股份有限公司 | Kiwi fruit picking and sorting robot and kiwi fruit sorting method thereof |
| CN113115638A (en) * | 2019-12-31 | 2021-07-16 | 季华实验室 | Intelligent tea picking robot |
| CN113273395A (en) * | 2021-05-21 | 2021-08-20 | 佛山市中科农业机器人与智慧农业创新研究院 | Cotton topping robot based on visual identification and implementation method thereof |
| CN113408410A (en) * | 2021-06-18 | 2021-09-17 | 重庆科技学院 | Traffic sign detection method based on YOLOv4 algorithm |
| CN214508055U (en) * | 2021-03-09 | 2021-10-29 | 甄圣超 | Mobile picking robot based on vision |
| CN113692854A (en) * | 2021-09-10 | 2021-11-26 | 江苏理工学院 | Tomato picking machine and control method thereof |
| EP3918904A1 (en) * | 2020-06-02 | 2021-12-08 | Exel Industries | Automated method for selecting and harvesting fruit and mechanical equipment implementing the method |
| CN113971775A (en) * | 2021-10-14 | 2022-01-25 | 江苏省电力试验研究院有限公司 | Optimized yolov4 algorithm-based violation behavior identification method and system |
| CN216087713U (en) * | 2021-11-10 | 2022-03-22 | 双髻鲨(威海)机器人科技有限公司 | Robot for automatically picking oranges |
-
2022
- 2022-04-13 CN CN202210384198.2A patent/CN114830915B/en active Active
Patent Citations (27)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2004180554A (en) * | 2002-12-02 | 2004-07-02 | National Agriculture & Bio-Oriented Research Organization | Method and apparatus for selectively harvesting fruit vegetables |
| CN107094429A (en) * | 2017-07-03 | 2017-08-29 | 彭紫薇 | A kind of multiaxis fruit picking robot |
| JP2019037214A (en) * | 2017-08-29 | 2019-03-14 | 株式会社デンソー | End effector |
| CN207678305U (en) * | 2017-12-18 | 2018-08-03 | 艾力·艾散 | A kind of mechanical arm for walnut picking |
| CN208079866U (en) * | 2018-04-16 | 2018-11-13 | 苏州大学 | A kind of fruit automatic picking apparatus |
| CN208047324U (en) * | 2018-04-17 | 2018-11-06 | 石河子大学 | A kind of safflower picking end effector |
| CN108557143A (en) * | 2018-05-01 | 2018-09-21 | 浙江工业大学 | A kind of fruit picking packing machine |
| CN108575320A (en) * | 2018-05-23 | 2018-09-28 | 芮杰 | Strawberry picking robot |
| CN108834576A (en) * | 2018-06-07 | 2018-11-20 | 华南农业大学 | A kind of citrus picking robot and its implementation based on binocular vision |
| CN209047050U (en) * | 2018-08-17 | 2019-07-02 | 长沙理工大学 | A kind of Portable electric fruit picker |
| CN109328643A (en) * | 2018-12-17 | 2019-02-15 | 青岛科技大学 | A kind of self-balancing type apple-picking vehicle |
| CN109743966A (en) * | 2019-02-18 | 2019-05-14 | 德施普科技发展温州有限公司 | A kind of fruit picking |
| CN211020054U (en) * | 2019-09-16 | 2020-07-17 | 中国农业科学院农业信息研究所 | Fruit and vegetable picking device |
| CN110832991A (en) * | 2019-10-24 | 2020-02-25 | 广州大学 | Picking machine based on image recognition and control method thereof |
| CN211415245U (en) * | 2019-12-26 | 2020-09-04 | 青岛悟牛智能科技有限公司 | Picking robot |
| CN113115638A (en) * | 2019-12-31 | 2021-07-16 | 季华实验室 | Intelligent tea picking robot |
| CN111742709A (en) * | 2020-04-23 | 2020-10-09 | 华南农业大学 | Banana picking device and method based on depth vision |
| EP3918904A1 (en) * | 2020-06-02 | 2021-12-08 | Exel Industries | Automated method for selecting and harvesting fruit and mechanical equipment implementing the method |
| CN111673755A (en) * | 2020-06-24 | 2020-09-18 | 西北农林科技大学 | Picking robot control system and method based on visual servo |
| CN111709489A (en) * | 2020-06-24 | 2020-09-25 | 广西师范大学 | Citrus identification method based on improved YOLOv4 |
| CN112425373A (en) * | 2020-12-02 | 2021-03-02 | 陕西中建建乐智能机器人股份有限公司 | Kiwi fruit picking and sorting robot and kiwi fruit sorting method thereof |
| CN214508055U (en) * | 2021-03-09 | 2021-10-29 | 甄圣超 | Mobile picking robot based on vision |
| CN113273395A (en) * | 2021-05-21 | 2021-08-20 | 佛山市中科农业机器人与智慧农业创新研究院 | Cotton topping robot based on visual identification and implementation method thereof |
| CN113408410A (en) * | 2021-06-18 | 2021-09-17 | 重庆科技学院 | Traffic sign detection method based on YOLOv4 algorithm |
| CN113692854A (en) * | 2021-09-10 | 2021-11-26 | 江苏理工学院 | Tomato picking machine and control method thereof |
| CN113971775A (en) * | 2021-10-14 | 2022-01-25 | 江苏省电力试验研究院有限公司 | Optimized yolov4 algorithm-based violation behavior identification method and system |
| CN216087713U (en) * | 2021-11-10 | 2022-03-22 | 双髻鲨(威海)机器人科技有限公司 | Robot for automatically picking oranges |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114830915A (en) | 2022-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Mu et al. | Design and simulation of an integrated end-effector for picking kiwifruit by robot | |
| CN114830915B (en) | Litchi vision picking robot based on laser radar navigation and implementation method thereof | |
| Hayashi et al. | Robotic harvesting system for eggplants | |
| Sarig | Robotics of fruit harvesting: A state-of-the-art review | |
| Feng et al. | Design and test of tomatoes harvesting robot | |
| CN110122073B (en) | Strawberry picking robot based on machine vision | |
| Han et al. | Strawberry harvesting robot for bench-type cultivation | |
| CN111955164B (en) | Safflower filament occlusion type parallel picking robot | |
| CN108811766B (en) | Man-machine interactive greenhouse fruit and vegetable harvesting robot system and harvesting method thereof | |
| CN110089350A (en) | A kind of Mushroom Picking Robot system and picking method | |
| CN113330915B (en) | Self-adaptive cotton harvesting method based on binocular vision recognition and intelligent mechanical harvesting device | |
| CN211745437U (en) | Robot is picked to overhead fruit vegetables intelligence | |
| Hu et al. | Simplified 4-DOF manipulator for rapid robotic apple harvesting | |
| Ji et al. | Research on key technology of truss tomato harvesting robot in greenhouse | |
| CN115316129B (en) | Self-adaptive bionic picking device based on binocular vision recognition and fruit stringing picking method | |
| CN105259832A (en) | Intelligent cotton picker control system based on multi-view stereo vision | |
| CN108093868A (en) | A kind of picker system based on embedded intelligence control and binocular machine vision | |
| CN116439018A (en) | Seven-degree-of-freedom fruit picking robot and picking method thereof | |
| CN113812262B (en) | Tea-oil camellia fruit picking robot based on machine vision | |
| Jin et al. | Intelligent tomato picking robot system based on multimodal depth feature analysis method | |
| CN108718704B (en) | Man-machine interactive field fruit and vegetable harvesting robot system and harvesting method thereof | |
| CN211020054U (en) | Fruit and vegetable picking device | |
| Libin et al. | Overview of research on agricultural robot in China | |
| Hu et al. | Design and experiment of a new citrus harvesting robot | |
| CN115139315A (en) | Grabbing motion planning method for picking mechanical arm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |