CN101273688A - Flexible picking device and method for citrus picking robot - Google Patents

Abstract

The invention relates to a method and device for picking spherical citrus fruits, which integrates multi-sensor information such as force sense, temperature sense, vision, distance, etc., and feeds back information in time to realize the flexible picking of fruits by executive components. The flexible picking device is composed of a fruit picking hardware device and a software control system. The fruit picking hardware device is composed of a computer vision system, an industrial control system, a mechanical arm, and an end effector; it uses a stereo vision system to detect and identify pickable Ripe fruit; in the world coordinates, calculate the specific position of the fruit, extract the features of the obtained fruit position information and obstacle position information, build a 3D model in the virtual environment to obtain the robot movement path, and then control the mechanical arm by the industrial computer system and the claws reach the designated position; finally, control the end effector to stretch out the suction cup to suck and pull the fruit, then fix the fruit with the claws, and then the cutting device rotates to the designated position to cut off the fruit stem.

Description

Flexible picking device and method for citrus picking robot

technical field

The invention relates to a method and device for picking spherical citrus fruits, which integrates multi-sensor information such as force sense, temperature sense, vision, distance, etc., and feeds back information in time to realize the flexible picking of fruits by executive components.

Background technique

In fruit production operations, fruit picking accounts for about 33-50% of the entire workload. Due to the complexity of picking operations, picking automation is still low. At present, the agricultural labor force will gradually transfer to other industries in the society. Countries around the world are facing the problem of population aging, and the shortage of labor resources will gradually become a reality. As an important branch of agricultural machinery, fruit harvesting robots can reduce human labor intensity, improve labor productivity and Product quality ensures that the fruit is harvested in a timely manner, and has great development potential.

When picking citrus by hand, adopt the "multi-pruning method" to pick the fruit. The first cut is to cut off the fruit pedicle 1cm away, and then the second cut is made to make the fruit pedicle even. The fruit pedicle is required to be flat and the sepals are complete. Since the ripening period of the fruit of the same tree is different, the immature fruit and branches and leaves should not be damaged when picking.

In the citrus picking robot system, the design of the end effector is a difficult point. It is required to accurately grasp the fruit and realize the separation of the fruit and the tree. How to cut off the short and hard fruit stem of the citrus is the key to the separation of the fruit and the tree. Japanese scholars have studied a summer orange harvesting robot. The device grasps the fruit with rubber fingers, and the cylinder drives the scissors to cut off the stems. If there are obstacles such as branches and leaves between the end effector and the fruit, the rubber fingers will bend prematurely when they are close to the fruit, and the fruit cannot be grasped. Bulky, affecting the overall structure of the robot. The citrus picking robot developed by American researchers has an arc-shaped blade at the end effector, which can quickly rotate 180 degrees to cut off the stalk. Because the citrus stalk is hard, if the blade is not sharp enough, it is difficult to cut it in one cut. This design is very difficult. It is easy to injure the fruit, and it is reported that 44% of the fruit is damaged when picking, which shows that this device lacks flexibility. After US patent search, Fruit Picker (5428947), Citrus Harvesting Machine (5666795), Dual Pressure Release Assembly For A Fruit Harvesting Machine (6006507), Continuously Advancing Fruit Harvesting Machine (US 6263652 mainly uses comb-shaped mobile B1) and other patents. The fruit picking arm extends into the fruit tree. When the arm is retracted, the hook installed on the picking arm hooks the fruit stem and breaks it to realize the separation of fruit and tree. This is a simple harvesting machine, which neither considers the maturity period of the fruit nor judges whether the hook is a fruit stem, so it is very easy to damage the fruit and the fruit tree. Other citrus fruit picking devices are mostly designed to facilitate manual picking, such as Fruit Picking Apparatus (US 6182431 B1), etc. The patents of domestic related citrus picking devices are also mainly designed for manual picking.

Contents of the invention

In view of the defects in the existing devices and methods, the object of the present invention is to provide a flexible picking device and method for a citrus picking robot.

Technical scheme of the present invention is as follows:

The flexible picking device of the citrus picking robot of the present invention consists of two parts: a fruit picking hardware device and a software control system.

The fruit picking hardware device is composed of computer vision system, industrial control system, robotic arm and end effector.

The end effector part includes: vacuum suction device, gripper and cutting device.

1) Robotic arm: complete the work of sending the end effector to the designated position. As shown in Figure 1, the robotic arm consists of 5 degrees of freedom rotating joints. The robotic arm includes a base (1), a waist joint (2), a shoulder (3), a shoulder joint (4), a large arm (5), an elbow Joint (6), forearm (7), wrist joint (8, 9), end effector mounting frame (10), end effector (11). Wherein the wrist joint has two degrees of freedom, which can realize rotation (8) and swing up and down (9). Each joint is composed of a gear (16), a toothed belt (17), a code disc (18), a speed reducer (19), a motor (20) and a motor driver (21).

2) Vision system: It mainly realizes the signal acquisition of non-structural working environment, completes the target and background segmentation, fruit maturity judgment and positioning functions, as shown in Figure 1. The main hardware is a binocular stereo vision system (12) installed on the base of the mechanical arm, which is connected with a host computer (13) through a USB interface to realize the collection of environmental information. The self-compiled computer software completes the identification and positioning of mature fruits, the identification of obstacles, and the planning of the robot's movement path.

3) Industrial control system: realize the motion control of each component. As shown in Fig. 1, it includes an industrial computer (14), and a plurality of motion control cards (15) are interpolated to control the movement of the joints (2, 4, 6, 8, 9) of the mechanical arm and the end effector (11).

4) Vacuum adsorption device: realize the function of absorbing and pulling fruits. As shown in Figure 2, this device end is rubber sucker (30), is installed on the sucker guide rod (47), and the sucker guide rod is fixed on the sucker support (48), and the sucker support links to each other with screw rod (49), and screw rod end and The suction cup motor (27) is connected, and the motor drives to realize the forward and backward movement of the suction cup, and the motor is fixed on the end effector mounting frame (10). When fruit is picked, the sucker stretches out and drives the vacuum pump (29) to vacuumize. When the sucker sucks the fruit, the pressure sensor (31) in the sucker detects a sudden change in the degree of vacuum and controls the sucker to recover, which can pull out the fruit in the branch. Easy to pick.

5) Claw: Complete the function of fruit grabbing and fixing. As shown in Figure 2, it is mainly composed of 3 two-joint fingers (23), installed on the palm (22), and the joint drive of the root of the finger (44) and the fingertip (43) is driven by the root of the finger driver (46) and the fingertip. The driver (24) is driven to realize the opening and closing and stretching functions of the fingers. Root of finger driver (46) detects root of finger rotation angle by angle sensor (50), and motor (56), motor driver (57), speed reducer (58) are directly connected with root of finger to control finger root (44) motion; (46) detect fingertip rotation angle by angle sensor (59), motor (54), motor driver (55), speed reducer (53) connect driving wheel (52), drive fingertip driving belt (51) control fingertip ( 43) Exercise. Each fingertip is equipped with a proximity sensor (40) to prevent the finger from colliding with obstacles. A force sensor (41) and a temperature sensor (42) are installed on the inner side of each finger for real-time detection of the grip strength of the finger and the temperature of the grasped object. Ultrasonic sensor (25) is housed in the palm (22), is used for detecting the distance of grasping object.

6) Cutting device: used to cut off the hard stalks of citrus to separate the fruit from the tree. As shown in Figure 2, the support (32) of the cutting device is fixed on the palm (22), and the blade holder (34) is driven to rotate positively and negatively by the rotating device (33) (the same finger root driver (45) structure). Motor (36), belt pulley (35) are installed on the blade holder, drives blade (38) to rotate and cut fruit stem at a high speed by belt (37). Protective cover (39) is arranged above the blade to avoid cutting surrounding branches and leaves. Because the blade cuts the fruit stem under high-speed rotation, the gripping force of the claw is less required.

The software control system mainly includes: stereo vision calibration, target information acquisition and identification, positioning, manipulator path planning, manipulator arm and end effector control and other modules, as shown in Figure 3.

1) Stereo vision calibration module: Since the baselines of the two cameras in binocular stereo vision can be changed and the lenses can be adjusted, the system needs to be calibrated before detection to obtain accurate internal and external parameters of the camera, so as to calculate the citrus and Accurate three-dimensional information between the cameras ensures the positioning accuracy of the robot's gripper.

2) Target information acquisition, recognition and positioning module: use various algorithms, such as image preprocessing to remove image acquisition noise, correct the impact of light on image quality, and in terms of feature extraction, use image shape features and color features to remove useless backgrounds in natural scenes , extract fruit information and obstacle information, and realize the identification and positioning of ripe fruit and the calculation of three-dimensional position of obstacles.

3) Robot path planning module: Use the bounding box technology to search the free space that can avoid obstacles in the pose space; through the probability map algorithm (PRM), plan a robotic arm that can avoid obstacles and move the claw without collision The movement path to the ripe fruit; the planned path is smoothed to avoid sudden changes in the speed and direction of the mechanical arm movement and damage to the motor.

4) Manipulator and gripper control module: According to the free path obtained by path planning, control the manipulator and end effector to move according to the specified trajectory. When the end effector reaches the designated position, control the vacuum adsorption device to stretch, suck and pull the fruit, control the fingers to grasp the fruit according to the prescribed procedure, and control the fruit stem cutting device to rotate and cut the fruit stem to complete the picking.

Citrus picking method technical scheme of the present invention is:

First, a stereo vision system is used to detect and identify ripe fruit that is ready for picking. Mainly use the color and shape difference between mature citrus and branches and leaves to identify information such as targets and obstacles; then in the world coordinates, calculate the specific position of the fruit, and determine whether the position of the obstacle affects fruit picking.

The obtained fruit position information and obstacle position information are extracted, and a three-dimensional model is established in a virtual environment, and the robot path planning calculation based on the probability map algorithm (PRM) is used to obtain a collision-free path. After obtaining the motion path of the robot, the industrial computer system controls the mechanical arm and the gripper to reach the designated position.

Finally, control the end effector to stretch out the suction cup to suck and pull the fruit, then fix the fruit with the claws, and then rotate to the designated position by the cutting device to cut off the fruit stem.

The advantage of the present invention is that it can sense environmental information in real time based on the fusion of multi-sensing information such as force sense, temperature sense, vision, distance, etc., and guide the end effector of the robot to realize the separation of fruit and tree at a suitable position.

Description of drawings

Figure 1 Schematic diagram of the structure of the citrus picking machine

1-arm base, 2-waist joint (including: 16-gear, 17-toothed belt, 18-code disc, 19-reducer, 20-motor, 21-motor driver), 3-shoulder, 4- Shoulder joint (same structure as waist joint), 5-big arm, 6-elbow joint (same structure as waist joint), 7-little arm, 8-rotation wrist joint (same structure as waist joint), 9-swing wrist joint (same structure as waist joint) waist joint), 10-end effector mounting frame, 11-end effector, 12-stereo vision system, 13-main computer, 14-industrial computer, 15-motion control card.

Figure 2 Schematic diagram of robotic arm joints

16-gear, 17-toothed belt, 18-code disc, 19-reducer, 20-motor, 21-motor driver

Figure 3 Schematic diagram of the end effector structure

Figure 3(1) Front view

Figure 3(2) side view

Figure 3 (3) Schematic diagram of two-joint finger assembly

22-palm, 23-two-joint finger assembly, 24-vacuum adsorption device assembly, 25-ultrasonic ranging device assembly, 26-cutting device assembly, 10-end effector mounting bracket, 27-suction cup motor, 28 -vacuum pump hose, 29-vacuum pump, 30-rubber suction cup, 31-air pressure sensor, 32-cutting device bracket, 33-blade holder rotation device, 34-blade holder, 35-pulley, 36-motor, 37-belt, 38 -Blade, 39-Blade Guard, 40-Proximity Sensor, 41-Force Sensor, 42-Temperature Sensor, 43-Fingertip, 44-Finger Root, 45-Finger Root Drive (Including: 50-Angle Sensor, 56 -motor, 57-driver, 58-reducer), 46-fingertip driver (including: 51-fingertip drive belt, 52-drive wheel, 53-reducer, 54-motor, 55-motor driver, 59-angle sensor), 47-suction cup guide rod, 48-suction cup bracket, 49-screw

Figure 4 Citrus picking robot software control module

Detailed ways

The flexible citrus picking method and device of the present invention have universal applicability in picking spherical fruits. The three-finger two-joint structure can securely grasp spherical fruits, and the force feedback system on the inside of the fingers can implement different gripping forces according to needs, which is suitable for picking thin-skinned and fragile fruits. The key difficulty of the present invention is that citrus has a hard fruit stem, and a sharp knife saw must be used to realize the separation of fruit and tree, and it is difficult to cut off once with a general blade. If you pick other fruits with easy-to-break fruit stems, such as tomatoes, you can remove the knife and saw.

According to the different growth conditions and sizes of citrus fruits, the maximum opening diameter of the designed claws of the present invention is 9 cm, which can accommodate most citrus fruits. Due to gravity, the citrus generally hangs from the branches, some are shaded by the foliage. The specific position of the exposed citrus can be measured by the stereo vision system.

Before image acquisition, it is necessary to calibrate the internal parameters of the stereo vision system, such as focal length and lens distortion coefficient, and also calibrate the external parameters to obtain the baseline distance between the two cameras and the position of the camera in world coordinates. Provide precise parameters for further positioning calculations.

When the robot performs picking operations, two images are first taken by the binocular stereo vision system, and the image preprocessing is carried out by self-developed software to extract the color and shape features of the target, and the overlapping fruits are segmented by Hough transform and minimum external rectangle technology, and the Objects are extracted for feature-based stereo matching, and various constraints are used during matching, such as epipolar constraints, unique constraints, and order-preserving constraints to ensure the uniqueness of matching and obtain 3D information of fruits and obstacles. After the fruit is positioned, the obstacle information and target information are converted to the virtual space, and path planning is performed in the pose space to search for a free path. In order to ensure the smooth movement of the robot arm, the walking path must also be smoothed to reduce the phenomenon of reverse rotation, emergency stop and rapid acceleration when the motor is rotating.

When the end effector reaches the designated position, the vacuum pump starts to evacuate, and starts the driving motor of the suction cup, and extends the suction cup to suck the fruit. The distance of the suction cup is obtained by the distance measuring sensor in the palm. When the fruit is sucked, the vacuum degree in the suction cup pipeline will rise sharply. When the vacuum degree reaches a certain threshold, the suction cup drive motor will reverse, and the suction cup will pull the fruit back to the palm and start the finger drive motor at the same time. The fingers will grasp and fix according to the specified path. For fruit, if there is an obstacle in the process of finger movement, the proximity sensor on the fingertip will send a signal to stop the finger extension; the force sensor on the inside of the finger can detect the grasping force in real time, so as to avoid scratching the fruit or causing insufficient grasping force. The event of fruit falling off occurs. After the fingers wrap the fruit, the suction cup and the three fingers form four fulcrums to fix the fruit and adjust the position with the palm. When it reaches the specified position, the cutting blade bracket will start to move towards the palm, and the rotating blade will start to work. When the blade meets the fruit stem, the load of the blade driving motor will increase and start cutting the fruit stem. When the load disappears, it means that the fruit stem has been cut off and the picking is over. . If the size of the fruit exceeds the design range, the cutting device cannot be activated, effectively protecting the fingers and fruits of the manipulator from being cut. After picking, the blade returns to the starting position, the picked fruit is put into the designated position, and the fingers return to the initial state.

Claims (2)

1. the flexible picker of oranges and tangerines picking robot, it is characterized in that forming by picking fruit hardware unit and software control system two parts, the picking fruit hardware unit is made up of computer vision system, industrial control system, mechanical arm, end effector, and end effector partly comprises: vacuum absorption device, paw and cutter sweep;

1) mechanical arm: form by 5 degree of freedom rotary joints, mechanical arm comprises base (1), waist joint (2), shoulder (3), shoulder joint (4), big arm (5), elbow joint (6), forearm (7), wrist joint (8,9), end effector installing rack (10), wherein wrist joint has 2 degree of freedom, and each joint is all connected to form by gear (16), flute profile band (17), coding disk (18), decelerator (19), motor (20) and motor driver (21);

2) vision system: for being installed in the binocular tri-dimensional vision system (12) on the mechanical arm pedestal, link to each other with master computer (13), realize the collection of environmental information by USB interface;

3) industrial control system: comprise industrial computer (14), a plurality of motion control cards (15) that are used to control each joint of mechanical arm (2,4,6,8,9) and end effector (11) of interpolation;

4) vacuum absorption device: be rubber suction cups (30), be installed on the sucker guide rod (47), the sucker guide rod is fixed on the sucker stand (48), and sucker stand links to each other with screw rod (49), screw rod is terminal to be connected with sucker motor (27), and motor is fixed on the end effector installing rack (10);

5) paw: constitute by 3 two joint fingers (23), be installed on the palm (22), the joint that refers to root (44), finger tip (43) drives by referring to that root driver (46) and finger tip driver (24) drive, refer to that root driver (46) refers to the root rotational angle by angular transducer (50) detection, motor (56), motor driver (57), decelerator (58) directly link to each other with the finger root to control and refer to root (44) motion; Finger tip driver (46) detects the finger tip rotational angle by angular transducer (59), motor (54), motor driver (55), decelerator (53) connect driving wheel (52), drive finger tip rotating band (51) control finger tip (43) motion, each finger tip has been installed proximity transducer (40), power sense sensor (41) and sense of heat sensor (42) are equipped with in each finger inboard, and sonac (25) is housed in the palm (22);

6) cutter sweep: its support (32) is fixed on the palm (22), links to each other with whirligig (33), and motor (36), belt pulley (35) are installed on the blade holder, and there is protective cover (39) the blade top;

Software control system comprises: stereoscopic vision is demarcated, and target information is obtained identification, location, Robot Path planning, mechanical arm and end effector control module.

2. the harvesting method of the flexible picker of oranges and tangerines picking robot according to claim 1 is characterized in that: at first, and the mature fruit that utilizes the stereo visual system detection and Identification to pluck; Utilize color and information such as morphological differences recognition objective, barrier between ripe oranges and tangerines and the branches and leaves; Then under world coordinates, calculate the particular location of fruit, whether the disturbance in judgement object location influences fruit picking;

The fruit positional information and the barrier positional information that obtain are carried out feature extraction, under virtual environment, set up threedimensional model, utilization is carried out robot path planning's calculating based on the probability map algorithm, obtain the collisionless path, after obtaining the robot motion path, arrive at assigned address by industrial computer system control mechanical arm and paw;

At last, the control end effector stretches out sucker and holds and draw fruit, then with the fixing fruit of paw, rotates to assigned address by cutter sweep again, the cut-out carpopodium.

Images