CN114793631B - Picking end effector based on visual guidance dynamic control - Google Patents
Picking end effector based on visual guidance dynamic control Download PDFInfo
- Publication number
- CN114793631B CN114793631B CN202210388993.9A CN202210388993A CN114793631B CN 114793631 B CN114793631 B CN 114793631B CN 202210388993 A CN202210388993 A CN 202210388993A CN 114793631 B CN114793631 B CN 114793631B
- Authority
- CN
- China
- Prior art keywords
- clamping
- clamping arm
- fruit
- stepping motor
- screw rod
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A01—AGRICULTURE; FORESTRY; ANIMAL HUSBANDRY; HUNTING; TRAPPING; FISHING
- A01D—HARVESTING; MOWING
- A01D46/00—Picking of fruits, vegetables, hops, or the like; Devices for shaking trees or shrubs
- A01D46/30—Robotic devices for individually picking crops
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Environmental Sciences (AREA)
- Harvesting Machines For Specific Crops (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a picking end effector based on visual guidance dynamic control, and relates to the field of agricultural machinery. The invention comprises a support, a clamping arm, a driving device, a control system, a depth camera and a pressure sensor; the clamping arms comprise a first clamping arm and a second clamping arm, and the first clamping arm and the second clamping arm are driven by a driving device; the first clamping arm is provided with a cutter, and the second clamping arm is provided with the pressure sensor; the depth camera is arranged on the support and is connected with the driving device through the control system. The fruit harvester can finish harvesting of string-type or individual-type fruits at one time, has high efficiency and compact integral structure, solves the problems of shielding and overlapping, and improves the operation convenience and the safety and reliability.
Description
Technical Field
The invention relates to the field of agricultural machinery, in particular to a picking end effector based on visual guidance dynamic control.
Background
With the increase of the aging population and the reduction of labor force, intelligent automation is urgently needed in agriculture at present. For intelligent automatic picking of crop fruits, due to shielding of branches and leaves of the fruits and overlapping of the fruits, great interference is caused on fruit detection and fruit stem grabbing, and picking failure is caused. In the aspect of vision, most of the existing picking end effectors based on visual guidance utilize deep learning to detect picking points in front of fruits, for the fruits with shielding and overlapping, the deep learning method cannot effectively detect the picking points, and a large amount of data is often needed for training a deep learning model, so that the calculation cost is greatly increased. In the aspect of picking the end effector, most of researches on the structure and motion path planning of the mechanical arm are carried out, and the end effector mainly comprises an air-suction type, a flexible finger type and a mechanical target holder, and cutting parts such as scissors and an electric saw. The air suction type and the flexible finger type have high requirements on positioning accuracy, and can not dynamically control the positioning accuracy in real time, and the mechanical single-closed grabbing easily damages overlapped fruits by the existing method for visually positioning picking points in front of the fruits. Therefore, there is a need to develop a picking end effector based on dynamic control of visual guidance to improve the fruit harvesting efficiency, reduce the harvesting cost and realize timely harvesting of crops.
Disclosure of Invention
In view of the above, the invention provides a picking end effector based on visual guidance dynamic control, which aims to reduce the calculation cost, solve the problems of difficult picking of blocked and overlapped fruits and the like, and overcome the defects that the existing air-suction type, flexible finger type and mechanical end effectors only can perform single grabbing and are easy to grab failure and damage crops.
In order to achieve the purpose, the invention adopts the following technical scheme:
a picking end effector based on visual guidance dynamic control comprises a support, a clamping arm, a driving device, a control system, a depth camera and a pressure sensor; the clamping arms comprise a first clamping arm and a second clamping arm, and the first clamping arm and the second clamping arm are driven by a driving device; the first clamping arm is provided with a cutter, and the second clamping arm is provided with the pressure sensor; the depth camera is arranged on the support and is connected with the driving device through the control system.
Optionally, the driving device comprises a clamping stepping motor, a coupler and a screw rod; the clamping stepping motor is fixed on one side of the bracket, and a power output shaft of the clamping stepping motor is connected with the screw rod through the coupler; the first clamping arm and the second clamping arm are connected to the screw rod.
Optionally, the cutter is slidably connected to the first clamping arm, and the cutter is driven by a sliding rail stepping motor.
Optionally, the slide rail stepping motor on the first clamping arm controls the slide rail trapezoidal screw rod to move, and the slide block is used for moving the slice back and forth.
Optionally, small deep groove ball bearings are mounted at two ends of the sliding rail trapezoidal screw rod.
Optionally, two ends of the screw rod are provided with large deep groove ball bearings.
Optionally, a fan is further provided, and the fan is disposed on one side of the depth camera.
Optionally, the control system obtains contour information of the bottom of the fruit by using a Canny operator, performs polygonal curve fitting on the contour to obtain edge points, and uses the contour points at the bottom as initial state values for the kalman filter to predict the state values at the next moment.
Optionally, the control system positions the fruit stem picking point by using depth information through a set constraint condition, the fruit stems are clamped by the clamping arms, the clamping force is given by the pressure sensor on the second clamping arm, and when the clamping force reaches a preset value, the driving device drives the cutter on the first clamping arm to move.
According to the technical scheme, compared with the prior art, the picking end effector based on the visual guide dynamic control can finish harvesting of string-shaped or individual-shaped fruits at one time, is high in efficiency and compact in overall structure, solves the problems of shielding and overlapping, and improves the operation convenience and the safety and reliability.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
FIG. 1 is a schematic structural view of an end effector of the present invention;
FIG. 2 is a top view of an end effector of the present invention;
FIG. 3 is a flow chart of the visual guidance dynamic control of the present invention;
FIG. 4 is a flowchart illustrating operation of the end effector of the present invention;
FIG. 5 is a diagram of a control system for an end effector of the present invention;
in the figure: the device comprises a clamping stepping motor 1, a coupler 2, a bearing end cover 3, a support 4, a depth camera 5, a fan 6, a clamping trapezoidal screw 7, a cutting stepping motor 8, a circular saw tooth cutter 9, a sliding block 10, a sliding rail trapezoidal screw 11, a sliding rail stepping motor 12, a right clamping arm 13, a pressure sensor 14, a left clamping arm 15, a large deep groove ball bearing 16 and a small deep groove ball bearing 17.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The embodiment of the invention discloses a picking end effector based on visual guidance dynamic control, which is shown in figures 1-5, wherein the picking end effector comprises a clamping stepping motor 1, a coupler 2, a bearing end cover 3, a support 4, a depth camera 5, a fan 6, a clamping trapezoidal screw rod 7, a cutting stepping motor 8, a circular saw tooth cutter 9, a slide block 10, a slide rail trapezoidal screw rod 11, a slide rail stepping motor 12, a right clamping arm 13, a pressure sensor 14, a left clamping arm 15, a large deep groove ball bearing 16 and a small deep groove ball bearing 17, and specifically comprises the following steps:
(1) The device comprises a support, a depth camera, a left clamping arm, a right clamping arm, a coupler, a fan, a clamping stepping motor, a cutting stepping motor, a sliding rail stepping motor, a sliding block, a clamping trapezoidal screw rod, a sliding rail trapezoidal screw rod, a large deep groove ball bearing, a small deep groove ball bearing, a circular serrated knife and a pressure sensor;
(2) The clamping stepping motor is fixed on the right side of the bracket, a power output shaft of the clamping stepping motor is connected with the clamping trapezoidal screw rod through a coupler, deep groove ball bearings are arranged at two ends of the clamping trapezoidal screw rod, a left clamping arm and a right clamping arm are connected to the clamping trapezoidal screw rod, a rotating pair for clamping the trapezoidal screw rod is converted into a moving pair of the clamping arms, and the trapezoidal screw rod is self-locked, so that the clamping stability can be greatly improved;
(3) The upper end of the support is fixedly provided with a depth camera and a fan; a pressure sensor is arranged at the finger position of the right clamping arm; a slide rail stepping motor on the left clamping arm controls the motion of a slide rail trapezoidal screw rod, and the circular saw blade moves back and forth through a slide block;
(4) The two ends of the clamping trapezoidal screw rod are provided with large deep groove ball bearings, and the two ends of the sliding rail trapezoidal screw rod are provided with small deep groove ball bearings, so that the transmission stability can be ensured;
(5) The center of the base circle of the circular saw tooth cutter is fixedly connected with an output shaft of the cutting stepping motor; the clamping stepping motor and the cutting stepping motor are controlled by a controller;
(6) The cutting range of the circular serrated knife is slightly larger than the maximum diameter of the main stem of the fruit.
(7) When the visual system of the picking end effector detects the position of the bottom of the fruit, the mechanical arm drives the end effector to be close to the bottom of the fruit, and the safety distance between the support and the fruit is controlled by utilizing the depth information.
(8) The real-time contour prediction algorithm can dynamically control the opening and closing size of the end effector, and prevent the end effector from damaging a target fruit while pushing away overlapped fruits.
(9) The start of the fan needs whether the real-time contour prediction algorithm finds that the occlusion exists, and when the occlusion of the leaves exists, the fan is started to blow the leaves away.
(10) The sensor detects whether the fruit stalks need to enter the clamping position of the end effector, if the fruit stalks enter the clamping position, the clamping stepping motor rotates to drive the left clamping arm and the right clamping arm to be closed, meanwhile, the pressure sensor detects the pressure on the fruit stalks, and when the pressed fruit stalks reach the preset minimum damage force (the force which does not damage the fruit stalks and ensures that the fruits cannot slide from the paws under the action of gravity), the clamping stepping motor stops rotating and keeps the current state.
(11) Then the slide rail stepping motor and the cutting stepping motor rotate simultaneously to drive the circular serrated knife to move towards the fruit stalks, and the fruit stalks are cut off. The fruit can be picked without damage.
(12) The lowest point of the fruit is located through the depth information acquired by the depth camera, the end effector is guided to move upwards from the bottom of the fruit, the fruit is generally wide at the top and small at the bottom, so that the bottom is not influenced by overlapping, and the end effector is prevented from damaging the overlapped fruit by starting to move from the bottom.
(13) A real-time contour prediction algorithm: firstly, contour information of the bottom of a fruit is obtained by using a Canny operator, polygonal curve fitting is carried out on the contour to obtain edge points, the contour points at the bottom are used as initial state values to be used for a Kalman filter to predict the state values of the next moment, and the specific principle is shown in formulas (2) - (4).
(14) And (3) predicting the contour in real time by using a Kalman filter, and comparing the contour with the contour obtained by a Canny operator at the same moment so as to judge whether the prediction is abnormal or whether leaves are shielded.
(15) The depth camera is arranged above the support, the path of the end effector of the fruit can be planned in advance according to the outline of the fruit through a visual algorithm of real-time outline prediction, the opening and closing size of the clamping arm can be dynamically controlled in real time through a clamping mechanism consisting of a stepping motor and a trapezoidal screw rod at the right end of the support as well as a left clamping arm and a right clamping arm, so that overlapped fruits can be pushed away while crops are prevented from being damaged, and the influence of overlapping factors on picking is avoided;
(16) When the contour prediction is abnormal, the upper part can be judged to be shielded by the leaves, and the fan above the support can blow the leaves open, so that the shielding of the leaves is avoided;
(17) Through the constraint condition who sets for, utilize degree of depth information to fix a position fruit stalk picking point, carry out the centre gripping to the fruit stalk through fixture, the clamping-force size is given by the pressure sensor on the right centre gripping arm finger, and when the clamping-force reached the default, slide rail motor on the left centre gripping arm drove the slider and moves forward, and the cutting motor starts simultaneously for circle serrated knife cuts off the fruit stalk, realizes the picking of fruit.
The working principle of the embodiment is as follows: the fruit picking robot is used for accurately picking overlapped and shielded fruits, the bottom position of the fruit to be picked is detected by a visual system of a depth camera 5 of the picking robot in real time to obtain the depth information of the fruit, and the end effector is driven by the mechanical arm to reach a certain distance below the bottom of the fruit in the range that the support 4 can not collide with the fruit. The gripper is composed of a left clamping arm and a right clamping arm, the depth camera 5 is installed above the support 4, the fruit outline of the gripper can be predicted in advance in real time, the distance between the marginal point and the central axis of the fruit is calculated, the mechanical arm is controlled in real time to drive the end effector to move from bottom to top, and the gripper stepper motor 1 is dynamically guided to dynamically control the opening and closing size of the gripper. When the contour prediction is abnormal, the upper part is judged to be shielded by the leaves, and the fan 6 is started to blow the leaves. When the sudden change of the outline width is detected, the fruit stalks can be positioned by using a formula (1) in combination with depth information, then the fruit stalks can enter a grabbing state, the clamping stepping motor 1 rotates to control the closing of the paw to finish the fruit stalk clamping action, when the pressure measured by the pressure sensor 14 reaches the preset minimum fruit stalk damage force, the paw is kept at the current position, then the slide rail stepping motor 12 on the left clamping arm 15 is started to drive the slide block 10 to move towards the fruit stalk direction, and meanwhile, the cutting stepping motor 8 on the slide block 10 drives the circular serrated knife 9 to rotate at a high speed to finish the fruit stalk shearing action. After the shearing is completed, the sliding rail stepping motor 12 resets, the cutting stepping motor 8 stops rotating, the fan 6 stops, the mechanical arm drives the end effector to move to the fruit collecting device, the gripper releases fruits, the fruit picking action is completed, if the fruit picking quantity is not zero, the picking action of the next fruit is started, if the fruit picking quantity is zero, each part resets, and the picking is finished.
Wherein
The depth range of the fruit stem is shown as r, the maximum radius of the fruit is shown as r, dc is the nearest distance from the camera to the fruit, and deltad is an error value.
Wherein, X (k) represents the state value at the time of k, X (k + 1) represents the state value at the time of k +1, A (k) is a state transition matrix, T (k) is an interference transition matrix, W (k) represents the system state noise of the motion model, Z (k) represents an observation vector, H (k) represents an observation matrix, V (k) represents the observation noise, (xi, yi) represents the position of the actual track point, (xi ', yi') represents the position information of the predicted track point, and k represents the number of the predicted track points.
In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims (1)
1. A picking end effector based on visual guidance dynamic control is characterized by comprising a support, a clamping arm, a driving device, a control system, a depth camera and a pressure sensor; the clamping arms comprise a first clamping arm and a second clamping arm, and the first clamping arm and the second clamping arm are driven by a driving device; the first clamping arm is provided with a cutter, and the second clamping arm is provided with the pressure sensor; the depth camera is arranged on the support and is connected with the driving device through the control system;
the driving device comprises a clamping stepping motor, a coupler and a clamping trapezoidal screw rod; the clamping stepping motor is fixed on one side of the support, and a power output shaft of the clamping stepping motor is connected with the clamping trapezoidal screw rod through the coupler; the clamping trapezoidal screw rod is connected with the first clamping arm and the second clamping arm;
the cutter is connected to the first clamping arm in a sliding mode and driven by a sliding rail stepping motor;
the sliding rail stepping motor on the first clamping arm controls the sliding rail trapezoidal screw rod to move, and the sliding block is used for realizing the forward and backward movement of the cutter;
two ends of the slide rail trapezoidal screw rod are provided with small deep groove ball bearings; two ends of the clamping trapezoidal screw rod are provided with large deep groove ball bearings;
the fan is arranged on the support;
the control system obtains the outline information of the bottom of the fruit by using a Canny operator, carries out polygonal curve fitting on the outline to obtain edge points, takes the outline points at the bottom as initial state values for a Kalman filter to predict the state values at the next moment,
if the R does not meet the preset value, starting the fan and predicting the contour state value by adopting the Kalman filter again;
if R meets a preset value, calculating the distance from the edge point to the central axis of the fruit, dynamically controlling the opening and closing size of the clamping arm, and driving the end effector to move upwards by the mechanical arm;
wherein:
wherein, (xi, yi) represents the position of the actual track point, (xi ', yi') represents the position information of the predicted track point, and k represents the number of the predicted track points;
the control system positions fruit stem picking points by judging whether the profile has sudden change or not and using a formula (1) in combination with depth information, fruit stems are clamped by the clamping arms, the clamping force is given by the pressure sensor on the second clamping arm, and when the clamping force reaches a preset value, the sliding rail stepping motor drives the cutter on the first clamping arm to move;
the formula (1) is: d f ＝r+d c ±Δd；
Wherein D f The depth range of the fruit stem, r is the maximum radius of the fruit, d c The closest distance from the camera to the fruit, and Δ d is the error value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210388993.9A CN114793631B (en) | 2022-04-13 | 2022-04-13 | Picking end effector based on visual guidance dynamic control |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210388993.9A CN114793631B (en) | 2022-04-13 | 2022-04-13 | Picking end effector based on visual guidance dynamic control |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114793631A CN114793631A (en) | 2022-07-29 |
| CN114793631B true CN114793631B (en) | 2023-04-18 |
Family
ID=82536918
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210388993.9A Active CN114793631B (en) | 2022-04-13 | 2022-04-13 | Picking end effector based on visual guidance dynamic control |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114793631B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115413479B (en) * | 2022-09-27 | 2024-03-22 | 江苏大学 | Deep vision-based water shield picking end effector and picking method thereof |
| CN116686545B (en) * | 2023-06-29 | 2024-01-16 | 佛山市中科农业机器人与智慧农业创新研究院 | Litchi picking robot shade removing method based on machine vision control |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| FR2760595B1 (en) * | 1997-03-13 | 1999-05-14 | Tecnoma | METHOD FOR HARVESTING OBJECTS CARRIED BY A ROD, SUCH AS GRAPES OF GRAPES |
| EP3539735A1 (en) * | 2018-03-13 | 2019-09-18 | Soluciones Robóticas Agrícolas S.L. | Robotic arm end effector for fruit harvesting |
| CN108401685A (en) * | 2018-05-08 | 2018-08-17 | 湖州佳创自动化科技有限公司 | A kind of spheral fruit picking robot |
| CN112715163A (en) * | 2021-02-02 | 2021-04-30 | 广州大学 | Fruit picking robot |
| CN114175927B (en) * | 2021-11-29 | 2022-10-18 | 季华实验室 | Cherry tomato picking method and cherry tomato picking manipulator |
-
2022
- 2022-04-13 CN CN202210388993.9A patent/CN114793631B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN114793631A (en) | 2022-07-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114793631B (en) | Picking end effector based on visual guidance dynamic control | |
| CN207476246U (en) | A kind of ridge culture strawberry picking robot | |
| CN107711067A (en) | A kind of ridge culture strawberry picking robot | |
| CN102550216B (en) | End effector of intelligent harvesting robot | |
| CN109496563A (en) | A kind of fruit picking robot | |
| CN112715163A (en) | Fruit picking robot | |
| JP2010178633A (en) | Hand apparatus for harvesting asparagus | |
| EP4140286A1 (en) | Crop harvesting method, crop harvesting device, and crop harvesting system | |
| CN108370740A (en) | A kind of topping end effector and the topping machine containing the actuator | |
| CN210746116U (en) | Fruit stalk clamping and shearing type picking robot end effector | |
| CN109121799B (en) | Reciprocating tool and control method thereof | |
| CN217706206U (en) | Small-size clear water hyacinth machine of dragging for | |
| KR102511629B1 (en) | End Effector for Crop Implements | |
| KR100399484B1 (en) | An end effector for harvesting fruits and vegetables | |
| CN215991990U (en) | Automatic picking machine for elevated strawberries | |
| CN215530076U (en) | Plum tree fruit picking equipment | |
| CN115072104A (en) | Automatic tear tape unit open | |
| CN108370746B (en) | Guide cutter structure for tree obstacle clearing aerial robot and control method | |
| CN220422495U (en) | Device is picked to ridged strawberry | |
| CN219311289U (en) | End of general agricultural picking robot | |
| Li et al. | End-effector for robotic harvesting of a tomato fruit with calyx | |
| JP3052470B2 (en) | Manipulator of fruit harvesting robot | |
| CN113618747B (en) | End actuating mechanism of apple picking robot | |
| CN220630126U (en) | Leek harvesting device | |
| CN212936776U (en) | Novel apple is picked device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |