CN114474050A - Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures - Google Patents
Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures Download PDFInfo
- Publication number
- CN114474050A CN114474050A CN202111642690.7A CN202111642690A CN114474050A CN 114474050 A CN114474050 A CN 114474050A CN 202111642690 A CN202111642690 A CN 202111642690A CN 114474050 A CN114474050 A CN 114474050A
- Authority
- CN
- China
- Prior art keywords
- image
- workpiece
- grabbing
- robot
- arm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 238000013528 artificial neural network Methods 0.000 claims abstract description 23
- 238000012549 training Methods 0.000 claims abstract description 13
- 238000012545 processing Methods 0.000 claims abstract description 8
- 230000009471 action Effects 0.000 claims description 25
- 238000002360 preparation method Methods 0.000 claims description 15
- 238000009826 distribution Methods 0.000 claims description 4
- 238000011156 evaluation Methods 0.000 claims description 4
- 238000004891 communication Methods 0.000 claims description 3
- 238000010586 diagram Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 238000004519 manufacturing process Methods 0.000 description 6
- 239000011159 matrix material Substances 0.000 description 6
- 230000009466 transformation Effects 0.000 description 4
- 210000000078 claw Anatomy 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000008901 benefit Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1602—Programme controls characterised by the control system, structure, architecture
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1612—Programme controls characterised by the hand, wrist, grip control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1679—Programme controls characterised by the tasks executed
- B25J9/1682—Dual arm manipulator; Coordination of several manipulators
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Automation & Control Theory (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Orthopedic Medicine & Surgery (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a multi-topology structure workpiece sorting method of a double-arm robot based on grabbing prediction, which comprises the following steps: the robot controller sends an instruction and reads an RGB-D image from a robot vision system; dividing the RGB-D image into an RGB image and a depth image for standardization treatment respectively; constructing a capturing and predicting two-channel neural network and initializing network parameters; constructing a workpiece data set required by network training; respectively taking the RGB image and the depth image after the standardization processing as the input of a capture prediction dual-channel neural network; grabbing the grippable position of the workpiece output by the two-channel neural network; converting the grippable position of the workpiece from a camera coordinate system to a mechanical arm coordinate system; and sending an instruction by the robot controller, grabbing the workpiece by adopting a multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system, and placing the workpiece in a designated area to finish sorting. The invention can realize the accurate sorting of workpieces with different topological structures.
Description
Technical Field
The invention belongs to the technical field of industrial robot assembly, and particularly relates to a method for sorting workpieces with various topological structures by a double-arm robot based on grabbing prediction.
Background
Aerospace manufacturing has very high requirements on precision, reliability, standard flow and the like, so a large number of national craftsmen are cultivated in the field of aerospace manufacturing. However, as time goes on, some skilled artisans become retired, and the new 80 th and 90 th labour population is not interested in doing simple and repetitive industrial work with poor working environment, so that it is a good choice to complete a part of the work by robot instead of manpower while cultivating the new generation of the great national craftsman, and the shift from aerospace manufacturing to aerospace manufacturing becomes a great trend.
The industrial robot assembly used in the manufacturing industry at present is mainly used for sorting a single mechanical arm or sorting a plurality of independent mechanical arms on an assembly line. According to the sorting method, the mechanical arms are not matched with each other, and sorting actions are edited in advance, so that the mechanical arms are poor in sorting precision, low in assembly efficiency and poor in adaptability.
Modern smart manufacturing presents new challenges to the visual sorting of industrial robots, including frequently changing part types, mix and mutual occlusion of parts, and uncertainty in the sorted objects. At present, three-dimensional vision is mainly adopted for sorting scattered parts of a material basket of a robot to obtain object point cloud information, and objects are identified by establishing a vision template or combining artificial features. The visual programming method depends on the programming experience of a visual engineer, cannot adapt to frequently changed sorting objects and an unknown working environment, and cannot meet the challenges brought by aerospace wisdom creation.
Disclosure of Invention
The technical problem solved by the invention is as follows: the defects of the prior art are overcome, the grabbing prediction-based method for sorting the workpieces with various topological structures by the double-arm robot is provided, and the workpieces with various different topological structures can be accurately sorted.
The purpose of the invention is realized by the following technical scheme: a multi-topology workpiece sorting method based on grabbing prediction for a double-arm robot comprises the following steps: the robot controller sends an instruction and reads an RGB-D image from a robot vision system; dividing the RGB-D image into an RGB image and a depth image for standardization treatment respectively; constructing a capturing and predicting two-channel neural network and initializing network parameters; constructing a workpiece data set required by network training, and training a network; respectively taking the RGB image and the depth image after the standardization processing as the input of a capture prediction dual-channel neural network; the method comprises the steps of (1) grabbing and predicting a grippable position, a grabbing angle, a grabbing paw width and a grabbing fraction of a workpiece output by a two-channel neural network; converting the grippable position of the workpiece from a camera coordinate system to a mechanical arm coordinate system; and sending an instruction by the robot controller, grabbing the workpiece by adopting a multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system, and placing the workpiece in a designated area to finish sorting.
In the above grasping prediction-based workpiece sorting method for multiple topological structures of the double-arm robot, reading the RGB-D image comprises the following steps: the robot controller sends an instruction to the omnibearing mobile platform, and the robot moves to a designated sorting station; the robot controller sends an instruction to the robot vision system, and the RGB-D camera starts to shoot images at a speed of 20 fps; the robot controller reads the RGB-D image from the robot vision system.
In the above grasping prediction-based method for sorting workpieces with multiple topological structures by using a double-arm robot, the step of dividing the RGB-D image into the RGB image and the depth image and respectively carrying out standardization processing comprises the following steps: respectively calculating the average value and the standard deviation of all pixel data of the RGB image and the depth image; and respectively traversing the pixel data of the RGB image and the depth image, and subtracting the average value from each pixel data and then dividing the average value by the standard deviation.
In the multi-topology-structure workpiece sorting method based on the grabbing prediction, in a grabbing prediction double-channel neural network, a network architecture adopts a convolution module, a residual error module and a convolution replacement module; the network input end is a dual-channel image, and a characteristic diagram of the channel image is extracted through respective channels; the input image of the network is adjusted in size to adapt to the configuration of the convolution layer at the input end of the network, and meanwhile, the length-width ratio of the original image is not changed; the output end of the network is a user-defined grabbing action parameter.
In the above method for sorting workpieces with multiple topological structures by using a double-arm robot based on grabbing prediction, the self-defined grabbing action parameters are obtained by the following formula:
Action＝F(cp,θ,w,r)；
the method comprises the following steps that an Action is a grabbing Action of the double-arm robot, F () is mapping from neural network output information to the grabbing Action of the double-arm robot, cp is a central point of a position where a workpiece can be grabbed, and theta is a rotating angle of the tail end of an mechanical arm around an axis; w is the opening width of the mechanical arm paw; and r is the evaluation score of the mechanical arm grabbing action.
In the above method for sorting workpieces with multiple topological structures by using a double-arm robot based on grabbing prediction, the adjustment of the size of the input image of the network comprises the following steps: calculating the maximum value of the image width Iw and the height Ih, and dividing the maximum value by the image target size Is to obtain an image adjustment ratio Ir; if the image width Iw Is not less than the height Ih, adjusting the image width Iw to be an image target size Is, and adjusting the image height Ih to be a value obtained by dividing the image height Ih by an image adjustment proportion Ir; if the image width Iw Is smaller than the height Ih, the image width Iw Is adjusted to be the value of dividing the image height Iw by the image adjustment ratio Ir, and the image height Ih Is adjusted to be the image target size Is.
In the above grasping prediction-based workpiece sorting method for multiple topological structures of the double-arm robot, the step of constructing a workpiece data set required by network training comprises the following steps: placing workpieces with various topological structures on a workpiece table; the depth camera is fixed at the tail end of the mechanical arm, generates a motion track according to Gaussian distribution, moves along the track, and controls the depth camera to shoot images while moving; and splitting the acquired image into an RGB image and a depth image, and manually marking the grabbing action parameters.
In the multi-topology-structure workpiece sorting method based on the grabbing prediction, the step of grabbing the workpiece by adopting a multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system and placing the workpiece in a designated area comprises the following steps:
(1) dividing the desktop into a middle area to be sorted, a left workpiece placing area and a right workpiece placing area, placing the workpieces in the middle area to be sorted, and defining the left workpiece placing area and the right workpiece placing area as designated workpiece placing positions;
(2) the robot controller sends a sorting starting instruction to the two mechanical arms respectively in a multi-process communication mode, and simultaneously sends information of the current position of a workpiece, the opening and closing angle of the paw and the placement position of the workpiece;
(3) after receiving the instruction, the two mechanical arms move to the preparation positions;
(4) the left mechanical arm starts to enter the middle to-be-sorted area to grab the workpieces, meanwhile, the robot controller locks the middle to-be-sorted area, the other mechanical arm is forbidden to enter the middle to-be-sorted area, the left mechanical arm returns to a preparation position after grabbing the workpieces, and the robot controller unlocks the middle to-be-sorted area;
(5) the left mechanical arm puts the grabbed workpieces to a specified workpiece placing position and returns to a preparation position, meanwhile, the right mechanical arm starts to enter a middle to-be-sorted area to grab the workpieces, the controller locks the middle to-be-sorted area, the other mechanical arm is forbidden to enter the middle to-be-sorted area, the right mechanical arm returns to the preparation position after grabbing the workpieces, and the robot controller unlocks the middle to-be-sorted area;
(6) the right mechanical arm places the grabbed workpiece at the appointed workpiece placing position and returns to the preparation position, and meanwhile, the left mechanical arm repeats the step (4);
(7) and (5) repeating the steps (4) to (6) until the workpieces in the middle area to be sorted are sorted.
Compared with the prior art, the invention has the following beneficial effects:
(1) the method realizes the identification of the types and the directions of the workpieces through the workpiece sorting method based on the grabbing prediction for the double-arm robot with various topological structures, and simultaneously calculates the grabbing positions of the mechanical arms, the opening and closing sizes of the claws, the grabbing angles and the like, thereby realizing the accurate grabbing and the fine sorting of the workpieces with various topological structures;
(2) according to the invention, by adopting a multi-process double-mechanical-arm cooperative grabbing control method, the double-mechanical-arm cooperative control is realized from the aspect of an operating system, so that the collision of the double mechanical arms in the motion process is fundamentally avoided, workpieces are sorted one by one relative to the double mechanical arms, and the sorting efficiency is doubled;
(3) the invention controls the opening and closing size and the grabbing angle of the paw through the controller, ensures that the paw can grab a workpiece more stably without falling off, and simultaneously ensures that the outer edge of the finger does not touch other workpieces.
Drawings
Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The drawings are only for purposes of illustrating the preferred embodiments and are not to be construed as limiting the invention. Also, like reference numerals are used to refer to like parts throughout the drawings. In the drawings:
FIG. 1 is a flowchart of a method for sorting workpieces with various topological structures by a double-arm robot based on grabbing prediction according to an embodiment of the present invention;
fig. 2 is a flowchart of a multi-process two-robot cooperative grabbing control method according to an embodiment of the present invention.
Detailed Description
Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art. It should be noted that the embodiments and features of the embodiments may be combined with each other without conflict. The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
The robot comprises an all-directional mobile platform, a robot body, two six-degree-of-freedom mechanical arms, two tail end pneumatic flexible paws and a vision system. The robot body is arranged on the moving platform, the two six-degree-of-freedom mechanical arms are symmetrically arranged on two sides of the robot body, the two tail end pneumatic flexible claws are respectively arranged on tail end flanges of the mechanical arms, and the robot vision system is arranged on the body.
Fig. 1 is a flowchart of a method for sorting workpieces with multiple topological structures by a dual-arm robot based on grabbing prediction according to an embodiment of the present invention. As shown in fig. 1, the method comprises the steps of:
the robot controller sends an instruction and reads an RGB-D image from a robot vision system;
dividing the RGB-D image into an RGB image and a depth image for standardization treatment respectively;
constructing a capturing and predicting two-channel neural network and initializing network parameters;
constructing a workpiece data set required by network training, and training a network;
respectively taking the RGB image and the depth image after the standardization processing as the input of a capture prediction dual-channel neural network;
the method comprises the steps of (1) grabbing and predicting a grippable position, a grabbing angle, a grabbing paw width and a grabbing fraction of a workpiece output by a two-channel neural network;
converting the grippable position of the workpiece from a camera coordinate system to a mechanical arm coordinate system;
and sending an instruction by the robot controller, grabbing the workpiece by adopting a multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system, and placing the workpiece in a designated area to finish sorting.
Acquiring an RGB-D image comprises the following steps: the robot controller sends an instruction to the omnibearing mobile platform, and the robot moves to a designated sorting station; the robot controller sends an instruction to the vision system, and the RGB-D camera starts to shoot images at the speed of 20 fps; the robot controller reads the RGB-D image from the robot vision system.
The method for dividing the RGB-D image into the RGB image and the depth image and respectively carrying out standardization processing comprises the following steps: respectively calculating the average value and the standard deviation of all pixel data of the RGB image and the depth image; and respectively traversing the pixel data of the RGB image and the depth image, and subtracting the average value from each pixel data and then dividing the average value by the standard deviation.
The method for constructing the grabbing prediction dual-channel neural network comprises the following steps: digitally defining grabbing action parameters according to tasks of grabbing and predicting a two-channel neural network; the network mainly adopts a convolution module, a residual error module and a convolution permutation module; the network input end is a dual-channel image, and the characteristic diagram of the corresponding image is extracted through respective channels; the input image of the network is adjusted in size to adapt to the configuration of the convolution layer at the input end of the network, and meanwhile, the length-width ratio of the original image is not changed; the output of the network corresponds to the user-defined grabbing action parameters respectively.
Digitally defining the detailed information of the grabbing action parameters as follows:
define grab action equation 1: action ═ F (cp, θ, w, r);
wherein the parameter cp of formula 1 represents the center point of the graspable position of the workpiece;
wherein the parameter θ of equation 1 represents the rotation angle of the end of the robot arm about the axis;
wherein the parameter w of formula 1 represents the width of the mechanical arm paw opening;
wherein the parameter r of equation 1 represents the evaluation score of the robot arm gripping action.
The image resizing comprises the steps of: calculating the maximum values of the image width Iw and the image height Ih, and dividing the maximum values by the image target size Is to obtain an image adjustment ratio Ir; if the image width Iw Is not less than the height Ih, adjusting the image width Iw to be an image target size Is, and adjusting the image height Ih to be a value obtained by dividing the image height Ih by an image adjustment proportion Ir; if the image width Iw Is smaller than the height Ih, the image width Iw Is adjusted to be the value of dividing the image height Iw by the image adjustment ratio Ir, and the image height Ih Is adjusted to be the image target size Is.
The method for constructing the workpiece data set required by network training comprises the following steps: selecting representative workpieces with different topological structures, and randomly placing the workpieces on a workpiece table; the depth camera is fixed at the tail end of the mechanical arm, generates a motion track according to Gaussian distribution, moves along the track, and controls the camera to shoot images during the motion process; and splitting the acquired image into an RGB image and a depth image, and manually marking the grabbing action parameters.
The training of the grabbing prediction two-channel neural network comprises the following steps: initializing parameters of a capturing and predicting two-channel neural network; after the RGB image and the depth image are subjected to standardization processing and size adjustment, inputting a capture and prediction dual-channel neural network, respectively extracting feature maps of corresponding images through a dual-channel convolution layer, superposing the feature maps according to channel dimensions, performing forward propagation through the convolution layer, a residual error layer and a convolution replacement layer, extracting feature data, outputting a result, and performing backward propagation to reduce an error function; the trained network predicts the untrained data set, artificially modifies the data with overlarge deviation on the basis of prediction, inputs the modified data into the neural network again for training, and trains repeatedly until the network can accurately predict more untrained data.
The detailed information of the error function used in the neural network training is as follows:
define the loss function equation 2: loss is losscp+lossθ+lossw+lossr；
Define equation 3 in equation 2:
define equation 4 in equation 2:
define equation 5 in equation 2:
define equation 6 in equation 2:
wherein the parameters x, y, z of formula 3 represent the three-dimensional center point of the graspable position of the workpiece;
wherein the parameter θ of equation 4 represents the rotation angle of the end of the robot arm about the axis;
wherein the parameter w of equation 5 represents the width of the mechanical arm paw opening;
wherein the parameter r of equation 6 represents the evaluation score of the robot arm gripping action.
Converting the graspable position from the camera coordinate system to the robot arm coordinate system comprises the steps of:
the depth camera is fixed at the tail end of the mechanical arm, the mechanical arm base is fixed, and the calibration plate is fixed;
the mechanical arm generates different motion pose points according to Gaussian distribution, and controls the camera to acquire the image of the calibration plate after the mechanical arm moves in place;
carrying out camera calibration calculation on the collected calibration plate image to obtain an external parameter matrix T of the cameracamera2object；
The following equation 1 is listed:
Trobot2end×Tend2camera×Tcamera2object
＝T′robot2end×T′end2camera×T′camera2object
resolving a conversion matrix from a camera coordinate system to a mechanical arm coordinate system through equation 1;
wherein T in equation 1robot2endA transformation matrix representing a robot arm base coordinate system to a robot arm end coordinate system;
wherein T in equation 1end2cameraA transformation matrix representing the robot arm end coordinate system to the camera coordinate system;
wherein T in equation 1camera2objectA transformation matrix representing the camera coordinate system to the calibration plate coordinate system.
The following equation 2 is listed:
Trobot2object＝Trobot2end×Tend2camera×Tcamera2object
calculating the graspable position of the mechanical arm in the coordinate system through equation 2;
wherein T in equation 2robot2objectA transformation matrix representing the robot arm coordinate system to the graspable target coordinate system.
As shown in fig. 2, the following steps are included in the method for grabbing a workpiece and placing the workpiece in a designated area by using a multi-process double-mechanical-arm cooperative grabbing control method:
(1) dividing the desktop into a middle to-be-sorted area, a left workpiece placing area and a right workpiece placing area, placing workpieces in the middle to-be-sorted area immediately, and defining appointed workpiece placing positions in the left and right workpiece placing areas;
(2) the robot controller sends a sorting starting instruction to the two mechanical arms respectively in a multi-process communication mode, and simultaneously sends information such as the current position of a workpiece, the opening and closing angle of a paw, the placement position of the workpiece and the like;
(3) after receiving the instruction, the two mechanical arms move to the preparation positions;
(4) the left mechanical arm starts to enter a to-be-sorted area to grab workpieces, meanwhile, the controller locks the to-be-sorted area, the other mechanical arm is forbidden to enter the to-be-sorted area, the left mechanical arm returns to a preparation position after grabbing the workpieces, and the controller unlocks the to-be-sorted area;
(5) the left mechanical arm puts the grabbed workpieces to an appointed placing position and returns to a preparation position, meanwhile, the right mechanical arm starts to enter a to-be-sorted area to grab the workpieces, meanwhile, the controller locks the to-be-sorted area, the other mechanical arm is forbidden to enter the to-be-sorted area, the right mechanical arm returns to the preparation position after grabbing the workpieces, and the controller unlocks the to-be-sorted area;
(6) the right mechanical arm places the grabbed workpiece at the appointed placing position and returns to the preparation position, and meanwhile, the left mechanical arm repeats the step (4);
(7) and (5) repeating the steps (4) to (6) until the workpieces in the area to be sorted are sorted.
The method realizes the identification of the types and the directions of the workpieces through the workpiece sorting method based on the grabbing prediction for the double-arm robot with various topological structures, and simultaneously calculates the grabbing positions of the mechanical arms, the opening and closing sizes of the claws, the grabbing angles and the like, thereby realizing the accurate grabbing and the fine sorting of the workpieces with various topological structures; according to the invention, by adopting a multi-process double-mechanical-arm cooperative grabbing control method, the double-mechanical-arm cooperative control is realized from the aspect of an operating system, so that the collision of the double mechanical arms in the motion process is fundamentally avoided, workpieces are sorted one by one relative to the double mechanical arms, and the sorting efficiency is doubled; the invention controls the opening and closing size and the grabbing angle of the paw through the controller, ensures that the paw can firmly grab the workpiece without falling off, and simultaneously ensures that the outer edge of the finger does not touch other workpieces.
Although the present invention has been described with reference to the preferred embodiments, it is not intended to limit the present invention, and those skilled in the art can make variations and modifications of the present invention without departing from the spirit and scope of the present invention by using the methods and technical contents disclosed above.
Claims (8)
1. A multi-topology workpiece sorting method based on grabbing prediction for a double-arm robot is characterized by comprising the following steps:
the robot controller sends an instruction and reads an RGB-D image from a robot vision system;
dividing the RGB-D image into an RGB image and a depth image for standardization treatment respectively;
constructing a capturing and predicting two-channel neural network and initializing network parameters;
constructing a workpiece data set required by network training, and training a network;
respectively taking the RGB image and the depth image after the standardization processing as the input of a capture prediction dual-channel neural network;
the method comprises the following steps of (1) grabbing and predicting a grippable position, a grabbing angle, a grabbing paw width and a grabbing score of a workpiece output by a two-channel neural network;
converting the grippable position of the workpiece from a camera coordinate system to a mechanical arm coordinate system;
and sending an instruction by the robot controller, grabbing the workpiece by adopting a multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system, and placing the workpiece in a designated area to finish sorting.
2. The multi-topology workpiece sorting method based on grabbing prediction for a two-arm robot as claimed in claim 1, wherein: reading an RGB-D image comprises the steps of:
the robot controller sends an instruction to the omnibearing mobile platform, and the robot moves to a designated sorting station;
the robot controller sends an instruction to the robot vision system, and the RGB-D camera starts to shoot images at a speed of 20 fps;
the robot controller reads the RGB-D image from the robot vision system.
3. The multi-topology workpiece sorting method based on grabbing prediction for a two-arm robot as claimed in claim 1, wherein: the method for dividing the RGB-D image into the RGB image and the depth image and respectively carrying out standardization processing comprises the following steps:
respectively calculating the average value and the standard deviation of all pixel data of the RGB image and the depth image;
and respectively traversing the pixel data of the RGB image and the depth image, and subtracting the average value from each pixel data and then dividing the average value by the standard deviation.
4. The multi-topology workpiece sorting method based on grabbing prediction for a two-arm robot as claimed in claim 1, wherein: in a grab-predict two-channel neural network,
the network architecture adopts a convolution module, a residual error module and a convolution permutation module;
the network input end is a dual-channel image, and a characteristic diagram of the channel image is extracted through respective channels;
the input image of the network is adjusted in size to adapt to the configuration of the convolution layer at the input end of the network, and meanwhile, the length-width ratio of the original image is not changed;
the output end of the network is a user-defined grabbing action parameter.
5. The multi-topology workpiece sorting method based on grabbing prediction for double-arm robot as claimed in claim 4, wherein: the user-defined grabbing action parameters are obtained through the following formula:
Action＝F(cp,θ,w,r)；
the method comprises the following steps that an Action is a grabbing Action of the double-arm robot, F () is mapping from neural network output information to the grabbing Action of the double-arm robot, cp is a central point of a position where a workpiece can be grabbed, and theta is a rotating angle of the tail end of an mechanical arm around an axis; w is the opening width of the mechanical arm paw; and r is the evaluation score of the mechanical arm grabbing action.
6. The multi-topology workpiece sorting method based on grabbing prediction for double-arm robot as claimed in claim 4, wherein: the resizing of the input image of the network comprises the steps of:
calculating the maximum value of the image width Iw and the height Ih, and dividing the maximum value by the image target size Is to obtain an image adjustment ratio Ir;
if the image width Iw Is not less than the height Ih, adjusting the image width Iw to be an image target size Is, and adjusting the image height Ih to be a value obtained by dividing the image height Ih by an image adjustment ratio Ir;
if the image width Iw Is smaller than the height Ih, the image width Iw Is adjusted to be the value of dividing the image height Iw by the image adjustment ratio Ir, and the image height Ih Is adjusted to be the image target size Is.
7. The multi-topology workpiece sorting method based on grabbing prediction for a two-arm robot as claimed in claim 1, wherein: the method for constructing the workpiece data set required by network training comprises the following steps:
placing workpieces with various topological structures on a workpiece table;
the depth camera is fixed at the tail end of the mechanical arm, generates a motion track according to Gaussian distribution, moves along the track, and controls the depth camera to shoot images while moving;
and splitting the acquired image into an RGB image and a depth image, and manually marking the grabbing action parameters.
8. The multi-topology workpiece sorting method based on grabbing prediction for a two-arm robot as claimed in claim 1, wherein: the method for grabbing the workpiece and placing the workpiece in the designated area by adopting the multi-process double-mechanical-arm cooperative grabbing control method according to the grabbing position of the workpiece converted into the mechanical arm coordinate system comprises the following steps:
(1) dividing the desktop into a middle area to be sorted, a left workpiece placing area and a right workpiece placing area, placing the workpieces in the middle area to be sorted, and defining the left workpiece placing area and the right workpiece placing area as designated workpiece placing positions;
(2) the robot controller sends a sorting starting instruction to the two mechanical arms respectively in a multi-process communication mode, and simultaneously sends information of the current position of a workpiece, the opening and closing angle of the paw and the placement position of the workpiece;
(3) after receiving the instruction, the two mechanical arms move to the preparation positions;
(4) the left mechanical arm starts to enter the middle to-be-sorted area to grab the workpieces, meanwhile, the robot controller locks the middle to-be-sorted area, the other mechanical arm is forbidden to enter the middle to-be-sorted area, the left mechanical arm returns to a preparation position after grabbing the workpieces, and the robot controller unlocks the middle to-be-sorted area;
(5) the left mechanical arm puts the grabbed workpieces to a specified workpiece placing position and returns to a preparation position, meanwhile, the right mechanical arm starts to enter a middle to-be-sorted area to grab the workpieces, the controller locks the middle to-be-sorted area, the other mechanical arm is forbidden to enter the middle to-be-sorted area, the right mechanical arm returns to the preparation position after grabbing the workpieces, and the robot controller unlocks the middle to-be-sorted area;
(6) the right mechanical arm places the grabbed workpiece at the appointed workpiece placing position and returns to the preparation position, and meanwhile, the left mechanical arm repeats the step (4);
(7) and (5) repeating the steps (4) to (6) until the workpieces in the area to be sorted in the middle are sorted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111642690.7A CN114474050A (en) | 2021-12-29 | 2021-12-29 | Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111642690.7A CN114474050A (en) | 2021-12-29 | 2021-12-29 | Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114474050A true CN114474050A (en) | 2022-05-13 |
Family
ID=81508970
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111642690.7A Pending CN114474050A (en) | 2021-12-29 | 2021-12-29 | Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114474050A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108115671A (en) * | 2016-11-26 | 2018-06-05 | 沈阳新松机器人自动化股份有限公司 | Tow-armed robot control method and system based on 3D visual sensors |
| CN108520535A (en) * | 2018-03-26 | 2018-09-11 | 天津大学 | Object classification method based on depth recovery information |
| CN111598172A (en) * | 2020-05-18 | 2020-08-28 | 东北大学 | Dynamic target grabbing posture rapid detection method based on heterogeneous deep network fusion |
| CN111906784A (en) * | 2020-07-23 | 2020-11-10 | 湖南爱米家智能科技有限公司 | Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method |
| CN113814973A (en) * | 2021-09-10 | 2021-12-21 | 北京精密机电控制设备研究所 | Sorting task control method for double-arm robot |
| CN113838042A (en) * | 2021-09-30 | 2021-12-24 | 清华大学 | Double-mechanical-arm operation question and answer method and device, electronic equipment and storage medium |
-
2021
- 2021-12-29 CN CN202111642690.7A patent/CN114474050A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108115671A (en) * | 2016-11-26 | 2018-06-05 | 沈阳新松机器人自动化股份有限公司 | Tow-armed robot control method and system based on 3D visual sensors |
| CN108520535A (en) * | 2018-03-26 | 2018-09-11 | 天津大学 | Object classification method based on depth recovery information |
| CN111598172A (en) * | 2020-05-18 | 2020-08-28 | 东北大学 | Dynamic target grabbing posture rapid detection method based on heterogeneous deep network fusion |
| CN111906784A (en) * | 2020-07-23 | 2020-11-10 | 湖南爱米家智能科技有限公司 | Pharyngeal swab double-arm sampling robot based on machine vision guidance and sampling method |
| CN113814973A (en) * | 2021-09-10 | 2021-12-21 | 北京精密机电控制设备研究所 | Sorting task control method for double-arm robot |
| CN113838042A (en) * | 2021-09-30 | 2021-12-24 | 清华大学 | Double-mechanical-arm operation question and answer method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108858199B (en) | Method for grabbing target object by service robot based on vision | |
| CN106737692B (en) | Mechanical gripper grabbing planning method based on depth projection and control device | |
| CN110640730B (en) | Method and system for generating three-dimensional model for robot scene | |
| EP2657863B1 (en) | Methods and computer-program products for generating grasp patterns for use by a robot | |
| CN109483573A (en) | Machine learning device, robot system and machine learning method | |
| CN111515945A (en) | Control method, system and device for mechanical arm visual positioning sorting and grabbing | |
| CN111462154B (en) | Target positioning method and device based on depth vision sensor and automatic grabbing robot | |
| CN111923053A (en) | Industrial robot object grabbing teaching system and method based on depth vision | |
| CN111347411B (en) | Two-arm cooperative robot three-dimensional visual recognition grabbing method based on deep learning | |
| US20220063099A1 (en) | Framework of robotic online motion planning | |
| CN113814986B (en) | Method and system for controlling SCARA robot based on machine vision | |
| CN112873205A (en) | Industrial robot disordered grabbing method based on real-time switching of double clamps | |
| US20230166398A1 (en) | Collision handling methods in grasp generation | |
| CN114714365B (en) | Disordered workpiece grabbing method and system based on cloud platform | |
| CN116673963A (en) | Double mechanical arm cooperation flexible assembly system and method for unordered breaker parts | |
| CN114347008A (en) | Industrial robot-based method and device for grabbing workpieces out of order and intelligent terminal | |
| CN117340895A (en) | Mechanical arm 6-DOF autonomous grabbing method based on target detection | |
| CN114474050A (en) | Grabbing prediction-based workpiece sorting method of double-arm robot with multiple topological structures | |
| Larsen et al. | Full automatic path planning of cooperating robots in industrial applications | |
| US20170076909A1 (en) | System and method for providing real-time visual feedback to control multiple autonomous nano-robots | |
| CN114407011B (en) | Special-shaped workpiece grabbing planning method, planning device and special-shaped workpiece grabbing method | |
| CN117103276A (en) | Precise grabbing method and system for robot | |
| Prew et al. | Evaluating Gaussian Grasp Maps for Generative Grasping Models | |
| CN115556102B (en) | Robot sorting and planning method and planning equipment based on visual recognition | |
| Zhang et al. | Grasping Prediction-Based Workpiece Sorting Method for Dual-Arm Robots with Multiple Topologies |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |