CN115446839A - Mechanical arm multi-type article grabbing method, system and device based on vision system - Google Patents
Mechanical arm multi-type article grabbing method, system and device based on vision system Download PDFInfo
- Publication number
- CN115446839A CN115446839A CN202211228721.9A CN202211228721A CN115446839A CN 115446839 A CN115446839 A CN 115446839A CN 202211228721 A CN202211228721 A CN 202211228721A CN 115446839 A CN115446839 A CN 115446839A
- Authority
- CN
- China
- Prior art keywords
- mechanical arm
- grabbing
- coordinate system
- camera
- tail end
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 238000006243 chemical reaction Methods 0.000 claims abstract description 40
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000004519 manufacturing process Methods 0.000 claims abstract description 20
- 230000000007 visual effect Effects 0.000 claims description 23
- 238000013461 design Methods 0.000 description 5
- 238000012937 correction Methods 0.000 description 3
- 238000009776 industrial production Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000009133 cooperative interaction Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000001179 sorption measurement Methods 0.000 description 1
Images
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1612—Programme controls characterised by the hand, wrist, grip control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1628—Programme controls characterised by the control loop
- B25J9/163—Programme controls characterised by the control loop learning, adaptive, model based, rule based expert control
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1656—Programme controls characterised by programming, planning systems for manipulators
- B25J9/1664—Programme controls characterised by programming, planning systems for manipulators characterised by motion, path, trajectory planning
Landscapes
- Engineering & Computer Science (AREA)
- Robotics (AREA)
- Mechanical Engineering (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Orthopedic Medicine & Surgery (AREA)
- Image Analysis (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a method, a system and a device for grabbing various types of articles by a mechanical arm based on a vision system, belonging to the technical field of vision system application.A vision system is established by acquiring high-definition images and depth information through a high-definition camera and a depth camera, a conversion relation between an image coordinate system and a space coordinate system is obtained by calibrating the image coordinate system of the camera and the space coordinate system at the tail end of the mechanical arm, and the conversion relation between the depth information and the space coordinate system is calibrated at the same time to accurately determine the position of an object; carrying out object identification on the high-definition image to acquire object information; aiming at the object identification result and the accurate coordinate information of the object obtained by the processing of the vision system, positioning the position to be reached by the tail end of the mechanical arm, planning the motion track, selecting the corresponding tail end type, and tracking the corresponding object; and grabbing the object to the designated position to finish grabbing the objects of multiple types. The invention can ensure the grabbing accuracy and improve the grabbing efficiency of objects on the production line.
Description
Technical Field
The invention relates to the technical field of visual system application, in particular to a mechanical arm multi-type article grabbing method, system and device based on a visual system.
Background
Along with the development of science and technology, the demand of many trades to automation is more and more, and the arm has also more and more applied in the trade production, like fields such as automobile manufacturing, goods are transported, intelligent letter sorting, public safety. However, the traditional industrial mechanical arm can only move according to a motion track and a position which are planned in advance, and cannot acquire external information, so that the application range is limited. In order to increase the flexibility and the usability of the mechanical arm, research on cooperative interaction of a vision system and the mechanical arm is rapidly developed in recent years, such as robot adaptive vision tracking control, an image jacobian matrix which is provided through the deviation of a current image and a specified image and the nonlinear relation between the motion of the mechanical arm and does not depend on camera calibration, and further the motion of the mechanical arm is controlled.
In the current stage, the mechanical arm based on the vision system is used for grabbing, the design of the tail end is single, and the simultaneous processing of objects in various forms cannot be realized simultaneously, so that the objects which are processed on one production line need to be of the same type. The traditional mechanical arm grabbing is a mode of executing a set of fixed motion tracks and positions, but with the development of industrial production, the requirement of production cannot be met by simple grabbing, and the important direction of research is to process object information and automatically perform the next operation.
Disclosure of Invention
The technical task of the invention is to provide a mechanical arm multi-type article grabbing method based on a vision system, which can ensure grabbing accuracy and improve grabbing efficiency of objects on a production line.
The technical scheme adopted by the invention for solving the technical problems is as follows:
a multi-type article grabbing method of a mechanical arm based on a vision system comprises the steps that a high-definition image and depth information are obtained through a high-definition camera and a depth camera to establish the vision system, a conversion relation between an image coordinate system and a space coordinate system is obtained by calibrating the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm, the conversion relation between the depth information and the space coordinate system is calibrated, and the position of an object is accurately determined;
carrying out object identification on the high-definition image to acquire object information; the next grabbing and moving process is facilitated; aiming at the object identification result and the accurate coordinate information of the object obtained by the processing of the vision system, the position to be reached by the tail end of the mechanical arm is positioned, the motion track is planned, the corresponding tail end type is selected, the corresponding object is tracked, and the stability and the accuracy of the operation are ensured; and grabbing the object to the designated position to finish grabbing the objects of multiple types.
According to the method, the high-definition camera and the depth camera are combined to form a vision processing system, hand-eye calibration of eyes outside hands is carried out on the high-definition camera and the tail end of the mechanical arm, conversion of a camera pixel coordinate system and a mechanical arm space coordinate system is achieved, the depth camera is used for assisting in judging the distance of an object, and the tail end of the mechanical arm can conveniently grab and offset the object more accurately. The method acquires image information through a vision system, carries out object identification, can accurately identify various objects to be acquired, sets different terminal processing modes aiming at the various objects, and increases the grabbing efficiency under a complex object scene by adopting different terminal processing modes according to an identification result; the object video information is acquired through the visual system, the object to be grabbed is tracked by combining an object tracking algorithm, the grabbing accuracy is guaranteed, and the assembly line object grabbing efficiency is improved.
Preferably, the high-definition camera and the depth camera are both arranged outside the mechanical arm, namely eyes are outside hands;
according to the calibration principle of eyes outside the hand, calibrating an image coordinate system of a camera and a space coordinate system of the tail end of the mechanical arm, keeping the coordinate system of the tail end of the mechanical arm and a calibration plate unchanged in the calibration process, and solving the conversion relation between the image coordinate system and the space coordinate system of the tail end of the mechanical arm;
the depth information of the depth camera is also calibrated with the terminal space coordinate system, and the conversion relation between the depth information and the terminal space coordinate system is determined, so that the position coordinate of the terminal which should move is obtained when the image information is processed.
Preferably, when the object passes through the field of view of the high-definition camera, the Yolov3-Tiny network is used for carrying out object recognition on the image.
In industrial production, efficiency is an important aspect in production, and YOLOV3-Tiny removes some characteristic layers on the basis of YOLOV3, only retains 2 independent prediction branches, and improves the identification speed. According to the method, the objects are identified, so that the various objects can be grabbed on the same production line, and the use scene of the grabbing process is expanded.
Preferably, the corresponding object is tracked according to a KCF algorithm.
The KCF is a discriminant tracking method, firstly, a target detector is trained, the target detector is used for detecting whether the next frame prediction position is a target, and then a new detection result is used for updating a training set so as to update the target detector. The KCF algorithm converts the operation of the matrix into a Hadamad product of the vector, greatly reduces the operation amount and can meet the requirement of real-time performance.
Preferably, the result of the object identification and the object accurate coordinate information processed by the vision system,
after the vision system acquires the high-definition image information, the actual position of the object is obtained through coordinate conversion, and the depth information acquired by the depth camera is used for carrying out coordinate correction and accurately acquiring the distance from the object to the tail end of the mechanical arm, so that the position coordinate of the object is accurately acquired.
Further, after specific coordinate information of the object is acquired, the specific coordinate information is converted into coordinates in a terminal coordinate system, the distance from the object to the terminal and the movement speed of a production line are determined by combining depth information, the specific position to which the terminal is to reach is determined, after the specific position to which the terminal is to move is obtained, the movement track of the mechanical arm is solved through an inverse kinematics algorithm of the mechanical arm, and the object is grabbed and placed according to the type of the terminal and the movement track of the object.
Preferably, the mechanical arm is a six-axis mechanical arm, and the inverse kinematics solution is a TracIK kinematics solver. The TracIK is a kinematics plug-in based on numerical solution, has high solving efficiency and wide range of planning results, and is suitable for solving the six-axis mechanical arm inverse kinematics.
Furthermore, the object is grabbed and placed according to the tail end type and the object motion track, the tail end grabbing devices in different forms are grabbed and designed for various objects, the program judges which type of tail end grabbing device is used according to the object identification result, the motion track can be planned according to different tail ends, and the tail end device which is most suitable for grabbing is used; after the end device grabs the object, different end positions are placed according to different objects, and the object recognition and grabbing process is completed.
The invention also claims a multi-type article grabbing system of the mechanical arm based on the visual system, which comprises a high-definition camera, a depth camera and the mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the visual system, the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, and meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated to accurately determine the position of an object;
carrying out object identification on the high-definition image to acquire object information; aiming at the object identification result and the accurate coordinate information of the object obtained by the processing of the vision system, positioning the position to be reached by the tail end of the mechanical arm, planning the motion track, selecting the corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to complete grabbing of the objects of multiple types;
the system realizes the mechanical arm multi-type article grabbing method based on the vision system.
The invention also claims a multi-type article grabbing device of the mechanical arm based on the visual system, which comprises a high-definition camera, a depth camera and the mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the visual system, the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, and meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated to accurately determine the position of an object;
carrying out object identification on the high-definition image to acquire object information; aiming at the object identification result and the accurate coordinate information of the object obtained by the processing of the vision system, positioning the position to be reached by the tail end of the mechanical arm, planning the motion track, selecting the corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to complete grabbing of the objects of multiple types;
the device realizes the multi-type article grabbing method of the mechanical arm based on the vision system.
Compared with the prior art, the mechanical arm multi-type article grabbing method, system and device based on the vision system have the following beneficial effects:
according to the method, when the visual system is built, a mode of combining a high-definition camera and a depth camera is used, the position of an object is accurately judged through the combination of image information and depth information, the grabbing stability and efficiency are greatly improved, and the robustness of the grabbing system is improved.
The real-time performance is considered in the aspect of algorithm design by considering the requirement on speed in an actual application scene. The selection of the YOLOV3-Tiny algorithm used in object identification, the KCF algorithm for object tracking and the inverse kinematics solving plugin has real-time performance, the use efficiency of the grabbing system is further improved, and the use scene of the system is widened.
The problem of traditional arm can not the perception external environment, hardly make the reaction according to external environment change is solved, through real-time vision system, can perceive external environment and change, pursue in real time the motion situation and the position of object, handle various circumstances in a flexible way, reduced the condition of grabbing by mistake and neglected grabbing, improved the usability of system.
Through vision system, increased object recognition function, conveniently handled the scene of polymorphic type object at the assembly line simultaneously, solved traditional assembly line and snatched the single problem of object, improved production efficiency, had very big help to being suitable for the expansion and the optimization of scene.
According to actual conditions, various end device types are designed for various objects, different forms such as adsorption and clamping can be achieved, the targeted end devices are suitable for different objects, the diversity of grabbing functions is achieved, and the stability and the production efficiency of object grabbing are improved.
Drawings
Fig. 1 is a process diagram of an implementation of a multi-type article grabbing method of a robot arm based on a vision system according to an embodiment of the present invention.
Detailed Description
The present invention is further illustrated by the following examples.
The mechanical arm is grabbed through interaction of the visual system and the mechanical arm, the mechanical arm visual grabbing scheme can be divided into two types according to the difference of the cameras, one type is based on a monocular visual grabbing scheme, a single camera is directly used for acquiring a plane image to directly grab, the mode is easy to design, high-precision grabbing cannot be achieved, and the robustness is poor. The other method is to acquire a high-definition image and a depth image by using a high-definition camera and a depth camera, and to grasp an object by processing a point cloud coordinate through depth information.
The embodiment of the invention provides a mechanical arm multi-type article grabbing method based on a visual system, which comprises the steps of acquiring a high-definition image and depth information through a high-definition camera and a depth camera to establish the visual system, obtaining a conversion relation between an image coordinate system and a space coordinate system through calibrating the image coordinate system of the camera and the space coordinate system of the tail end of a mechanical arm, calibrating the conversion relation between the depth information and the space coordinate system at the same time, and accurately determining the position of an object;
carrying out object identification on the high-definition image to acquire object information; the next grabbing and moving process is facilitated; aiming at the object identification result and the accurate coordinate information of the object obtained by the processing of the vision system, the position to be reached by the tail end of the mechanical arm is positioned, the motion track is planned, the corresponding tail end type is selected, the corresponding object is tracked, and the stability and the accuracy of the operation are ensured; and grabbing the object to the designated position to finish grabbing the objects of multiple types.
According to the method, the high-definition camera and the depth camera are combined to form a vision processing system, the high-definition camera and the tail end of the mechanical arm are subjected to hand-eye calibration with eyes out of the hand, conversion of a camera pixel coordinate system and a mechanical arm space coordinate system is achieved, the depth camera is used for assisting in judging the distance of an object, and the tail end of the mechanical arm can conveniently grab and offset the object more accurately. The method obtains image information through a vision system, carries out object identification, can accurately identify various objects to be taken, sets different terminal processing modes aiming at the various objects, adopts different terminal processing modes according to the identification result, and increases the grabbing efficiency under a complex object scene; the object video information is obtained through the visual system, the object to be grabbed is tracked by combining an object tracking algorithm, the grabbing accuracy is guaranteed, and the assembly line object grabbing efficiency is improved.
The method adopts six-axis mechanical arms, designs various types of tail ends aiming at various types of objects, and selects different tail end processing modules according to the object types identified by the objects.
The method comprises the following concrete implementation processes:
1. and (5) building a visual system and calibrating.
In order to be convenient to use, the high-definition camera and the depth camera are both placed outside the mechanical arm, an image coordinate system of the camera and a space coordinate system of the tail end of the mechanical arm are calibrated according to a calibration principle that eyes are outside the hands, the tail end coordinate system of the mechanical arm and a calibration plate are kept unchanged in the calibration process, and the conversion relation between the image coordinate system and the tail end space coordinate system of the mechanical arm is solved. The depth information of the depth camera is also calibrated with the terminal space coordinate system, and the conversion relation between the depth information and the terminal space coordinate system is determined, so that the position coordinate of the terminal which should move is obtained when the image information is processed.
2. And acquiring image information for object recognition.
The method uses a YOLOV3-Tiny network to identify objects, and uses the YOLOV3-Tiny network to identify objects in images when the objects pass through the field range of the high-definition camera.
In industrial production, efficiency is an important aspect in production, and YOLOV3-Tiny removes some characteristic layers on the basis of YOLOV3, only retains 2 independent prediction branches, and improves the identification speed. According to the method, the objects are identified, so that the various objects can be grabbed on the same production line, and the use scene of the grabbing process is expanded.
3. And determining the position of the object according to the high-definition image information and the depth information.
After the coordinate calibration in the step 1, the conversion relation between the image coordinate and the terminal space coordinate is determined, after the vision system acquires the image information, the actual position of the object is obtained through coordinate conversion, the depth information acquired by the depth camera is used for carrying out coordinate correction and accurately acquiring the distance from the object to the terminal of the mechanical arm, and therefore the position of the object is accurately acquired.
The method is used for recognizing and grabbing the object on the production line, and in order to ensure the accuracy of object grabbing, the method uses a KCF algorithm to track the recognized object. The KCF is a discriminant tracking method, firstly, a target detector is trained, the target detector is used for detecting whether the next frame prediction position is a target or not, and then a new detection result is used for updating a training set so as to update the target detector. The KCF algorithm converts the operation of the matrix into a Hadamad product of the vector, greatly reduces the operation amount and can meet the requirement of real-time performance.
4. And performing coordinate conversion to determine the arrival position of the tail end of the mechanical arm and performing motion planning.
And (4) after acquiring the specific coordinate information of the object according to the step (3), converting the information into coordinates under a terminal coordinate system, determining the distance from the object to the terminal and the movement speed of the production line by combining the depth information, determining the specific position to which the terminal is to reach, and after obtaining the specific position to which the terminal is to move, solving the movement track of the six-axis mechanical arm through an inverse kinematics algorithm of the mechanical arm.
The inverse kinematics solution used in the method is a TracIK kinematics solver, the TracIK is a kinematics plug-in based on numerical solution, the solution efficiency is high, the planning result range is wide, and the method is suitable for solving the six-axis mechanical arm inverse kinematics.
5. And grabbing and placing the object according to the type of the tail end and the motion track of the object.
The method comprises the steps of designing different forms of tail end grabbing devices for grabbing various objects, judging which form of tail end grabbing device is used by a program according to the object identification result in the step 2, planning different tail ends when a motion track is planned in the step 4, and using the tail end device which is most suitable for grabbing. After the end device grabs the object, different end positions are placed according to different objects, and the object recognition and grabbing process is completed.
The embodiment of the invention also provides a mechanical arm multi-type article grabbing system based on the visual system, which comprises a high-definition camera, a depth camera and a mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the visual system, the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, and meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated to accurately determine the position of an object;
carrying out object identification on the high-definition image to acquire object information; aiming at the result of object identification and accurate coordinate information of the object obtained by processing of a vision system, positioning the position to be reached by the tail end of the mechanical arm, planning a motion track, selecting a corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to finish grabbing the objects of multiple types;
the system realizes the multi-type article grabbing method of the mechanical arm based on the vision system in the embodiment.
The system is characterized in that a high-definition camera and a depth camera form a hardware basis of a visual system, object identification is carried out through image information acquired by the camera and the depth camera, the specific position of an object is determined, the object is tracked, position information and object classification information are interacted with a mechanical arm, the mechanical arm plans a motion track according to the specific tail end position, a corresponding tail end device is selected for grabbing, and the mechanical arm is placed at the specific position.
The high-definition camera and the depth camera are combined to provide images required by visual processing together, so that accurate acquisition of image information is facilitated, the depth information provided by the depth camera assists in processing the images acquired by the high-definition camera, processing of two-dimensional images is improved to processing of three-dimensional images, and accuracy is improved.
Algorithms such as object recognition, object tracking and the like are added in the grabbing process of the traditional mechanical arm, so that the usability of processing objects on the production line is higher, and the production efficiency is improved.
The high-definition camera and the depth camera are both arranged outside the mechanical arm, namely eyes are outside hands;
according to the calibration principle of eyes outside the hand, calibrating an image coordinate system of a camera and a space coordinate system of the tail end of the mechanical arm, keeping the coordinate system of the tail end of the mechanical arm and a calibration plate unchanged in the calibration process, and solving the conversion relation between the image coordinate system and the space coordinate system of the tail end of the mechanical arm;
the depth information of the depth camera is also calibrated with the terminal space coordinate system, and the conversion relation between the depth information and the terminal space coordinate system is determined, so that the position coordinate of the terminal which should move is obtained when the image information is processed.
When an object passes through the visual field range of the high-definition camera, carrying out object identification on the image by using a YOLOV3-Tiny network; tracking the corresponding object according to a KCF algorithm; the mechanical arm is a six-axis mechanical arm, and the inverse kinematics solution is a TracIK kinematics solver.
The result of the object identification and the object accurate coordinate information processed by the vision system,
after the vision system acquires the high-definition image information, the actual position of the object is obtained through coordinate conversion, and the depth information acquired by the depth camera is used for carrying out coordinate correction and accurately acquiring the distance from the object to the tail end of the mechanical arm, so that the position coordinate of the object is accurately acquired.
The method comprises the steps of obtaining specific coordinate information of an object, converting the information into coordinates under a terminal coordinate system, determining the distance between the object and the terminal and the movement speed of a production line by combining depth information, determining the specific position to which the terminal is to reach, solving the movement track of a mechanical arm through an inverse kinematics algorithm of the mechanical arm after obtaining the specific position to which the terminal is to move, and grabbing and placing the object according to the type of the terminal and the movement track of the object.
The object is grabbed and placed according to the type of the tail end and the motion track of the object, the tail end grabbing devices in different forms are grabbed and designed for various objects, the program judges which type of tail end grabbing device is used according to the object identification result, planning can be carried out according to different tail ends when the motion track is planned, and the tail end device which is most suitable for grabbing is used; after the end device grabs the object, different end positions are placed according to different objects, and the object recognition and grabbing process is completed.
Aiming at various types of objects, the system designs various types of tail ends, and selects different tail end processing modules for the object types identified by the objects, thereby greatly enriching the types of the grabbed objects and meeting the diversified grabbing requirements.
The invention also claims a mechanical arm multi-type article grabbing device based on the vision system, which comprises a high-definition camera, a depth camera and a mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the vision system, the image coordinate system of the camera and the space coordinate system at the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, and meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated to accurately determine the position of an object;
carrying out object identification on the high-definition image to acquire object information; aiming at the result of object identification and accurate coordinate information of the object obtained by processing of a vision system, positioning the position to be reached by the tail end of the mechanical arm, planning a motion track, selecting a corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to finish grabbing the objects of multiple types;
the device realizes the mechanical arm multi-type article grabbing method based on the vision system.
The high-definition camera and the depth camera are both arranged outside the mechanical arm, namely eyes are outside hands;
according to the calibration principle of eyes outside the hand, calibrating an image coordinate system of a camera and a space coordinate system of the tail end of the mechanical arm, keeping the coordinate system of the tail end of the mechanical arm and a calibration plate unchanged in the calibration process, and solving the conversion relation between the image coordinate system and the space coordinate system of the tail end of the mechanical arm;
the depth information of the depth camera is also calibrated with the terminal space coordinate system, and the conversion relation between the depth information and the terminal space coordinate system is determined, so that the position coordinate of the terminal which should move is obtained when the image information is processed.
When an object passes through the visual field range of the high-definition camera, carrying out object identification on the image by using a YOLOV3-Tiny network; tracking the corresponding object according to a KCF algorithm; the mechanical arm is a six-axis mechanical arm, and the inverse kinematics solution is a TracIK kinematics solver.
The result of the object identification and the object accurate coordinate information processed by the vision system,
after the vision system acquires high-definition image information, the actual position of the object is obtained through coordinate conversion, and the distance from the object to the tail end of the mechanical arm is corrected and accurately acquired through the depth information acquired by the depth camera, so that the position coordinate of the object is accurately acquired.
The method comprises the steps of obtaining specific coordinate information of an object, converting the information into coordinates in a terminal coordinate system, determining the distance from the object to the terminal and the movement speed of a production line by combining depth information, determining the specific position to which the terminal is to reach, solving the movement track of a mechanical arm through an inverse kinematics algorithm of the mechanical arm after obtaining the specific position to which the terminal is to move, and grabbing and placing the object according to the type of the terminal and the movement track of the object.
The object is grabbed and placed according to the type of the tail end and the motion track of the object, the tail end grabbing devices in different forms are grabbed and designed for various objects, the program judges which type of tail end grabbing device is used according to the object identification result, planning can be carried out according to different tail ends when the motion track is planned, and the tail end device which is most suitable for grabbing is used; after the end device grabs the object, different end positions are placed according to different objects, and the object recognition and grabbing process is completed.
The present invention can be easily implemented by those skilled in the art from the above detailed description. It should be understood, however, that the intention is not to limit the invention to the particular embodiments described. On the basis of the disclosed embodiments, a person skilled in the art can combine different technical features at will, thereby implementing different technical solutions.
Except for the technical features described in the specification, the method is known by the technical personnel in the field.
Claims (10)
1. A mechanical arm multi-type article grabbing method based on a vision system is characterized in that a vision system is established by acquiring high-definition images and depth information through a high-definition camera and a depth camera, a conversion relation between an image coordinate system and a space coordinate system is obtained by calibrating the image coordinate system of the camera and the space coordinate system of the tail end of a mechanical arm, and the conversion relation between the depth information and the space coordinate system is calibrated at the same time to accurately determine the position of an object;
carrying out object identification on the high-definition image to acquire object information; aiming at the result of object identification and accurate coordinate information of the object obtained by processing of a vision system, positioning the position to be reached by the tail end of the mechanical arm, planning a motion track, selecting a corresponding tail end type, and tracking the corresponding object; and grabbing the object to the designated position to finish grabbing the objects of multiple types.
2. The multi-type item grabbing method of the mechanical arm based on the vision system as claimed in claim 1, wherein said high definition video camera and said depth camera are both placed outside the mechanical arm, i.e. eyes are outside the hand;
according to the calibration principle of eyes outside the hand, calibrating an image coordinate system of a camera and a space coordinate system of the tail end of the mechanical arm, keeping the coordinate system of the tail end of the mechanical arm and a calibration plate unchanged in the calibration process, and solving the conversion relation between the image coordinate system and the space coordinate system of the tail end of the mechanical arm;
the depth information of the depth camera is also calibrated with the terminal space coordinate system, and the conversion relation between the depth information and the terminal space coordinate system is determined, so that the position coordinate of the terminal which should move is obtained when the image information is processed.
3. The method for grabbing multi-type items by a mechanical arm based on a vision system as claimed in claim 1 or 2, wherein the YOLOV3-Tiny network is used to perform object recognition on the image when the object passes through the field of view of the high definition camera.
4. A multi-type item grabbing method by a robotic arm based vision system according to claim 1 or 2, characterized in that the corresponding objects are tracked according to KCF algorithm.
5. The method for grabbing by a robotic arm based on a vision system as claimed in claim 1, wherein said object recognition result and object accurate coordinate information processed by the vision system,
after the vision system acquires high-definition image information, the actual position of the object is obtained through coordinate conversion, and the distance from the object to the tail end of the mechanical arm is corrected and accurately acquired through the depth information acquired by the depth camera, so that the position coordinate of the object is accurately acquired.
6. The method for grabbing multi-type items by a mechanical arm based on a vision system as claimed in claim 5, wherein the obtained specific coordinate information of the object is converted into coordinates in a terminal coordinate system, the distance from the end of the object to the terminal and the moving speed of a production line are determined by combining depth information, the specific position to which the terminal is to reach is determined, the specific position to which the terminal is to move is obtained, the moving track of the mechanical arm is solved by an inverse kinematics algorithm of the mechanical arm, and the object is grabbed and placed according to the type of the terminal and the moving track of the object.
7. The method for grabbing a multi-type item by a mechanical arm based on a vision system as claimed in claim 6, wherein said mechanical arm is a six-axis mechanical arm, and said inverse kinematics solution is a TracIK kinematics solver.
8. The method for grabbing multi-type objects by a mechanical arm based on a vision system as claimed in claim 6, wherein said object is grabbed and placed according to the end type and the object motion track, different forms of end grabbing devices are grabbed and designed for multiple objects, according to the result of object recognition, the program determines which form of end grabbing device is used, the motion track is planned for different ends, and the most suitable end device is used for grabbing; after the end device grabs the object, different end positions are placed according to different objects, and the object recognition and grabbing process is completed.
9. A mechanical arm multi-type article grabbing system based on a vision system is characterized by comprising a high-definition camera, a depth camera and a mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the vision system, the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated, and the position of an object is accurately determined;
carrying out object identification on the high-definition image to acquire object information; aiming at the result of object identification and accurate coordinate information of the object obtained by processing of a vision system, positioning the position to be reached by the tail end of the mechanical arm, planning a motion track, selecting a corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to finish grabbing the objects of multiple types;
the system realizes the multi-type object grabbing method of the mechanical arm based on the vision system according to any one of claims 1 to 8.
10. A multi-type article grabbing device of a mechanical arm based on a visual system is characterized by comprising a high-definition camera, a depth camera and the mechanical arm for grabbing articles, wherein the high-definition camera and the depth camera are used for acquiring high-definition images and depth information to establish the visual system, the image coordinate system of the camera and the space coordinate system of the tail end of the mechanical arm are calibrated to obtain the conversion relation between the image coordinate system and the space coordinate system, meanwhile, the conversion relation between the depth information and the space coordinate system is calibrated, and the position of an object is accurately determined;
carrying out object identification on the high-definition image to acquire object information; aiming at the result of object identification and accurate coordinate information of the object obtained by processing of a vision system, positioning the position to be reached by the tail end of the mechanical arm, planning a motion track, selecting a corresponding tail end type, and tracking the corresponding object; grabbing the object to a designated position to complete grabbing of the objects of multiple types;
the device realizes the multi-type object grabbing method of the mechanical arm based on the vision system according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211228721.9A CN115446839A (en) | 2022-10-08 | 2022-10-08 | Mechanical arm multi-type article grabbing method, system and device based on vision system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211228721.9A CN115446839A (en) | 2022-10-08 | 2022-10-08 | Mechanical arm multi-type article grabbing method, system and device based on vision system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115446839A true CN115446839A (en) | 2022-12-09 |
Family
ID=84309727
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211228721.9A Pending CN115446839A (en) | 2022-10-08 | 2022-10-08 | Mechanical arm multi-type article grabbing method, system and device based on vision system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115446839A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117841041A (en) * | 2024-02-05 | 2024-04-09 | 北京新雨华祺科技有限公司 | Mechanical arm combination device based on multi-arm cooperation |
| CN118322215A (en) * | 2024-05-20 | 2024-07-12 | 维宏感应(山东)科技有限公司 | Robot vision automatic grabbing system |
-
2022
- 2022-10-08 CN CN202211228721.9A patent/CN115446839A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117841041A (en) * | 2024-02-05 | 2024-04-09 | 北京新雨华祺科技有限公司 | Mechanical arm combination device based on multi-arm cooperation |
| CN117841041B (en) * | 2024-02-05 | 2024-07-05 | 北京新雨华祺科技有限公司 | Mechanical arm combination device based on multi-arm cooperation |
| CN118322215A (en) * | 2024-05-20 | 2024-07-12 | 维宏感应(山东)科技有限公司 | Robot vision automatic grabbing system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111496770B (en) | Intelligent carrying mechanical arm system based on 3D vision and deep learning and use method | |
| CN115446839A (en) | Mechanical arm multi-type article grabbing method, system and device based on vision system | |
| CN108399639B (en) | Rapid automatic grabbing and placing method based on deep learning | |
| CN107618030B (en) | Robot dynamic tracking grabbing method and system based on vision | |
| CN105234943B (en) | A kind of industrial robot teaching device and method of view-based access control model identification | |
| CN111347411B (en) | Two-arm cooperative robot three-dimensional visual recognition grabbing method based on deep learning | |
| CN107139178B (en) | Unmanned aerial vehicle and vision-based grabbing method thereof | |
| CN105729468B (en) | A kind of robotic workstation based on the enhancing of more depth cameras | |
| CN110900581B (en) | Four-degree-of-freedom mechanical arm vision servo control method and device based on RealSense camera | |
| CN112171661A (en) | Method for grabbing target object by mechanical arm based on visual information fusion | |
| CN112518748B (en) | Automatic grabbing method and system for visual mechanical arm for moving object | |
| CN113379849A (en) | Robot autonomous recognition intelligent grabbing method and system based on depth camera | |
| CN113269723B (en) | Unordered grabbing system for parts with three-dimensional visual positioning and manipulator cooperative work | |
| CN115070781B (en) | Object grabbing method and two-mechanical-arm cooperation system | |
| CN114347008A (en) | Industrial robot-based method and device for grabbing workpieces out of order and intelligent terminal | |
| CN105096341A (en) | Mobile robot pose estimation method based on trifocal tensor and key frame strategy | |
| CN115861780A (en) | Mechanical arm detection and grabbing method based on YOLO-GGCNN | |
| CN114074331A (en) | Disordered grabbing method based on vision and robot | |
| CN114770461A (en) | Monocular vision-based mobile robot and automatic grabbing method thereof | |
| CN117340929A (en) | Flexible clamping jaw grabbing and disposing device and method based on three-dimensional point cloud data | |
| Xu et al. | Industrial robot base assembly based on improved Hough transform of circle detection algorithm | |
| Wang et al. | Design of multi-robot cooperative transport system | |
| CN116330284A (en) | Automatic control method and system for mechanical arm suitable for grabbing blocked object | |
| CN211890823U (en) | Four-degree-of-freedom mechanical arm vision servo control system based on RealSense camera | |
| Ngo | Using HSV-based Approach for Detecting and Grasping an Object by the Industrial Mechatronic System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |