CN115366105A - Workpiece grabbing method, device, electronic device and storage medium - Google Patents

Abstract

Embodiments of the present application provide a workpiece grasping method, device, electronic equipment, and storage medium. The workpiece grasping method includes: acquiring a first workpiece image captured by a camera; determining at least one first feature from the first workpiece image point; according to the pixel coordinates and calibration information of at least one first feature point, determine the first space coordinate of the first workpiece in the space coordinate system of the manipulator; generate control information according to the first space coordinate, and send the control information to the machine arm, so that the robotic arm grabs the first workpiece according to the control information. This solution uses the mapping relationship between the pixel coordinate system and the space coordinate system to identify the position of the workpiece, and controls the robotic arm to automatically grab the workpiece, which can solve the problem of manually moving the target during the process of grabbing the workpiece. The method of placing the object at a specific angle and position takes a long time, resulting in a problem of low production efficiency.

Description

Workpiece grabbing method, device, electronic device and storage medium

technical field

The embodiments of the present application relate to the technical field of computer vision, and in particular to a workpiece grasping method, device, electronic equipment, and storage medium.

Background technique

In the production process of workpieces, in order to reduce labor costs and improve safety, it is necessary to use robotic arms instead of manual grasping of workpieces to transfer, flip, weld, etc. the workpieces. By simulating the way the human eye sees objects, and driving the robotic arm to complete the grasping of the workpiece. However, the grasping effect of the robotic arm is quite different from that of manual grasping.

At present, in order to deal with the above differences, a common method is to place the workpiece directly under the robotic arm, and manually place the workpiece at a specific angle and position to ensure the gripping effect of the robotic arm.

However, it takes a long time to manually arrange the position of the workpiece to a specific angle and position, resulting in low production efficiency.

Contents of the invention

In order to solve the above-mentioned technical problems, embodiments of the present invention provide a workpiece grasping method, device, electronic equipment, and storage medium, so as to at least solve or alleviate the above-mentioned problems.

According to the first aspect of the embodiments of the present application, there is provided a workpiece grasping method, including: acquiring a first workpiece image captured by a camera, wherein the first workpiece image includes an image of the first workpiece located on the stage image; determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate the position of the image of the first workpiece in the first workpiece image; according to the The pixel coordinates and calibration information of at least one first feature point determine the first spatial coordinates of the first workpiece in the spatial coordinate system of the manipulator, wherein the calibration information is used to indicate the pixels of the image captured by the camera The mapping relationship between the coordinate system and the space coordinate system; generate control information according to the first space coordinates, and send the control information to the robot arm, so that the robot arm can control the robot according to the control information The first workpiece is gripped.

According to a second aspect of the embodiments of the present application, there is provided an object grasping device, including: an acquisition module, configured to acquire a first workpiece image captured by a camera, wherein the first workpiece image includes An image of the first workpiece; an extraction module, configured to determine at least one first feature point from the image of the first workpiece, wherein the first feature point is used to indicate that the image of the first workpiece is in the image of the first workpiece A position in the workpiece image; a determination module, configured to determine the first spatial coordinates of the first workpiece in the spatial coordinate system of the robotic arm according to the pixel coordinates and the calibration information of the at least one first feature point, wherein, The calibration information is used to indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the space coordinate system; the capture module is used to generate control information according to the first space coordinates, and the The control information is sent to the robot arm, so that the robot arm grabs the first workpiece according to the control information.

According to a third aspect of the embodiments of the present application, an electronic device is provided, including: a processor, a communication interface, a memory, and a communication bus, and the processor, the memory, and the communication interface complete mutual communication through the communication bus; the memory is used to store At least one executable instruction, the executable instruction causes the processor to execute the operation corresponding to any method in the above multiple method embodiments.

According to a fourth aspect of the embodiments of the present application, a computer storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any method in the above-mentioned multiple method embodiments is implemented.

According to a fifth aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to any one of the above method embodiments.

It can be seen from the above technical solution that at least one first feature point determined from the first workpiece image can indicate the position of the image of the first workpiece in the first workpiece image, and the calibration information can indicate the pixel coordinate system of the image collected by the camera and The mapping relationship of the space coordinate system of the manipulator, through the calibration information and the pixel coordinates of the first feature point, can determine the position of the first workpiece in the space coordinate system of the manipulator, and then according to the position of the first work piece in the space coordinate system The position generates control information, and after sending the control information to the robot arm, the robot arm grabs the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the manipulator can be determined, the manipulator can automatically grab the workpiece randomly placed on the stage without placing the workpiece on the load at a specific angle and position. on the stage, thereby saving the time required to place the workpiece on the stage, thereby improving the efficiency of grasping the workpiece.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments described in the embodiments of the present application, and those skilled in the art can also obtain other drawings based on these drawings.

Fig. 1 is the flowchart of the workpiece grasping method of an embodiment of the present application;

FIG. 2 is a flow chart of a second spatial coordinate acquisition method according to an embodiment of the present application;

FIG. 3 is a flowchart of a method for determining an offset amount according to an embodiment of the present application;

FIG. 4 is a flowchart of a method for determining calibration information according to an embodiment of the present application;

Fig. 5 is a schematic diagram of the motion sequence of the mechanical arm according to an embodiment of the present application;

Fig. 6 is a schematic diagram of the coordinates of the movement point of the mechanical arm according to an embodiment of the present application;

Fig. 7 is a schematic diagram of an object grabbing device according to an embodiment of the present application;

Fig. 8 is a schematic diagram of an electronic device according to an embodiment of the present application.

List of reference signs:

100: Workpiece picking method 200: Control information generation method 300: Offset determination method

400: Calibration information determination method 700: Object grabbing device 701: Obtaining module

702: extracting module 703: determining module 704: grabbing module

800: Electronic equipment 801: Processor 802: Communication interface

803: memory 804: communication bus 805: program

101: Obtain the first workpiece image captured by the camera

102: Determine at least one first feature point from the first workpiece image

103: Determine the first space coordinate of the first workpiece in the space coordinate system of the robotic arm

104: Generate control information according to the first space coordinates, and send the control information to the robotic arm

201: Acquiring the second workpiece image pre-collected by the camera

202: Determine at least one second feature point from the second workpiece image

203: Determine the second space coordinate of the first workpiece in the space coordinate system

301: Determine the X-axis displacement offset of the position of the first workpiece in the space coordinate system corresponding to the standard position

302: Determine the Y-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position

303: Determine the Z-axis rotation offset of the position of the first workpiece in the spatial coordinate system relative to the standard position

401: Determine the calibration reference point on the second workpiece

402: Control the manipulator to perform N translations along the X-axis and/or Y-axis in the spatial coordinate system

403: Obtain the translation calibration image collected by the camera after each translation of the robotic arm

404: Control the robotic arm to rotate M times around the Z axis in the space coordinate system

405: Obtain the rotation calibration image collected by the camera after each rotation of the robotic arm

406: Determine calibration information

Detailed ways

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described The embodiments are only some of the embodiments of the present application, but not all of them. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the embodiments of the present application shall fall within the protection scope of the embodiments of the present application.

Workpiece grabbing method

Fig. 1 is a workpiece grasping method 100 according to an embodiment of the present application. As shown in Fig. 1, the workpiece grasping method 100 includes the following steps:

Step 101 , acquiring a first workpiece image captured by a camera.

The first workpiece image includes an image of a first workpiece, and the first workpiece may be a workpiece such as a circuit board. When collecting the image of the first workpiece, the image of the first workpiece is placed on the stage, and after the robot arm moves above the stage, the camera installed on the robot arm and moving synchronously with the robot arm takes an image of the first workpiece Acquisition to obtain the first workpiece image.

Step 102. Determine at least one first feature point from the first workpiece image.

After the image of the first workpiece to be captured captured by the camera is acquired, it is necessary to determine the position of the image of the first workpiece in the image of the first workpiece. Determine at least one first feature point from the first workpiece image, the first feature point refers to a point in the first workpiece image where the gray value changes sharply or a point with a large curvature on the edge of the first workpiece image. The first feature point may indicate the position of the image of the first workpiece to be captured in the image of the first workpiece.

Step 103. Determine the first space coordinate of the first workpiece in the space coordinate system of the robot arm.

The robotic arm has a corresponding spatial coordinate system based on which the robotic arm can move from one location to another. In order to enable the robot arm to grab the first workpiece, it is necessary to determine the first space coordinate of the first workpiece in the space coordinate system of the robot arm, so that the robot arm can determine the space coordinate of the first workpiece according to the first space coordinate of the first workpiece position in the system, and then grab the first workpiece.

The calibration information can indicate the mapping relationship between the pixel coordinate system of the image captured by the camera and the spatial coordinate system of the robot arm. Since the robot arm and the camera move synchronously, the calibration information is fixed. Since the pixel coordinates of each first feature point can indicate the position of the image of the first workpiece in the first workpiece image, according to the pixel coordinates and calibration information of each first feature point, the pixel coordinates of each first feature point can be transformed into is the space coordinate in the space coordinate system, and then obtains the first space coordinate that can indicate that the first job is in the space coordinate system.

Step 104: Generate control information according to the first space coordinates, and send the control information to the robotic arm.

The first space coordinate indicates the position of the first workpiece in the space coordinate system. According to the first space coordinate, the corresponding control information can be generated. After the control information is sent to the robot arm, the robot arm can move to the appropriate position according to the control information for the first workpiece. A workpiece is grasped. Control information includes, but is not limited to, commands such as grabbing, translation, rotation, and stretching.

In the embodiment of the present application, the at least one first feature point determined from the first workpiece image can indicate the position of the image of the first workpiece in the first workpiece image, and the calibration information can indicate the pixel coordinate system of the image collected by the camera The mapping relationship with the space coordinate system of the manipulator, through the calibration information and the pixel coordinates of the first feature point, the position of the first workpiece in the space coordinate system of the manipulator can be determined, and then according to the position of the first workpiece in the space coordinate system The position of the control information is generated, and after the control information is sent to the robot arm, the robot arm grabs the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the manipulator can be determined, the manipulator can automatically grab the workpiece randomly placed on the stage without placing the workpiece on the load at a specific angle and position. on the stage, thereby saving the time required to place the workpiece on the stage, thereby improving the efficiency of grasping the workpiece.

In a possible implementation manner, when the control information is generated according to the first space coordinates, a predetermined second space coordinates may be obtained, and the second space coordinates are used to indicate that the first workpiece is located at the standard position on the stage. The coordinates of the workpiece in the space coordinate system, and then according to the first space coordinates and the second space coordinates, determine the offset of the position of the first workpiece in the space coordinate system relative to the standard position, and then determine the offset as the control information.

It should be understood that the second space coordinates are obtained in advance and indicate the coordinates of the first workpiece in the space coordinate system when the first workpiece is located at a standard position on the stage. The workpiece of the same type as the first workpiece to be grasped can be placed in the standard position on the stage in advance to determine the second space coordinates. In the process of actually controlling the mechanical arm to grasp the first workpiece, it is not necessary to Each grab places the first workpiece in a standard position on the stage.

Since the second spatial coordinate indicates the position of the first workpiece in the standard position in the spatial coordinate system, and the first spatial coordinate indicates the actual position of the first workpiece in the spatial coordinate system, it can be based on the first spatial coordinate and the second Space coordinates, determine the offset of the actual position of the first workpiece in the space coordinate system relative to the standard position.

The movement process of the robotic arm is preset for the standard position. When the mechanical arm moves above the stage according to the movement process, it can grab the first workpiece placed in the standard position. In the process of actually grabbing the workpiece, the offset of the actual position of the first workpiece in the space coordinate system relative to the standard position is sent to the robotic arm as control information, and the robotic arm can move on the preset motion process according to the control information Perform corresponding translation or rotation to grab the first workpiece placed at any position on the stage.

In the embodiment of the present application, the first spatial coordinate indicates the actual position of the first workpiece in the spatial coordinate system, and the second spatial coordinate system indicates the position of the first workpiece in the spatial coordinate system when the first workpiece is at the standard position, According to the first space coordinates and the second space coordinates, the offset of the actual position of the first work relative to the standard position can be determined, and the offset can be sent to the robot arm as control information, so that the robot arm can The corresponding translation or rotation is carried out on the preset motion process, wherein the preset running process corresponds to the standard position, and then the workpiece located at the non-standard position on the stage is grasped, which is convenient for the motion control of the mechanical arm.

Fig. 2 is a flow chart of a second spatial coordinate acquisition method according to an embodiment of the present application. As shown in Figure 2, the second spatial coordinate acquisition method 200 includes the following steps:

Step 201. Obtain a second workpiece image previously captured by a camera.

When the camera collects the image of the second workpiece, the first workpiece is placed on the standard position on the stage, the mechanical arm moves above the first workpiece, and the camera moving synchronously with the mechanical arm collects the image of the first workpiece to obtain the second workpiece. Artifact image.

Step 202. Determine at least one second feature point from the second workpiece image.

After acquiring the image of the second workpiece to be captured captured by the camera, it is necessary to determine the position of the image of the first workpiece in the image of the second workpiece. Determine at least one second feature point from the second workpiece image, the second feature point refers to a point in the second workpiece image where the gray value changes sharply or a point with a large curvature on the edge of the second workpiece image. The second feature point may indicate a location of the image of the first workpiece in the image of the second workpiece.

Step 203, determining the second space coordinates of the first workpiece in the space coordinate system.

Each second feature point indicates the position of the image of the first workpiece at the standard position in the second workpiece image, and the calibration information is used to indicate the mapping relationship between the pixel coordinate system and the spatial coordinate system of the image collected by the camera, so it can be According to each second feature point and the calibration information, a second space coordinate for indicating the first workpiece at the standard position in the space coordinate system of the mechanical arm is determined.

In the embodiment of the present application, the first workpiece is placed in the standard position on the stage in advance, and the second workpiece image is collected by the camera, and one or more second feature points are determined from the second workpiece image, according to the second The feature points and calibration information determine the second space coordinates, ensuring that the determined second space coordinates can accurately indicate the position of the first workpiece in the space coordinate system of the manipulator when the first workpiece is at the standard position, thereby ensuring the determined control The accuracy of the information enables the robotic arm to accurately grasp the first workpiece according to the received control information.

In a possible implementation manner, the first feature point is a point on the contour line of the first workpiece in the first workpiece image, and the second feature point is a point on the contour line of the first workpiece in the second workpiece image.

In general, there is a large color difference between the workpiece and the stage, and the contour line of the workpiece can be accurately identified in the image of the workpiece collected by the camera, and then the points on the contour line of the workpiece can be determined as feature points. In this case, all possible first feature points are located on the outline of the first circuit board, and all possible second feature points are located on the outline of the second circuit board.

In the embodiment of the present application, since the first feature point is located on the contour line of the first workpiece, and the second feature point is located on the contour line of the first workpiece, the first workpiece can be accurately positioned according to each first feature point, The accuracy of the determined second space coordinates can be guaranteed according to the second feature points, thereby ensuring that the manipulator can smoothly grasp the workpiece.

In a possible implementation, the space coordinate system of the manipulator can be a three-dimensional Cartesian coordinate system, the plane where the X-axis and the Y-axis are located in the three-dimensional Cartesian coordinate system is parallel to the stage, and the first space coordinate and the second space The coordinates include an abscissa along the X-axis direction of the three-dimensional Cartesian coordinate system, a vertical coordinate along the Y-axis direction of the three-dimensional Cartesian coordinate system, and a rotation angle around the Z-axis direction of the three-dimensional Cartesian coordinate system. On this basis, FIG. 3 provides a method for determining an offset. As shown in FIG. 3 , the method 300 for determining an offset includes the following steps:

Step 301. Determine the X-axis displacement offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

Since the second space coordinate indicates the position of the standard position in the space coordinate system, the difference between the abscissa included in the first space coordinate and the abscissa included in the second space coordinate can be calculated, and the difference is determined as the first The X-axis displacement offset of the workpiece position relative to the standard position.

Step 302, determining the Y-axis displacement offset of the position of the first workpiece in the space coordinate system relative to the standard position.

Since the standard position indicated by the second spatial coordinate is in the spatial coordinate system, the difference between the vertical coordinate included in the first spatial coordinate and the vertical coordinate included in the second spatial coordinate can be calculated, and the difference is determined as the first The Y-axis offset of the position of a circuit board relative to the standard position.

Step 303, determining the Z-axis rotation offset of the position of the first workpiece in the spatial coordinate system relative to the standard position.

Since the standard position indicated by the second space coordinate is in the space coordinate system, the difference between the rotation angle included in the first space coordinate and the rotation angle included in the second space coordinate can be calculated, and the difference is determined as the first A Z-axis rotational offset of the position of the board relative to the standard position. Wherein, since the second space coordinate indicates the position of the standard position in the space coordinate system, and the standard position is used as a reference, the rotation angle included in the second space coordinate may be equal to zero.

In the embodiment of the present application, the X-axis displacement offset, the Y-axis displacement offset and the Z-axis rotation offset of the first workpiece relative to the standard position in the space coordinate system are calculated as the control information of the mechanical arm. The arm can translate or rotate relative to the standard position according to the control information, so that the robotic arm can grab the workpiece at any position on the stage.

FIG. 4 is a flowchart of a method for determining calibration information according to an embodiment of the present application. As shown in FIG. 4 , the method 400 for determining calibration information includes the following steps:

Step 401, determining a calibration reference point on a second workpiece.

The spatial coordinate system of the robotic arm is a three-dimensional Cartesian coordinate system, in which the X-axis, Y-axis and Z-axis are perpendicular to each other, and the plane where the X-axis and Y-axis are located is parallel to the stage. When the first workpiece and the second workpiece are respectively placed on the stage, the height of the first workpiece and the second workpiece in the Z-axis direction are the same. The second workpiece and the first workpiece may be the same type of workpiece, or may be different types of workpieces, for example, the second workpiece and the first workpiece are circuit boards of the same type.

When the camera is calibrated through the second workpiece, since the camera collects a planar image, the distance between the workpiece and the camera is different, which will affect the position of the workpiece image in the image collected by the camera, thereby affecting the calibration result, so it is necessary to ensure that the second The workpiece is at the same height as the first workpiece in the Z-axis direction.

In order to calibrate the camera through the image of the second workpiece, a calibration reference point is determined on the upper surface of the second workpiece, and the determined calibration reference point has a corresponding image in the image collected by the camera.

Step 402, controlling the mechanical arm to perform N translations along the X-axis and/or the Y-axis in the spatial coordinate system.

In the process of camera calibration, in order to determine the conversion relationship between the pixel coordinate system of the image captured by the camera and the space coordinate system of the manipulator, it is necessary to control the manipulator to drive and fix it on the The camera on the mechanical arm performs N translations along the X-axis and/or the Y-axis in the spatial coordinate system, so that the camera can collect images of the second workpiece placed on the stage at different positions. In order to solve the conversion relationship between the pixel coordinate system and the space coordinate system, at least three sets of pixel coordinates and space coordinates are required, so the number of translations N of the manipulator in the space coordinate system is a positive integer greater than or equal to 3. The camera collects an image of the second workpiece after each translation of the robotic arm as a translation calibration image.

It should be noted that when the mechanical arm translates along the X-axis and/or Y-axis in the space coordinate system, the position of the mechanical arm in the Z-axis direction remains unchanged, ensuring the distance between the camera and the stage in the Z-axis direction. The distance remains the same.

Step 403 , acquiring a translation calibration image captured by the camera after each translation of the robotic arm.

Acquire the translation calibration images collected by the camera. Since the camera will collect at least one translation calibration image after each translation of the robot arm, at least N translation calibration images can be obtained.

Step 404, controlling the mechanical arm to perform M rotations around the Z axis in the space coordinate system.

In order to improve the accuracy of camera calibration, so that the camera is not limited to moving along the X-axis and Y-axis, the mechanical arm can be controlled to drive the camera to rotate M times around the Z-axis in the space coordinate system according to the preset Z-axis rotation amount. . In order to determine the conversion relationship between pixel coordinates and space coordinates when the manipulator rotates, at least three sets of pixel coordinates and space coordinates of the reference point after the manipulator rotates are required, so M is a positive integer greater than or equal to 3.

Step 405 , acquiring the rotation calibration image captured by the camera after each rotation of the mechanical arm.

Obtain the rotation calibration images collected by the camera. Since the camera will collect at least one rotation calibration image after each rotation of the mechanical arm, at least M rotation calibration images can be obtained.

Step 406, determine the calibration information.

After obtaining the translation calibration image and rotation calibration image, according to the position offset of the calibration reference point image in different translation calibration images, the position offset of the calibration reference point image in different rotation calibration images, each translation The X-axis translation amount and the Y-axis translation amount of the mechanical arm when the calibration image is collected, and the Z-axis rotation amount of the mechanical arm when each rotating calibration image is collected determine the calibration information.

Since the second workpiece is on the stage and remains stationary, after the robot arm drives the camera to translate or rotate, the image of the reference point on the second workpiece will shift in the image captured by the camera, and the offset is the same as that of the robot arm. The translation amount of the X-axis and/or Y-axis and the rotation amount around the Z-axis are positively correlated, so the relationship between the pixel coordinate system and the space coordinate system used to indicate the image collected by the camera can be determined according to each translation calibration image and rotation calibration image The calibration information of the mapping relationship.

In the embodiment of the present application, the mechanical arm drives the camera to translate and rotate. After each translation and/or rotation, the camera captures the image of the second workpiece. According to the translation amount and/or rotation amount of the camera and the image of the calibration reference point The position offset in the collected image determines the mapping relationship between the pixel coordinate system of the image collected by the camera and the spatial coordinate system of the manipulator, so as to obtain the calibration information reflecting the mapping relationship. The camera is collected and placed on the stage After the image of the workpiece is obtained, the pixel coordinates of the workpiece in the image collected by the camera can be converted to the spatial coordinates in the space coordinate system through the calibration information, and then the robotic arm can grab it according to the spatial coordinates of the workpiece, realizing that the robotic arm can Grab workpieces randomly placed on the stage.

In addition, the camera is calibrated through multiple translation calibration images and multiple rotation calibration images to ensure that the generated calibration information can accurately reflect the mapping relationship between the pixel coordinate system of the image collected by the camera and the spatial coordinate system of the manipulator, and then Through the calibration information, the spatial coordinates of the workpiece in the spatial coordinate system can be accurately determined to ensure the success rate of the robotic arm grasping the workpiece.

In a possible implementation manner, when determining the calibration information, through the i-th translation among the N translations performed by the robotic arm, according to the image of the calibration reference point, the first translation captured by the camera before the i-th translation of the robotic arm The position of the i-1 translation calibration image is the position in the i-th translation calibration image collected by the camera after the i-th translation with the robotic arm, and the i-th pixel translation offset is calculated. Through the jth rotation of the M rotations performed by the robotic arm, according to the image of the calibration reference point, the position of the j-1th rotation calibration image collected by the camera before the jth rotation of the robotic arm, and the jth rotation with the robotic arm The position of the jth rotation calibration image collected by the camera after the second rotation, and the jth pixel rotation offset is calculated. According to the i-th pixel translation offset and the X-axis translation and Y-axis translation corresponding to the i-th translation of the robot arm, and the j-th pixel rotation offset and the Z-axis rotation corresponding to the j-th rotation of the robot arm , to determine the calibration information.

For example, the number of translations N is 8, and the number of rotations M is 4.

During the 8 translations performed by the manipulator, the image of the calibration reference point is calculated to translate the pixel translation offset in the calibration image before and after each translation. After the first translation, calculate the translation offset of the first pixel in the translation calibration image of the image of the calibration reference point before the first translation and after the first translation, and the calculation method of the translation offset of the subsequent i-th pixel The calculation method is the same as the above, and will not be repeated here.

During the 4 rotations of the manipulator, the image of the calibration reference point is calculated to rotate the pixel rotation offset in the calibration image before and after each rotation. After the first rotation, calculate the rotation offset of the first pixel in the image of the calibration reference point before the first rotation and after the first rotation, and calculate the rotation offset of the subsequent jth pixel The calculation method is the same as the above, and will not be repeated here.

According to the translation offset of the 1st to 8th pixels, and the X-axis translation and Y-axis translation corresponding to each translation, as well as the 1st to 4th pixel rotation offset, and the Z-axis rotation corresponding to each rotation, determine Calibration information.

In the embodiment of the present application, the pixel translation offset in the calibration image is translated before and after each translation by comparing the calibration reference point image, and the pixel rotation offset in the calibration image is rotated before and after each rotation of the calibration reference point image The offset of the calibration reference point matching the number of movements is obtained, and the accuracy of camera calibration is further improved by the movement offset in each space coordinate system and the corresponding offset of the calibration reference point.

Fig. 5 is a schematic diagram of the motion sequence of the mechanical arm according to an embodiment of the present application. As shown in Fig. 5, the number of translations N of the mechanical arm is 9, and the combination of translations along the X-axis direction and along the Y-axis direction when the mechanical arm performs 9 translations (x,0), (x,0), (0,y), (-x,0), (-x,0), (0,y), (x,0), (x,0 ) and (-x,-y), x and y are not equal to 0.

In the embodiment of the present application, by planning the translation path of the robotic arm, it is possible to prevent the robotic arm from exceeding the scope of the stage during the translation process, causing the camera to collect invalid pictures that do not contain workpieces, and improving the efficiency of camera calibration.

In a possible implementation manner, the preset X-axis translation amount is equal to the Y-axis translation amount, that is, x is equal to y.

FIG. 6 is a schematic diagram of the movement point coordinates of the robotic arm according to an embodiment of the present application. As shown in FIG. 6 , the X-axis translation amount and the Y-axis translation amount are equal to 5. When the mechanical arm performs 9 translations, the combinations of translations along the X-axis direction and along the Y-axis direction are (5,0), (5,0), (0,5), (-5,0), (-5 ,0), (0,5), (5,0), (5,0) and (-5,-5).

In the embodiment of the present application, the calculation complexity is reduced and the efficiency of camera calibration is improved by setting the X-axis translation amount and the Y-axis translation amount as equal offsets.

In a possible implementation manner, during the movement of the mechanical arm, translation and rotation operations are simultaneously performed during at least one movement.

In the embodiment of the present application, by synchronizing translation and rotation, the amount of images collected by the camera can be reduced, the amount of calculation can be reduced while the accuracy of calculation can be ensured, and the efficiency of camera calibration can be improved.

object grabber

Fig. 7 is a schematic diagram of an object grabbing device according to an embodiment of the present application. As shown in FIG. 7 , the object grabbing apparatus 700 includes an acquisition module 701 , an extraction module 702 , a determination module 703 and a grabbing module 704 .

After the acquiring module 701 acquires the first workpiece image captured by the camera, the extracting module 702 determines at least one first feature point according to the first workpiece image captured by the camera.

The determination module 703 determines the first space coordinates of the first workpiece in the space coordinate system of the robot arm according to the first feature points determined by the extraction module 702 .

The capturing module 704 generates control information according to the first spatial coordinate determined by the determining module 703 .

In the embodiment of the present application, at least one first feature point determined by the extraction module 702 can indicate the position of the first workpiece in the first workpiece image. The determining module 703 can determine the position of the first workpiece in the spatial coordinate system of the manipulator through the calibration information and the pixel coordinates of the first feature point. The grasping module 704 controls the mechanical arm to grasp the workpiece through the position information of the workpiece. Without placing the workpiece at a specific angle and position, the robotic arm can automatically grab the workpiece at any position on the stage, saving labor costs.

Electronic equipment

Fig. 8 is a schematic diagram of an electronic device according to an embodiment of the present application, and the specific embodiment of the present application does not limit the specific implementation of the electronic device. As shown in FIG. 8 , the electronic device 800 may include: a processor (processor) 801 , a communication interface (Communications Interface) 802 , a memory (memory) 803 , and a communication bus 804 . in:

The processor 801 , the communication interface 802 , and the memory 803 communicate with each other through the communication bus 804 .

The communication interface 802 is used for communicating with other electronic devices or servers.

The processor 801 is configured to execute the program 805, specifically, may execute relevant steps in any method embodiment in the aforementioned multiple method embodiments.

Specifically, the program 805 may include program codes including computer operation instructions.

The processor 801 may be a CPU, or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be of the same type, such as one or more CPUs, or may be different types of processors, such as one or more CPUs and one or more ASICs.

The memory 803 is used to store the program 805 . The memory 803 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

The program 805 may be specifically configured to enable the processor 801 to execute any method in the multiple method embodiments in the foregoing embodiments.

For the specific implementation of each step in the program 805, refer to the corresponding descriptions in the corresponding steps and units in the aforementioned embodiment of the workpiece grasping method, and details are not repeated here. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described devices and modules can refer to the corresponding process description in the foregoing method embodiments, and details are not repeated here.

Through the electronic device of the embodiment of the present application, at least one first feature point determined from the image of the first workpiece can indicate the position of the image of the first workpiece in the image of the first workpiece, and the calibration information can indicate the pixels of the image collected by the camera The mapping relationship between the coordinate system and the space coordinate system of the manipulator, through the calibration information and the pixel coordinates of the first feature point, the position of the first workpiece in the space coordinate system of the manipulator can be determined, and then the position of the first workpiece in the space coordinate system of the manipulator can be The position in the system generates control information, and after sending the control information to the robotic arm, the robotic arm grabs the first workpiece according to the control information. It can be seen that since the position of the workpiece in the spatial coordinate system of the manipulator can be determined, the manipulator can automatically grab the workpiece randomly placed on the stage without placing the workpiece on the load at a specific angle and position. on the stage, thereby saving the time required to place the workpiece on the stage, thereby improving the efficiency of grasping the workpiece.

computer storage media

The present application also provides a computer-readable storage medium storing instructions for causing a machine to execute any method in the multiple method embodiments described herein. Specifically, a system or device equipped with a storage medium may be provided, on which a software program code for realizing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU of the system or device) ) to read and execute the program code stored in the storage medium.

In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, so the program code and the storage medium storing the program code constitute a part of the present application.

Examples of storage media for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), Tape, non-volatile memory card, and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

computer program product

An embodiment of the present application further provides a computer program product, including computer instructions, where the computer instruction instructs a computing device to perform any corresponding operation in the foregoing multiple method embodiments.

It should be pointed out that, according to the needs of implementation, each component/step described in the embodiment of the present application can be divided into more components/steps, and two or more components/steps or partial operations of components/steps can also be combined into New components/steps to achieve the purpose of the embodiment of the present application.

The above method according to the embodiment of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or implemented by Computer code downloaded from a network that is originally stored on a remote recording medium or a non-transitory machine-readable medium and will be stored on a local recording medium so that the methods described herein can be stored on a computer code using a general-purpose computer, a dedicated processor, or a programmable Such software processing on a recording medium of dedicated hardware such as ASIC or FPGA. It will be appreciated that a computer, processor, microprocessor controller, or programmable hardware includes memory components (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code that, when When accessed and executed by a processor or hardware, implements the methods described herein. Furthermore, when a general purpose computer accesses code for implementing the methods shown herein, execution of the code transforms the general purpose computer into a special purpose computer for performing the methods shown herein.

It should be noted that not all the steps and modules in the above processes and system structure diagrams are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as required. The system structures described in the above embodiments may be physical structures or logical structures, that is, some modules may be realized by the same physical entity, or some modules may be realized by multiple physical entities, or may be realized by multiple Certain components in individual devices are implemented together.

In the above embodiments, the hardware modules may be implemented mechanically or electrically. For example, a hardware module may include permanently dedicated circuitry or logic (such as a dedicated processor, FPGA or ASIC) to perform the corresponding operations. The hardware modules may also include programmable logic or circuits (such as general-purpose processors or other programmable processors), which can be temporarily set by software to complete corresponding operations. The specific implementation (mechanical way, or a dedicated permanent circuit, or a temporary circuit) can be determined based on cost and time considerations.

The present invention has been shown and described in detail through the accompanying drawings and preferred embodiments above, but the present invention is not limited to these disclosed embodiments, and those skilled in the art based on the above-mentioned multiple embodiments can know that the above-mentioned different embodiments can be combined More embodiments of the present invention can be obtained by means of code review in the present invention, and these embodiments are also within the protection scope of the present invention.

Claims (14)

1. A workpiece grabbing method (100), comprising: acquiring a first workpiece image captured by the camera, wherein the first workpiece image includes an image of the first workpiece located on the stage; determining at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate a position of the image of the first workpiece in the first workpiece image; According to the pixel coordinates and calibration information of the at least one first feature point, determine the first spatial coordinates of the first workpiece in the spatial coordinate system of the manipulator, wherein the calibration information is used to indicate the captured by the camera The mapping relationship between the pixel coordinate system of the image and the space coordinate system; generating control information according to the first space coordinates, and sending the control information to the robotic arm, so that the robotic arm can grasp the first workpiece according to the control information. 2. The method according to claim 1, wherein said generating control information according to said first spatial coordinates comprises: Acquiring second space coordinates, wherein the second space coordinates are used to indicate the coordinates of the first workpiece in the space coordinate system when the first workpiece is located at a standard position on the stage; determining the offset of the position of the first workpiece in the space coordinate system relative to the standard position according to the first space coordinate and the second space coordinate; The offset is determined as control information. 3. The method of claim 2, further comprising: acquiring a second workpiece image previously captured by the camera, wherein the second workpiece image includes an image of the first workpiece at a standard position on the stage; determining at least one second feature point from the second workpiece image, wherein the second feature point is used to indicate a position of the image of the first workpiece in the second workpiece image; A second space coordinate of the first workpiece in the space coordinate system is determined according to the pixel coordinates of the at least one second feature point and the calibration information. 4. The method according to claim 3, wherein the first feature point is a point on the contour line of the first workpiece in the first workpiece image, and the second feature point is the second feature point A point on the contour line of the first workpiece in the workpiece image. 5. The method according to claim 2, wherein the spatial coordinate system is a three-dimensional Cartesian coordinate system, and the plane where the X-axis and the Y-axis are located in the three-dimensional Cartesian coordinate system is parallel to the stage, and the first Both the spatial coordinates and the second spatial coordinates include an abscissa along the X-axis direction of the three-dimensional Cartesian coordinate system, an ordinate along the Y-axis direction of the three-dimensional Cartesian coordinate system, and an abscissa around the three-dimensional Cartesian coordinate system The rotation angle of the Z-axis direction of the system; The determining the offset of the position of the first workpiece in the space coordinate system relative to the standard position according to the first space coordinate and the second space coordinate includes: Determining the difference between the abscissa included in the first space coordinate and the abscissa included in the second space coordinate as X of the position of the first workpiece in the space coordinate system relative to the standard position Axis displacement offset; Determining the difference between the ordinate included in the first space coordinate and the ordinate included in the second space coordinate as the Y of the position of the first workpiece in the space coordinate system relative to the standard position Axis displacement offset; Determining the difference between the rotation angle included in the first space coordinate and the rotation angle included in the second space coordinate as Z of the position of the first workpiece in the space coordinate system relative to the standard position Axis rotation offset. 6. The method according to any one of claims 1-5, further comprising: Determining a calibration reference point located on the second workpiece, wherein the second workpiece is at the same height as the first workpiece in the Z-axis direction of the spatial coordinate system, where the X-axis and the Y-axis are located in the spatial coordinate system The plane is parallel to the stage; According to the preset X-axis translation amount and Y-axis translation amount, control the manipulator to perform N times of translation along the X-axis and/or Y-axis direction in the space coordinate system, and obtain the data captured by the camera after each translation of the manipulator arm A translation calibration image, wherein the translation calibration image includes an image of the second workpiece located on the stage, and N is a positive integer greater than or equal to 3; According to the preset Z-axis rotation amount, control the mechanical arm to perform M rotations around the Z-axis in the space coordinate system, and obtain the rotation calibration image collected by the camera after each rotation of the mechanical arm, wherein the rotation calibration image Including the image of the second workpiece located on the stage, M is a positive integer greater than or equal to 3; According to the position offset of the image of the calibration reference point in different translation calibration images, the position offset of the image of the calibration reference point in different rotation calibration images, each of the translation calibration images The corresponding X-axis translation amount and Y-axis translation amount, and the Z-axis rotation amount corresponding to each of the rotated calibration images determine the calibration information. 7. The method according to claim 6, wherein, according to the position offset of the image of the calibration reference point in different translation calibration images, the image of the calibration reference point in different rotation calibration images The position offset in the image, the X-axis translation and the Y-axis translation corresponding to each of the translation calibration images, and the Z-axis rotation corresponding to each of the rotation calibration images determine the calibration information, including: For the i-th translation among the N translations performed by the mechanical arm, calculate the translation offset of the i-th pixel in the i-1th translation calibration image and the i-th translation calibration image of the image of the calibration reference point, wherein, The i-1th translation calibration image is the translation calibration image collected by the camera before the i-th translation of the robotic arm, and the i-th translation calibration image is the translation calibration image collected by the camera after the i-th translation of the robotic arm Image, i is a positive integer less than N; For the j-th rotation among the M rotations performed by the mechanical arm, calculate the j-th pixel rotation offset between the j-1th rotation calibration image and the j-th rotation calibration image of the image of the calibration reference point, wherein, The j-1th rotation calibration image is the rotation calibration image collected by the camera before the j-th rotation of the robotic arm, and the j-th rotation calibration image is the rotation calibration image collected by the camera after the j-th rotation of the robotic arm image, j is a positive integer less than M; According to the i-th pixel translation offset and the X-axis translation and Y-axis translation corresponding to the i-th translation of the robot arm, and the j-th pixel rotation offset corresponding to the j-th rotation of the robot arm Z-axis rotation amount to determine the calibration information. 8. The method according to claim 7, wherein N is equal to 9 and M is equal to 3; When the robot arm performs N times of translation, the combination of translation along the X-axis direction and along the Y-axis direction is (x, 0), (x, 0), (0, y), (-x, 0), (-x ,0), (0,y), (x,0), (x,0) and (-x,-y), where x is used to represent the position of the manipulator along the X-axis in the space coordinate system The translation amount, y is used to represent the translation amount of the manipulator along the Y-axis in the space coordinate system, and both x and y are not equal to 0. 9. The method of claim 8, wherein x is equal to y. 10. The method according to any one of claims 7-9, wherein the robot arm is rotated around the Z axis while translating along the X axis and/or the Y axis direction at least once. 11. An object grasping device (700), comprising: An acquisition module (701), configured to acquire a first workpiece image captured by a camera, wherein the first workpiece image includes an image of a first workpiece located on a stage; An extraction module (702), configured to determine at least one first feature point from the first workpiece image, wherein the first feature point is used to indicate that the image of the first workpiece is in the first workpiece image s position; A determination module (703), configured to determine the first spatial coordinates of the first workpiece in the spatial coordinate system of the manipulator according to the pixel coordinates and calibration information of the at least one first feature point, wherein the calibration information used to indicate the mapping relationship between the pixel coordinate system of the image collected by the camera and the space coordinate system; A grasping module (704), configured to generate control information according to the first space coordinates, and send the control information to the robotic arm, so that the robotic arm can grasp the first workpiece according to the control information to fetch. 12. An electronic device (800), comprising: a processor (801), a communication interface (802), a memory (803) and a communication bus (804), a processor (801), a memory (803) and a communication interface (802 ) complete mutual communication through the communication bus (804); The memory (803) is used to store at least one executable instruction, and the executable instruction causes the processor (801) to execute the operation corresponding to the workpiece grasping method according to any one of claims 1-10. 13. A computer storage medium, on which a computer program is stored, and when the program is executed by a processor, the workpiece grasping method according to any one of claims 1-10 is realized. 14. A computer program product, comprising computer instructions, said computer instructions instructing a computing device to perform operations corresponding to the workpiece grasping method according to any one of claims 1-10.

Images