TW202206243A - Mechanical arm system for correcting grip coordinates in real time including a conveying platform, a mechanical arm and a depth camera - Google Patents

Abstract

A mechanical arm system for correcting grip coordinates in real time includes a conveying platform, a mechanical arm and a depth-of-field camera. An image frame of an object is shot on initial coordinates by using the depth-of-field camera. A calculation unit analyzes the image frame to obtain the position, center coordinates and dimensional proportion of the object, and then calculates according to the center coordinates of the object to know grip coordinates and a displacement path between the initial coordinates and the grip coordinates. The calculation unit generates correction coordinates on the displacement path. When a grip paw moves to the correction coordinates, an image frame of the object is further shot and whether the grip coordinates are consistent is analyzed and judged. Therefore, the grip paw is enabled to move to the grip coordinates by precise positioning, and the grip paw can be finely turned and corrected in real time by utilizing an unobstructed image frame on the grip paw and a depth-of-field range finding function of the depth-of-field camera.

Description

Robotic arm system for real-time correction of gripping coordinates

The present invention relates to a robotic arm control system, in particular to a robotic arm system that uses a depth camera to achieve real-time correction of clamping coordinates for precise control of the position of a clamped object.

Press, in order to reduce the manufacturing cost, conventional factories usually develop towards the direction of automated manufacturing, and the automated manufacturing method is mainly realized through the robotic arm. When the existing robotic arm is in operation, the appropriate gripper is usually selected according to the actual process. The mechanical arm can only perform repeated actions on a single type of object, and in order to reduce the error rate of clamping, the position and angle of the object must be strictly limited. When the size of the object is inconsistent or untouched When the rows are neatly arranged, the robotic arm that generally operates the fixed clamping procedure is not suitable. Therefore, the conventional art provides a robotic arm combined with image analysis. The fixed photographing device is arranged in the working area of the robotic arm. The device captures the image of the object, and controls the robotic arm to make corresponding adjustments through the image features. However, it is not difficult to find that there are still some deficiencies in the above-mentioned conventional structure. The main reasons are as follows: the photographing device is in a fixed position and cannot be Verify the image features of the object multiple times through different distances or angles. When the captured image error prevents the object from being held correctly, the robotic arm must return to the original position. position, so that the robotic arm will not block the photographing of the object by the photographing device. The operation process that cannot be corrected in real time has the disadvantages of time-consuming and low clamping accuracy, and in some events that are difficult to clamp, the single image judgment After the object clamping method fails, no matter how many times the object is clamped again, the result will still be unable to successfully clamp the object. In conclusion, the above are all technical problems that this creation intends to improve.

In view of this, the inventor of the present invention has been engaged in the manufacture, development and design of related products for many years, aiming at the above-mentioned goals, after detailed design and careful evaluation, finally came up with a practical invention.

The technical problem to be solved by the present invention is to provide a robotic arm system for real-time correction of clamping coordinates in view of the above-mentioned deficiencies in the prior art.

A conveying platform is used to continuously convey objects to a designated position, a robotic arm is arranged on one side of the conveying platform, and the end of the robotic arm is provided with a gripper that can move in three-dimensional coordinates, and the robotic arm has built-in real-time judgment of the gripper A control unit for claw coordinates, and the fixed position of the gripper above the conveying platform is defined as the initial coordinate, a depth of field camera is fixed on the gripper of the robotic arm, and the front of the lens of the depth of field camera is aligned with the The direction of the jaws, and the depth of field camera is connected with the control unit to have an arithmetic unit, the depth of field camera shoots an image of the object at the starting coordinate, and analyzes the image through the arithmetic unit to know the position of the object, and based on the starting coordinate and The depth of field distance can be calculated to obtain a center coordinate and size ratio of the object.

The depth-of-field camera includes a main lens and a secondary lens, and the secondary lens is used for sensing distance, so that the image captured by the main lens has a depth-of-field effect.

The computing unit is connected to a cloud server, and the cloud server stores images used to identify objects and related data of the corresponding objects, so that the computing unit can deeply learn the images of the cloud server, thereby improving the accuracy of the objects. accuracy of judgment.

When the depth-of-field camera executes the shooting command, the calculation unit can record the image frames of all objects, and create an image project folder in the cloud server, so that the subsequent image frames captured by the depth-of-field camera can be directly compared with the image project data. The image screen in the folder can improve the operation speed and judgment accuracy of the calculation unit.

When the gripper moves along the displacement path, the calculation unit can record the entire displacement path and whether it reaches the specified calibration coordinates or clamping coordinates, and creates a path project folder in the cloud server, and judges the same through data analysis. When the jaws cannot reach the specified position with similar displacement paths, the calculation unit will eliminate the bad displacement paths and improve the positioning accuracy by detouring.

The calculation unit uses the known starting coordinates and the depth of field of the image frame to perform calculation, so that the image frame can draw a plane bounding box surrounding the object, and the plane bounding box is defined at one end corner as a bounding coordinate, and The calculation unit obtains the center coordinates and the ruler of the object by calculating the plane bounding box inch ratio.

The aspect ratio of the plane bounding box is set as (w, h), the boundary coordinates are set as (Xi, Yi), and the calculation unit calculates the center coordinate as ((Xi-w/2 ), (Yi-h/2)), and the starting coordinates are set to (X ₀ , Y ₀ ), and the size ratio of the object is set to Z, that is, the movement vector of the X point of the corresponding object can be obtained as -((Xi -w/2)-X ₀ )/Z, and the Y point movement vector of the corresponding object is -((Yi-h/2)-Y ₀ )/Z.

The clamping jaw is displaced to the clamping coordinate, so that the rotation angle of the clamping jaw aligns with the center coordinate of the object. When the clamping jaw is insufficient to hold the object or exceeds the allowable value of the specified width, the calculation unit judges that the clamping fails, and the control The unit displaces the gripper to the corrected coordinates of the previous step, and the depth camera takes pictures again, and the calculation unit recalculates the new gripping coordinates, and performs the gripping action again.

The robotic arm is provided with a load unit at the gripper. After the gripper is operated to perform the gripping action, the load unit does not detect an increase in the weight of the gripper, and the calculation unit can determine the gripper. If it fails, the control unit displaces the gripper to the calibration coordinates of the previous step, and the depth camera takes pictures again, and the calculation unit recalculates new gripping coordinates to perform the gripping action again.

When the gripper does not successfully grip the object under the self-set number of times, the control unit operates the gripper to rotate 90 degrees along the vertical axis of the conveying platform, thereby gripping different positions of the object, or tilting the gripper Claws make this depth of field The camera additionally captures a side image of the object, thereby analyzing the position of the center of gravity with respect to the height of the object and providing new clamping coordinates.

The main purpose of the present invention is that the depth-of-field camera shoots an image of the object at the starting coordinate, and the calculation unit analyzes the image to know the position, center coordinates and size ratio of the object, and then calculates the center coordinate of the object to know the clamping Coordinates, and the displacement path between the starting coordinate and the clamping coordinate. The calculation unit generates a calibration coordinate on the displacement path. When the clamping jaw is displaced to the calibration coordinate, the image screen of the object is captured again for analysis and judgment. Whether the gripping coordinates are consistent, so that the gripper can be accurately positioned and moved to the gripping coordinates, that is, the unobstructed image picture and the depth-of-field ranging function of the depth-of-field camera at the gripper can be used to fine-tune the gripper in real time. Correction.

Other objects, advantages, and novel features of the present invention will become more apparent from the following detailed description and the associated drawings.

〔this invention〕

10: Conveying platform

20: Robotic Arm

21: Gripper

22: Control unit

23: Load Cell

30: Depth of Field Camera

31: Calculation Unit

32: Main Shot

33: Secondary lens

34: Flat Bounding Box

341: Boundary coordinates

35: Cloud server

A: center coordinates

[Fig. 1] is a perspective view of the present invention.

[Fig. 2] is a schematic diagram of the lens of the depth-of-field camera of the present invention.

[Fig. 3] is a schematic diagram of the image analysis of the present invention.

[FIG. 4] is a block diagram of the element relationship of the present invention.

[FIG. 5] is a block diagram showing the operation steps of the present invention.

In order to make your examiners understand the purpose, features and effects of the present invention If you can have a further understanding and understanding, please follow the [schematic brief description] as follows:

First, please look at Figure 1, Figure 2 and Figure 4, a robotic arm system for real-time correction of clamping coordinates, including: a conveying platform 10, a robotic arm 20 and a depth of field camera 30, a conveying platform 10 is used to continuously transport objects to a designated position, a robotic arm 20 is arranged on one side of the conveying platform 10, and the end of the robotic arm 20 is provided with a gripper 21 that can move in three-dimensional coordinates, and the robotic arm 20 has built-in real-time A control unit 22 for judging the coordinates of the gripper 21, and the fixed position of the gripper 21 above the conveying platform 10 is defined as the initial coordinate, a depth of field camera 30 is fixed on the gripper 21 of the robotic arm 20, and The front of the lens of the depth-of-field camera 30 is aligned with the direction of the clamping jaw 21, and the depth-of-field camera 30 is connected with the control unit 22 to have a calculation unit 31, and the depth-of-field camera 30 includes a main lens 32 and a pair of lenses 33. The secondary lens 33 is used for sensing distance, so that the image captured by the primary lens 32 has a depth of field effect.

For its actual use, please refer to Figures 4 and 5 in conjunction with Figure 1. When the object is conveyed by the conveying platform 10 to the corresponding position of the robotic arm 20, the depth of field camera 30 is aligned at the starting coordinates. The object shooting image screen, if no object is captured or the captured object is incomplete, the shooting will be repeated after a short interval. If there is an object captured, the calculation unit 31 will analyze the image screen to know the position of the object , and based on the starting coordinates and the depth of field distance, the center coordinate A and the size ratio of the object can be calculated and obtained, and then the clamping coordinates can be calculated according to the center coordinate A of the object. The clamping coordinates are the appropriate place above the object, so that The gripper 21 can clamp the object at the center of gravity, and further plan the displacement path between the starting coordinate and the gripping coordinate, and basically let the gripper 21 move along the displacement path with a vector. Furthermore, the calculation unit 31 There is at least one calibration coordinate automatically generated on the displacement path. The calibration coordinate is any point on the displacement path. The calibration coordinates must be divided into multiple segments according to the path length or path time, so that the gripper 21 can continuously and autonomously correct the coordinates during the movement process. When the control unit 22 moves the gripper 21 to the calibration coordinate, the control unit 22 can first determine the coordinate of the gripper 21. When the error between the calibration coordinate and the actual position coordinate is too large, the control unit 22 Move the gripper 21 back to the starting coordinates, and start again. If there is only a small error, the control unit 22 can directly move the gripper 21 to the correct calibration coordinate, so that the depth-of-field camera 30 can take images of the object again screen, through the calculation unit 31 to analyze the image screen and compare whether it is consistent with the previously determined clamping coordinates, if it is consistent, move the clamping jaw 21 to the next calibration coordinate or clamping coordinate, so that the clamping jaw 21 is precisely positioned and moved to the clamping coordinates, and then the clamping jaw 21 is clamped to a specified width based on the size ratio of the object, that is, the unobstructed image screen and the depth-of-field ranging function of the depth-of-field camera 30 at the clamping jaw 21 can be used, The clamping jaws 21 are fine-tuned and corrected in real time, so as to achieve the effect of stably clamping the object.

To further explain the calculation method, please see from Figures 3 and 4 that the calculation unit 31 uses the known starting coordinates and the depth of field of the image frame to perform calculation, so that the image frame can draw a frame surrounding the object. A plane bounding box 34, and the plane bounding box 34 is defined as a boundary coordinate 341 at one end corner, and the calculation unit 31 obtains the center coordinate A and the size ratio of the object by calculating the plane bounding box 34, and the plane bounding box 34 The aspect ratio of A is set as (w, h), the boundary coordinate 341 is set as (Xi, Yi), and the calculation unit 31 calculates with the plane bounding box 34 to know that the center coordinate A is ((Xi-w/2) ,(Yi-h/2)), and the starting coordinate is set to (X ₀ ,Y ₀ ), and the size ratio of the object is set to Z, that is, the movement vector of the X point of the corresponding object can be obtained as -((Xi- w/2)-X ₀ )/Z, and the movement vector of the Y point of the corresponding object is -((Yi-h/2)-Y ₀ )/Z, where the movement vector of the X point and the Y point is represented by a negative number, It is mainly the corresponding relationship between the displacement direction and the vector. It is not limited to positive or negative numbers. In addition, the movement vectors of the X point and the Y point can also be multiplied and divided by the unit conversion value, so that the value of the movement vector can be amplified and directly read by the control unit 22. With the application, the coordinate points of the image frame are converted into actual coordinate points through the above calculation process, and the actual movement vector is further calculated and sent to the control unit 22 of the robotic arm 20 to achieve the purpose of driving the robotic arm 20 .

The clamping action mechanism of the clamping jaw 21 will be further described. As shown in Figures 4 and 5, after the clamping jaw 21 is displaced to the clamping coordinate, the rotation angle of the clamping jaw 21 is aligned with the center coordinate A of the object. And the control unit 22 drives the gripper 21 to perform the gripping action. When the gripper 21 grips the object insufficiently or exceeds the allowable value of the specified width, the arithmetic unit 31 judges that the gripping fails, and the control unit 22 determines the object to be gripped. The clamping jaws 21 are displaced to the corrected coordinates in the previous step, and are captured by the depth-of-field camera 30 again, and the calculation unit 31 recalculates new clamping coordinates to perform the clamping action again. In another embodiment, the robotic arm 20 is provided with a load unit 23 at the gripper 21 . After the gripper 21 is operated to perform the gripping action, the The load unit 23 does not detect the increase in weight of the gripper 21, and the calculation unit 31 can determine that the gripping fails, and the control unit 22 moves the gripper 21 to the calibration coordinates of the previous step, and again by the depth of field The camera 30 shoots, and the calculation unit 31 recalculates new clamping coordinates, and performs the clamping operation again. If the above two methods fail to grip the object, or the gripper 21 fails to grip the object under self-set times, the control unit 22 operates the gripper 21 to move along the conveying platform 10 . Rotate the vertical axis by 90 degrees, thereby gripping different positions of the object, or tilt the gripper 21 to make the depth-of-field camera 30 additionally take a side image of the object, thereby analyzing the position of the center of gravity according to the height of the object and providing new gripping coordinates , and after the object is successfully clamped, the control unit 22 controls the gripper 21 to return to the initial coordinate position, and waits for the next object to enter the movable range of the robotic arm 20 .

In another embodiment of the present creation, please refer to Figures 4 and 5 again, the computing unit 31 is connected to a cloud server 35, and the cloud server 35 stores the image screen and corresponding objects for identifying objects. The relevant data are used to allow the computing unit 31 to deeply learn the images of the cloud server 35, thereby improving the accuracy of object judgment. On the other hand, when the depth-of-field camera 30 executes the shooting command, the calculation unit 31 can record the image frames of all objects, and create an image project folder in the cloud server 35, so that the image frames captured by the depth-of-field camera 30 in the subsequent It can directly compare the image frames in the image project folder, and through a large number of self-deep learning processes, thereby improving the operation speed and judgment accuracy of the calculation unit 31 . Furthermore, the jaws 21 are displaced along the When the path performs the displacement action, the calculation unit 31 can record all the displacement paths and whether they reach the specified calibration coordinates or clamping coordinates, and create a path project folder in the cloud server 35, and determine the same or similar displacement paths through data analysis When the gripper 21 cannot reach the designated position, it is likely to be a long-term mechanism defect of the robotic arm 20, which causes the gripper 21 to deviate from the displacement path every time it moves to the fixed coordinate position. The bad displacement path will be eliminated. When the newly generated displacement path overlaps the excluded bad displacement path, a new path will be established on one side of the path, that is, the positioning accuracy will be improved by detouring, so as to improve the robot arm 20 displacement efficiency.

To sum up, the present invention has indeed achieved a breakthrough structural design, and has an improved content of the invention, and at the same time can achieve industrial applicability and progress, and the present invention has not been found in any publications, and it is also novel. In accordance with the provisions of the relevant laws and regulations of the Patent Law, the patent application for invention is filed in accordance with the law, and I urge the examination committee of the Jun Bureau to grant the legal patent right.

Only the above is only a preferred embodiment of the present invention, and should not be used to limit the scope of implementation of the present invention; that is, all equivalent changes and modifications made according to the scope of the patent application of the present invention should still belong to the present invention. covered by the patent.

10: Conveying platform

20: Robotic Arm

21: Gripper

30: Depth of Field Camera

Claims (10)

A robotic arm system for real-time correction of clamping coordinates, comprising: a conveying platform, which is used to continuously convey objects to a designated position; A robotic arm, which is arranged on one side of the conveying platform, and a gripper claw that can move with three-dimensional coordinates is arranged at the end of the robotic arm, and a control unit is built in the robotic arm to instantly determine the coordinates of the gripper claw, and the gripper The fixed position of the claw above the conveying platform is defined as the starting coordinate; a depth-of-field camera, which is fixed on the gripper of the robotic arm, the front of the lens of the depth-of-field camera is aligned with the gripper direction, and the depth-of-field camera is connected with the control unit to have an arithmetic unit; The depth-of-field camera shoots an image of the object at the starting coordinate, and the calculation unit analyzes the image to obtain the position of the object, and based on the starting coordinate and the depth-of-field distance, a center coordinate and a size ratio of the object can be obtained by calculation, and then according to The center coordinate of the object is calculated to obtain the clamping coordinate, and the displacement path between the starting coordinate and the clamping coordinate. The calculation unit automatically generates at least one correction coordinate on the displacement path. When the control unit moves the clamping jaw to the displacement When calibrating the coordinates, the depth-of-field camera shoots an image of the object again, analyzes the image through the calculation unit and compares it with the previously determined gripping coordinates, thereby allowing the gripper to move to the gripping coordinates accurately. , and then clamp the gripper to a specified width based on the size ratio of the object, that is, the unobstructed image of the depth-of-field camera at the gripper and the depth-of-field ranging function can be used to fine-tune the gripper in real time. The robotic arm system for real-time correction of clamping coordinates according to claim 1, wherein the depth-of-field camera includes a main lens and a secondary lens, and the secondary lens is used for sensing distance, so that the image captured by the main lens has a depth-of-field effect. For the robotic arm system for real-time calibration of gripping coordinates of claim 1, wherein the computing unit is connected to a cloud server, and the cloud server stores the image screen for identifying objects and the relevant data of the corresponding objects, so that the computing unit can Deep learning of the image screen of the cloud server to improve the accuracy of object judgment. According to the robotic arm system for real-time correction of clamping coordinates of claim 3, when the depth-of-field camera executes the shooting command, the computing unit can record the image frames of all objects, and create an image project folder in the cloud server, so that the Subsequent images captured by the depth-of-field camera can be directly compared with the images in the image project folder, thereby improving the calculation speed and the accuracy of the calculation unit. According to the robotic arm system for real-time correction of gripping coordinates in claim 3, when the gripper jaws perform the displacement action along the displacement path, the calculation unit can record the entire displacement path and whether it reaches the specified correction coordinates or gripping coordinates, and then record it in the The cloud server creates a path project folder, and when it is judged through data analysis that the same or similar displacement paths cannot make the gripper reach the specified position, the calculation unit will subsequently eliminate the bad displacement paths and improve the positioning accuracy by detouring. According to the robotic arm system for real-time correction of clamping coordinates according to claim 1, wherein the calculation unit uses the known starting coordinates and the depth of field of the image frame to perform calculation, so that the image frame can draw a plane bounding box surrounding the object, A corner of one end of the plane bounding box is defined as a bounding coordinate, and the calculating unit obtains the center coordinate and size ratio of the object by calculating the plane bounding box. The manipulator system for real-time correction of gripping coordinates of claim 6, wherein the aspect ratio of the plane bounding box is set to (w, h), the boundary coordinates are set to (Xi, Yi), and the calculation unit uses the plane boundary The frame operation knows that the center coordinate is ((Xi-w/2), (Yi-h/2)), and the starting coordinate is set to (X ₀ , Y ₀ ), and the size ratio of the object is set to Z, That is, the X point movement vector of the corresponding object can be obtained as -((Xi-w/2)-X ₀ )/Z, and the Y point movement vector of the corresponding object is -((Yi-h/2)-Y ₀ )/ Z. For the robotic arm system for real-time correction of gripping coordinates of claim 1, wherein the gripper jaws are displaced to the gripping coordinates, so that the rotation angle of the gripper jaws is aligned with the center coordinates of the object, and the gripper jaws grip the object by less than or exceeding the specified width. When the allowable value is reached, the calculation unit judges that the clamping fails, and the control unit moves the gripper to the calibration coordinates of the previous step, and the depth of field camera takes pictures again, and the calculation unit recalculates the new clamping coordinates and re- Carry out the gripping action. The robotic arm system for real-time correction of gripping coordinates as claimed in claim 1, wherein the robotic arm is provided with a load unit at the gripper jaw, which is used to operate the gripper. After the clamping jaws perform the clamping action, the load unit does not detect the increase in the weight of the clamping jaws, the calculation unit can judge the clamping failure, and the control unit moves the clamping jaws to the calibration coordinates of the previous step, Shooting is again performed by the depth-of-field camera, and new clamping coordinates are recalculated by the computing unit, and the clamping action is performed again. According to the robotic arm system for real-time correction of gripping coordinates in claim 1, when the gripper does not successfully grip the object under self-set times, the control unit operates the gripper to rotate 90 along the vertical axis of the conveying platform degrees, thereby gripping different positions of the object, or tilting the gripper to make the depth-of-field camera additionally take a side image of the object, thereby analyzing the position of the center of gravity with respect to the height of the object to provide new gripping coordinates.

Images