TW202221558A - Method for picking workpieces with 3d vision device - Google Patents

Abstract

The invention is to disclose a method for picking workpieces with a 3D vision device. The point cloud mode of a workpiece is photographed by a 3D vision device, and the geometry center of the point cloud mode projects onto the point cloud surface to form a geometry center projection point. The method alternately selects the geometry center projection point or the continuous smooth area to be a pick point to increase the pick success

Description

Stereo vision retrieval method

The present invention relates to a visual object retrieval method, in particular to a method in which a robot arm obtains a point cloud image by using stereoscopic vision, detects around the object retrieval point of an object, and selects the object to be retrieved.

With the rapid development of production technology, factory automation uses visual devices to locate objects, guide robotic arms to grab objects, and automatically assemble and manufacture to increase production speed. The main key to affecting the efficiency of automatic grasping of objects by the robot arm is the random bin picking of various object shapes.

In the prior art, the method of picking up objects from random piles is common. The 3D vision device is used to photograph the object, and after obtaining the point cloud image of the object, the pre-stored object model, such as geometric models such as cubes, spheres, and capsules, CAD models, or a 3D camera is used in advance. The created object point cloud, etc., are compared to identify the photographed object. Then, according to the pre-set specific holding points of various object models, control the robot arm to grasp the specific holding points of the object in an appropriate posture, and take out the object. This method of picking up objects is limited to picking objects from random piles of known shapes, and cannot be applied to picking objects from random piles of unknown shapes.

In addition, the prior art also uses artificial intelligence (AI) training for picking up objects, so that the algorithm can be trained for many times so that it can generate appropriate picking points for unknown object shapes, but the training takes a long time. , it is not easy to be applied in the industry, and the price and cost are high. Therefore, there is another prior art method of picking up objects with unknown shapes, using a 3D vision device to photograph objects, After the point cloud image of the object is obtained, the flat part in the point cloud is calculated, and the flat part of a specific area is selected, and the center of the specific area is used as the object fetching point.

However, although the aforementioned prior art is suitable for picking objects from random piles of unknown shape, the picking point is often far from the center of gravity of the object because the selected flat part is located at the edge of the object. cause the retrieval to fail. Therefore, there are still problems to be solved urgently in the method of picking up objects in stereoscopic vision.

The object of the present invention is to provide a method for retrieving objects in stereoscopic vision. The geometric center projection point is formed by projecting the geometric center point of the point cloud image of the object onto the surface of the point cloud, selecting the geometric center projection point as the object retrieving point, and then checking the preset object retrieving point. Flatness around the radius for successful retrieval.

Another object of the present invention is to provide a method for retrieving objects in stereo vision, using the center point of the connected area of the flatness in the point cloud surface of the selected geometric center projection point as the object extraction point, and then calculating the separation distance between the center point and the geometric center projection point , to stabilize the object's center of gravity.

Another object of the present invention is to provide a method for retrieving objects with stereoscopic vision, which is used interactively to increase the success rate of retrieving objects by selecting the geometric center projection point as the retrieving point or the center point of the connected area of flatness as the retrieving point.

In order to achieve the purpose of the foregoing invention, the method for retrieving objects with stereoscopic vision of the present invention uses the point cloud image of the object to be photographed in three dimensions, separates the point cloud image of a single object, calculates the geometric center of the object point cloud image, and then makes the object point according to the operation axis of the tool. The geometric center of the cloud image is projected onto the point cloud surface of the object, and the geometric center projection point is formed on the surface of the point cloud. Applicable evaluation operations are performed on the method of first stereo vision fetching to check whether the method of first stereo vision fetching is applicable. The method of checking the first stereo vision is applicable, control the robot arm to take the object, if checking the first stereo The method of picking up objects with stereo vision is not applicable, and then perform the applicable evaluation calculation on the method of picking up objects with second stereo vision. If the method of picking objects with second stereo vision is applicable, control the robotic arm to pick up objects. If the method of the object is not applicable, a false alarm is issued.

The two methods for retrieving objects in stereoscopic vision of the present invention are used alternately, and when the first method for retrieving objects in stereoscopic vision is the method for retrieving objects in stereoscopic vision at the geometric center projection point, the second method for retrieving objects in stereoscopic vision is the center point of the flat surface The method of stereo vision retrieval. Whereas the first method for retrieving objects in stereo vision is the method for retrieving objects in stereo vision at the center point of the flat surface, and the second method for retrieving objects in stereo vision is the method for retrieving objects in stereo vision at the projection point of the geometric center.

In the method of the present invention for retrieving objects by stereo vision of the geometric center projection point, when evaluating the operation, the geometric center projection point is selected as the object retrieving point. If the point cloud surface is not flat within a predetermined range around the inspection point, the geometric center projection point is not applicable as the object point. In the method for retrieving objects in stereo vision of the center point of the flat surface of the present invention, when evaluating the operation, the flatness of the point cloud surface where the geometric center projection point is located is calculated, and the flatness connected area larger than the preset area is selected, and the center point of the calculated connected area is Take the object point, calculate the distance between the center point of the point cloud surface and the projected point of the geometric center. If the inspection distance is less than the preset value, the center point of the point cloud surface will be used as the object taking point, which is suitable for taking objects. If it is less than the preset value, then check if there is an unselected connected area, then select another connected area, go back to the center point of the calculated connected area, and continue to search for the pick-up point. If the check distance is not less than the preset value, check again if there is no unselected , the center point of the connected area is not applicable as the fetching point.

10: Stereo vision retrieval system

11: Robot Arm

12: Stereoscopic vision device

13: Container

14: Objects

15: Controller

16: Pick up point

17: Tools

18: Predetermined range

19: Connected Regions

C: center point

G: geometric center

G': geometric center projection point

F: Operation axis

FIG. 1 is a schematic diagram of the stereoscopic vision picking system of the present invention.

FIG. 2 is a schematic diagram of determining an object picking point according to the first embodiment of the present invention.

FIG. 3 is a flowchart of a method for retrieving objects with stereoscopic vision according to the first embodiment of the present invention.

FIG. 4 is a schematic diagram of determining an object picking point according to the second embodiment of the present invention.

FIG. 5 is a flowchart of a method for retrieving objects with stereoscopic vision according to the second embodiment of the present invention.

FIG. 6 is a flowchart of a method for retrieving objects with stereoscopic vision according to a third embodiment of the present invention.

Regarding the technical means adopted by the present invention to achieve the above-mentioned objects and their effects, the following examples are given and described in conjunction with the drawings.

Please refer to FIG. 1 and FIG. 2 at the same time. FIG. 1 is a schematic diagram of a stereoscopic vision object picking system of the present invention, and FIG. 2 is a schematic diagram of determining an object picking point according to the first embodiment of the present invention. In FIG. 1 , in the stereoscopic vision object picking system 10 of the present invention, a stereoscopic vision device 12 is installed in the working environment of the robot arm 11 , and the stereoscopic vision device 12 is used to photograph a plurality of objects 14 piled up randomly in the container 13 to obtain the depth of the objects 14 piled up in disorder. The depth image or point cloud information is transmitted to the controller 15 , and the controller 15 uses image processing to calculate and separate the point cloud of a single object 14 from the cluttered objects 14 to locate the object 14 . At the object point 16, the controller 15 then controls the robot arm 11 to move a tool 17 such as a suction cup at the end to pick up the object 14 at the object pickup point 16 to complete the pickup operation. Although the stereoscopic vision device 12 is installed in the working environment outside the robot arm 11 in this embodiment, the present invention includes and is not limited to this embodiment, and the present invention can also be applied to dispose the stereoscopic vision device 12 on the robot arm 11 .

Figure 2. In the method for retrieving objects with stereoscopic vision according to the first embodiment of the present invention, when determining the retrieving point 16 of the object 14, the controller 15 first calculates the geometric center G of the point cloud of the object 14 from the point cloud of the separated object 14, and the controller 15 According to the attitude data of the control robot arm 11, obtain the With the operation axis F of 17, the geometric center G of the point cloud image of the object 14 is projected to the point cloud surface of the object 14 along the direction of the operation axis F to the tool 17, and the geometric center projection point G' is formed on the point cloud surface, and the geometric The center projection point G' is the center, the preset radius d is set, the predetermined range 18 is set, and the point cloud surface of the object 14 within the predetermined inspection range 18 is checked. During inspection, the controller 15 is based on the predetermined range 18. The difference is maintained at a fixed ratio respectively, and it is judged whether the surface of the point cloud within the predetermined range 18 is flat. If it is checked to be flat, the geometric center projection point G' can be set as the object fetching point 16, and the tool 17 can be fully approached to the fetching point. Around 16, ensure a stable center of gravity to pick up the object 14. On the contrary, if the projected point G' of the geometric center is just at the edge of the corner, the predetermined range 18 includes the point cloud surfaces on both sides of the corner, and the inspection is not flat, and the tool 17 cannot be fully supported. Around the retrieval point 16, the geometric center projection point G' is not suitable as the retrieval point 16, and must be discarded, and the retrieval point 16 must be searched again. Therefore, the method for retrieving objects in stereoscopic vision according to the first embodiment of the present invention may be simply referred to as the method for retrieving objects in stereoscopic vision at the geometric center projection point.

As shown in FIG. 3 , it is a flowchart of a method for retrieving objects with stereoscopic vision according to the first embodiment of the present invention. The detailed steps of the method for retrieving objects by stereo vision of the geometric center projection point according to the first embodiment of the present invention are described as follows: Step P1, the point cloud image of the object is stereoscopically photographed, and the point cloud image of a single object is separated; Step P2, the point cloud image of the object is calculated. Geometric center; Step P3, according to the operation axis of the tool, project the geometric center of the point cloud image of the object to the point cloud surface of the object, and form a geometric center projection point on the point cloud surface; Step P4, select the geometric center projection point as the fetching point; Step P5, check whether the point cloud surface is flat within a predetermined range around the picking point? If the inspection is flat, go to step P6, and the extraction point set by the geometric center projection point is suitable for the extraction. If the inspection is not flat, go to step P7, and the geometric center projection point is not suitable as the extraction point.

Please refer to FIG. 1 and FIG. 4 at the same time. FIG. 4 is a schematic diagram of determining an object picking point according to a second embodiment of the present invention. The stereoscopic vision acquisition of the second embodiment of the present invention is the same as that of the previous embodiment. When taking the object point 16 of the object 14, from the point cloud of the separated object 14, the controller 15 first calculates the geometric center G of the point cloud of the object 14, and the controller 15 obtains the operation of the tool 17 according to the attitude data of the control robot arm 11 Axis F, so that the geometric center G of the point cloud image of the object 14 is projected to the point cloud surface of the object 14 along the operation axis F toward the tool 17, and the geometric center projection point G' is formed on the point cloud surface.

The difference between this embodiment and the previous embodiment is that, in FIG. 4 , the surface flatness of the point cloud of the object 14 where the geometric center projection point G′ of the geometric center G is located is calculated, and the connected area 19 that is larger than the preset area flatness is found, so the connection is There may be more than one area 19, then calculate the center point C of one of the connected areas 19, and select the center point C of the connected area 19 as the extraction point 16, and then calculate the distance between the center point C of the connected area 19 and the geometric center projection point G' distance, check if the separation distance is less than the preset value? If the separation distance is less than the preset value, it means that the distance between the object fetching point 16 and the geometric center G of the point cloud image of the object 14 is very close, so as not to cause the center of gravity to be unstable when taking the object. On the contrary, if the separation distance is not less than the preset value, it means The distance between the object fetching point 16 and the geometric center G of the point cloud image of the object 14 is relatively far, which will cause the center of gravity to be unstable when fetching the object, and must be discarded. Re-find another connected area 19 and recalculate until the center of a connected area 19 is found. Point C can be used as the retrieval point 16, otherwise it must be discarded when no usable connecting area 19 is found. Although this embodiment uses the separation distance as an example, the present invention includes and is not limited to this embodiment, and the present invention can also set the default value of the separation distance as a difference between the overall size of the point cloud surface or the size of the point cloud surface of the whole object. Judgment by percentage. Therefore, the method for retrieving objects with stereoscopic vision according to the first embodiment of the present invention may be simply referred to as the method for retrieving objects with stereoscopic vision at the center point of a flat surface.

As shown in FIG. 5 , it is a flowchart of a method for retrieving objects with stereoscopic vision according to the second embodiment of the present invention. The detailed steps of the method for retrieving objects by stereo vision of the center point of the flat surface according to the second embodiment of the present invention are described as follows: step S1, the point cloud image of the object is stereoscopically photographed, and the point cloud image of a single object is separated; step S2, the point cloud image of the object is calculated. geometric center; step S3, according to the operation axis of the tool, project the geometric center of the object point cloud image to the point cloud surface of the object, and form a geometric center projection point on the point cloud surface; step S4, calculate the geometric center projection point on the point cloud surface Flatness; Step S5, select a connected area with a flatness greater than the preset area; Step S6, calculate the center point of the connected area as a fetch point; Step S7, calculate the separation distance between the center point and the geometric center projection point; Step S8, check Is the separation distance smaller than the preset value? If the check separation distance is less than the preset value, then go to step S9, the center point of the connected area is used as the object picking point, and if the check separation distance is not less than the preset value, then go to step S10, check whether there is an unselected connected area ? If there is an unselected connected area, then go to step S11, select another connected area, go back to step S6 to continue searching for the object fetching point, if there is no unselected connected area, then go to step S12, the center point of the connected area is used as the object fetching point Not available for pickup.

As shown in FIG. 6 , it is a flowchart of a method for retrieving objects with stereoscopic vision according to the third embodiment of the present invention. The third embodiment of the present invention is to use the first embodiment of the method for retrieving objects in stereo vision from the geometric center projection point and the method for retrieving objects in stereo vision from the center point of the flat surface in the second embodiment for interactive evaluation, so as to improve the success rate of retrieving objects, The detailed steps of the method for retrieving objects with interactive stereoscopic vision according to the third embodiment of the present invention are described as follows: step T1, the point cloud image of the object is stereoscopically photographed, and the point cloud image of a single object is separated; step T2, the geometric center of the object point cloud image is calculated; Step T3, according to the operation axis of the tool, the geometric center is projected to the surface of the point cloud of the object, and the projected point of the geometric center is formed on the surface of the point cloud; Step T4, the method of picking up the object in the first stereoscopic vision is subjected to a suitable evaluation calculation; Step T5 , is the method of checking the first stereo vision applicable? If the method for checking the first stereoscopic vision object is applicable, then go to step T6, control the robot arm to pick up the object, if the method for checking the first stereoscopic vision object is not applicable, then go to step T7, the second stereoscopic vision object extraction method Carry out the applicable evaluation operation; Step T8, check whether the method for picking up the second stereoscopic vision is applicable? If checking the second stereo If the method of visual retrieval is applicable, then go to step T6 to control the robotic arm to retrieve the object. If the method for checking the second stereo vision retrieval is not applicable, then go to step T9 and issue a false alarm.

The method for retrieving objects in stereo vision according to the third embodiment of the present invention, wherein the first method for retrieving objects in stereo vision can be the method for retrieving objects in stereo vision by the geometric center projection point of the first embodiment or the The method for taking objects with stereoscopic vision, when the first method for taking objects in stereoscopic vision is the method for taking objects in stereoscopic vision at the geometric center projection point of the first embodiment, the method for taking objects in second stereoscopic vision is the flat surface of the second embodiment The method of taking the object with the stereoscopic vision of the center point, and when the method of taking the object with the first stereoscopic vision is the method of taking the object with the stereoscopic vision of the center point of the flat surface of the second embodiment, the method of taking the object with the second stereoscopic vision is the first. The embodiment is a method for stereoscopically retrieving objects from a geometric center projection point.

Therefore, according to the method for extracting objects with stereoscopic vision of the present invention, the point cloud image of the object can be separated by stereoscopically shooting the point cloud image of the object, and the geometric center point of the object point cloud image can be projected onto the surface of the point cloud to form the geometric center projection point. The projection point of the geometric center is the extraction point, and then check the flatness around the preset radius of the extraction point, or select the flatness connected area of the point cloud surface of the projection point of the geometric center, calculate the center point of the point cloud plane as the extraction point, and then Calculate the distance between the center point of the point cloud plane and the projection point of the geometric center, or use the aforementioned three-dimensional vision object retrieval method interactively to stabilize the center of gravity of the object and achieve the purpose of the invention to improve the success rate of object retrieval.

The above descriptions are only for the convenience of describing the preferred embodiments of the present invention, and the scope of the present invention is not limited to these preferred embodiments. Any changes made according to the present invention shall not depart from the spirit of the present invention. , all belong to the scope of the patent application of the present invention.

10: Stereo vision retrieval system

11: Robot Arm

12: Stereoscopic vision device

13: Container

14: Objects

15: Controller

16: Pick up point

17: Tools

Claims (13)

A method for picking up objects with stereoscopic vision, the steps comprising: The point cloud image of the stereo shooting object is separated, and the point cloud image of a single object is separated; Calculate the geometric center of the object's point cloud; According to the operation axis of the tool, the geometric center of the object's point cloud image is projected to the object's point cloud surface, and the geometric center projection point is formed on the point cloud surface; Applicable evaluation operations are performed on the method of first stereo vision retrieval; Check whether the first stereo vision method is applicable; If the method for checking the first stereo vision is applicable, control the robotic arm to take the object; If the method for checking the first stereo vision object is not applicable, then perform the applicable evaluation calculation on the method for the second stereo vision object extraction. The method for picking up objects with stereoscopic vision as described in item 1 of the scope of the patent application, wherein the method for picking up objects with second stereoscopic vision is suitable for evaluating and calculating, and the method for checking objects picking up with second stereoscopic vision is applicable to control the robotic arm for picking up objects. If the method for picking up objects with stereoscopic vision described in item 2 of the scope of the patent application, wherein the method for checking objects with second stereoscopic vision is not applicable, a false alarm will be issued. The method for retrieving objects in stereoscopic vision as described in item 1 of the scope of application, wherein the first method for retrieving objects in stereoscopic vision is a method for retrieving objects in stereoscopic vision at the geometric center projection point, and the method for retrieving objects in second stereoscopic vision is flat The method of taking objects in the stereo vision of the center point of the surface. The method for retrieving objects in stereoscopic vision as described in item 1 of the scope of application, wherein the first method for retrieving objects in stereoscopic vision is a method for retrieving objects in stereoscopic vision at the center point of a flat surface, and the method for retrieving objects in second stereoscopic vision is geometric The method of stereo vision retrieval of the center projection point. The method for picking up objects with stereoscopic vision as described in item 4 or 5 of the scope of the application, wherein the method for picking up objects with stereoscopic vision of the geometric center projection point, the evaluation calculation step comprises: a method for a robot to safely compensate for weight, the steps comprising: : Select the geometric center projection point as the fetch point; Check that the surface of the point cloud is flat within a predetermined range around the extraction point, and it is suitable for extraction. According to the method for picking up objects with stereo vision as described in item 6 of the scope of the patent application, in which it is checked that the surface of the point cloud within a predetermined range around the picking point is not flat, the geometric center projection point is not suitable as the picking point. The method for extracting objects with stereo vision as described in item 7 of the scope of the patent application, wherein the predetermined range is centered on the projection point of the geometric center, and the predetermined radius forms the predetermined range. The method for picking up objects with stereoscopic vision as described in item 4 or 5 of the scope of the application, wherein the method for picking up objects with stereoscopic vision at the center point of the flat surface, the evaluation calculation steps include: Calculate the flatness of the point cloud surface where the geometric center projection point is located; Select a flatness connected area larger than the preset area; Calculate the center point of the connected area as the fetch point; Calculate the distance between the center point of the connected area and the projected point of the geometric center; Check that the separation distance is less than the preset value, then the center point of the point cloud surface is used as the picking point, which is suitable for picking up objects. According to the method for extracting objects with stereo vision as described in item 9 of the scope of the patent application, the distance between them is checked not less than a preset value, and if there is an unselected connected area, another connected area is selected and returned to the center of the calculated connected area. Click to continue to find the pickup point. For the method for extracting objects with stereo vision described in item 9 of the scope of the patent application, in which the distance between the inspections is not less than the preset value, and if there is no unselected connected area, the center point of the connected area cannot be used as the for the pick-up point. The method for retrieving objects with stereoscopic vision as described in item 9 of the scope of the patent application, wherein the preset value of the separation distance is set as a percentage compared to the overall size of the point cloud surface or the point cloud surface size of the overall object. The method for extracting objects with stereo vision as described in item 1 of the scope of the patent application, wherein the tool obtains the operation axis of the tool according to the attitude data of the robot arm, so that the geometric center of the point cloud of the object is along the operation axis to the direction of the tool, Project to the object's point cloud surface.

Images