TWI675000B - Object delivery method and system - Google Patents

Abstract

An object transport method includes: (A) In the case where an object is held by a holding module of a transport unit, a processing unit controls a depth camera to take a picture to obtain a reference, which indicates that the object appears (B) the processing unit compares the reference data with a pre-stored template data to generate a piece of correction data, the template data indicates a target posture presented by another object, and the correction data Related to the offset of the object in the three-dimensional space between the initial posture and the target posture; (C) the processing unit controls the holding module to be driven to a correction according to the correction data and a pre-stored predetermined path After holding the position, release the object.

Description

Object transportation method and system

The present invention relates to a method and system for object transportation, and more particularly, to a method and system for object transportation related to image recognition.

In modern society, many industries have gradually moved towards fully automated development to save labor costs. For example, in automated production, processing, or packaging lines, robotic arms are often used to transport multiple items one by one to the area to be processed (such as on a conveyor belt or processing platform) in preparation for the next processing of the object program.

However, in each process of transporting the object to the area to be processed, the position of the robot arm contacting the object when acquiring the object is different. Therefore, the posture of each object when held by the robot arm is also different. In this way, even if the path of the robotic arm to transport the objects is fixed, the objects placed by the robotic arm in the area to be processed may still have a significant deviation in angle or position.

For an automatic processing program that requires high accuracy, if the attitude deviation of the object in the area to be processed is too large, it will easily cause the processing result to be less than expected, and even cause processing failure and increase time and material costs. Therefore, the prior art still has significant room for improvement in the transportation of objects.

One object of the present invention is to provide a method for transporting articles which can improve the shortcomings of the prior art.

Thus, the object transport method of the present invention is implemented by an object transport system including an in-depth camera, a transport unit adapted to move the object, and a processing unit. The transport unit includes a transportable and applicable unit. A holding module for holding the object. The object transportation method includes: (A) in the case that the holding module holds the object, the processing unit controls the depth camera to take a picture to obtain a reference material, the reference material indicates a shot of the object on the depth camera An initial posture presented within the range; (B) the processing unit compares the reference data with a pre-stored template data to generate a piece of correction data, the template data indicates that another object is presented in the shooting range A target posture, and the correction data is related to the offset of the object in the three-dimensional space between the initial posture and the target posture; (C) the processing unit according to the correction data and a pre-stored predetermined path Control the holding module to be driven to a correction position and release the holding of the object.

In some embodiments of the method for transporting objects according to the present invention, the reference data and the template data each include multiple coordinate point data, and each coordinate point data includes a two-dimensional image part and a three-dimensional point cloud part, and step ( B) Contains: (b1) the processing unit uses the instance segmentation technology to identify the initial posture presented by the object based on the two-dimensional image portions of the reference material; (b2) the processing unit according to the initial posture and the target The difference in the two-dimensional plane between the attitudes produces an estimated offset result; (b3) the processing unit uses the estimated offset result as a reference to the three-dimensional point cloud portions of the reference data and the template data. The three-dimensional point cloud parts are subjected to three-dimensional matching, and the correction data is generated according to the three-dimensional matching result.

In some implementation aspects of the object transport method of the present invention, in step (b2), the estimated offset result indicates an estimated offset degree of the object in the three-dimensional space between the initial posture and the target posture. And, in step (b3), the processing unit first generates a pair of preliminary matching data that should be the estimated offset result according to the estimated offset result, and then according to the preliminary matching data to the three-dimensional reference data. The point cloud part is subjected to a preliminary matching process, and then the processing unit performs three-dimensional matching on the three-dimensional point cloud part of the reference material and the three-dimensional point cloud part of the template data after the preliminary matching process. The preliminary matching processing is to perform displacement and rotation of the three-dimensional point cloud portion of the reference material in the three-dimensional space according to the preliminary matching data, and the preliminary matching data includes a preliminary matching displacement direction, a preliminary matching displacement distance, and a preliminary matching. Rotation direction and a preliminary matching rotation angle.

In some embodiments of the method for transporting an object according to the present invention, in step (A), the reference material has a plurality of reference feature parts corresponding to the object, and in step (B), the template data has a plurality of reference features. Corresponding to target feature parts of the reference feature parts, and the processing unit generates the correction data according to a relative position in the three-dimensional space between each reference feature part and the target feature part corresponding to the reference feature part.

In some embodiments of the method for transporting an object according to the present invention, in step (B), the correction data includes a displacement correction data and a rotation correction data, and the displacement correction data corresponds to the object in the initial posture and the target posture. The position is offset in three-dimensional space, and the rotation correction data corresponds to the angular offset of the object in the three-dimensional space between the initial posture and the target posture.

In some embodiments of the object transport method of the present invention, the object transport system further includes a storage unit, and the object transport method further includes before step (A): (D) holding another object in the holding module In the case, the processing unit controls the holding module to be driven to a pair of shooting positions corresponding to the depth camera, and controls the depth camera to shoot to obtain the template data; (E) the processing unit stores the template data in the Storage unit; (F) in the case that the holding module holds the object, the processing unit controls the holding module to be driven to the shooting position, and executes step (A).

In some embodiments of the object transport method of the present invention, the storage unit stores an object recognition neural network model trained in a deep learning manner, and the transport unit further includes a robot arm for driving the holding module. , And a photographic module that can be driven by the robotic arm and corresponds to the position of the holding module; the object transportation method further includes a step between (E) and (F): (G) the processing unit borrows The object recognition-type neural network model and the machine vision of the photographic module identify the object from other objects, and control the robotic arm to drive the holding module to the position of the object, and control the holding mold The group holds the object.

In some embodiments of the object transport method of the present invention, the object transport system includes a plurality of depth cameras, and a plurality of shooting lenses of the depth cameras are respectively aligned to a shooting position from a plurality of different directions;

The object transportation method further includes before the step (A): (H) In a case where the holding module holds another object and is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain multiple A three-dimensional point cloud template, and merge the three-dimensional point cloud templates into the template data by three-dimensional stitching and store the template data. In step (A), when the holding module is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain a plurality of three-dimensional point cloud models, and uses the three-dimensional point cloud models to Three-dimensional stitching is incorporated into this reference.

Another object of the present invention is to provide an article transport system capable of implementing the article transport method.

The object transport system of the present invention is suitable for transporting an object. The object transport system includes a depth camera, a transport unit, and a processing unit electrically connected to the depth camera and the transport unit. The transport unit is adapted to move the object and includes a holding module which can be driven and adapted to hold the object. In the case where the holding module holds the object, the processing unit controls the depth camera to take a picture to obtain a reference, which indicates an initial posture that the object exhibits within a shooting range of the depth camera. , The processing unit compares the reference data with a pre-stored template data to generate a piece of correction data, the template data indicates a target posture presented by another object in the shooting range, and the correction data is related to The offset of the object in the three-dimensional space between the initial posture and the target posture, the processing unit controls the holding module to be driven to a correction position according to the calibration data and a pre-stored predetermined path, and then releases the holding of the module. object.

In some embodiments of the object transport system of the present invention, the object transport system includes a plurality of depth cameras, and a plurality of shooting lenses of the depth cameras are respectively aligned from a plurality of different directions to a shooting position, and the holding In the case where the module holds another object and is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain multiple 3D point cloud templates, and merges the 3D point cloud templates into a 3D stitching method into The template data is stored and the template data is stored. In the case that the holding module is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain a plurality of three-dimensional point cloud models, and the 3D point cloud The models are merged into this reference material in three-dimensional stitching.

The effect of the present invention is that the object transport system can generate the correction data according to the offset of the object in the three-dimensional space between the initial posture and the target posture, and control the holding mold according to the correction data and the predetermined path. The group is driven to the correction position, which can correct the object's offset in the three-dimensional space between the initial attitude and the target attitude. In this way, the object transport system can make multiple objects more uniform. The attitude is placed at the position to be processed, so it is conducive to subsequent automatic processing programs, and it can indeed improve the inconvenience of the existing technology.

Before the present invention is described in detail, it should be noted that in the following description, similar elements are represented by the same numbers. In addition, the "electrical connection" described in this patent specification generally refers to a wired electrical connection between multiple electronic devices / devices / components connected through conductive materials, and a radio connection for wireless signal transmission through wireless communication technology . In addition, the "electrical connection" described in this patent specification also refers to the "direct electrical connection" formed by direct connection between two electronic devices / devices / components, and also through the two electronic devices / devices / components. "Indirect electrical connection" formed by the connection of other electronic equipment / devices / components.

Referring to FIG. 1 and FIG. 2 at the same time, a first embodiment of the object transport system 1 of the present invention is suitable for transporting a stack of objects 5 (shown in FIG. 2) from a to-be-transported position to a to-be-processed position one by one. . The objects 5 are identical to each other and may be, for example, components of an electronic product, and the processing position may be, for example, a conveyor belt or a specific position on a processing platform, but is not limited thereto.

The object transport system 1 includes a depth camera 11, a transport unit 12 adapted to transport the objects 5 one by one, a storage unit 13, and a process electrically connected to the depth camera 11, the transport unit 12 and the storage unit 13. Unit 14.

As shown in FIG. 2, the depth camera 11 has a shooting lens 111, and the depth camera 11 is fixedly arranged so that the shooting lens 111 is aligned with a shooting position. It is added that, in this embodiment, the depth camera 11 is, for example, implemented as a depth camera using Time of Flight (ToF in English for short) technology. However, in other embodiments, the depth camera 11 may also be implemented as a depth camera using structured light (Structured Light in English) technology or stereo vision (Stereo Vision in English) technology, and is not limited to this embodiment. .

The transport unit 12 includes a movable robot arm 121, a holding module 122 capable of holding any one of the objects 5, and a photographing module 123.

The robot arm 121 has a fixed end 124, a free end 125 opposite to the fixed end 124, and a plurality of movable joints 126 located between the fixed end 124 and the free end 125 (only shown as an example in FIG. 2). There are two movable joints 126, but in an actual implementation, the number of the movable joints 126 may be more).

The holding module 122 is disposed at the free end 125 of the robot arm 121, whereby the robot arm 121 can drive the holding module 122 to move and rotate in various directions within a working range. Moreover, in this embodiment, the holding module 122 is implemented as, for example, a jig, and any one of the objects 5 can be held in a clamping manner. However, in other implementations, the holding module 122 can also be implemented as a different type of robot arm end tool (English: End of Arm Tooling, EOAT for short) according to different needs. That is, the specific implementation of the holding module 122 can be implemented as long as it can hold any one of the objects 5, and is not limited to this embodiment.

The photographing module 123 is disposed on the robot arm 121 and corresponds to the position of the holding module 122. More specifically, the photographing module 123 is, for example, farther away from the fixed end 124 of the robot arm 121 and closer to the robot arm 121. The free end 125 of the robot arm 121. Thereby, the photographing module 123 can be driven by the robot arm 121 together with the holding module 122. It is further explained that, in this embodiment, the photographing module 123 is implemented as, for example, another depth camera independent of the depth camera 11, and preferably, the photographing module 123 is implemented as an application, for example. The depth camera of the stereo vision technology helps reduce the overall cost of the transport unit 12, but is not limited to this.

As shown in FIG. 1, the storage unit 13 stores an object recognition neural network model M1 having a predetermined path R, a pair of objects that should wait for the appearance of the object 5, and a pair of instance segmentation neural networks that should wait for the appearance of the object 5. Road model M2.

The predetermined path R is used in this embodiment for the processing unit 14 to control the robot arm 121 to drive the holding module 122 from the shooting position to a predetermined position adjacent to the processing position, and the predetermined path R is, for example, It contains a plurality of three-dimensional coordinates that are sequential with each other, but it is not limited to this.

The object recognition neural network model M1 is trained in a deep learning manner, and with the object recognition neural network model M1, even if the objects 5 are staggered and stacked on each other, the processing unit 14 can still The machine vision of the photographing module 123 recognizes one or more of the objects 5 on the uppermost layer, and recognizes the position and angle of each of the objects 5 on the uppermost layer.

The instance segmentation neural network model M2 is also trained in a deep learning manner, and by using the instance segmentation neural network model M2, the processing unit 14 can use instance segmentation (English Segmentation) technology Identify the outline of each independent object from a two-dimensional image. Furthermore, in this embodiment, the example segmentation neural network model M2 is completed based on “Mask R-CNN” technology and deep learning, but is not limited thereto. It is added that the full name of "R-CNN" is "Region-based Convolutional Neural Network", and the Chinese name of "Convolutional Neural Network" is "Convolutional Neural Network". In addition, in other embodiments, the example segmentation neural network model M2 can also be completed based on other types of convolutional neural network related technologies and deep learning, so it is not limited to this embodiment.

The processing unit 14 is implemented as a central processing module of a computer in this embodiment, and preferably, the processing unit 14 is implemented as a central processing module of a workstation computer, for example. , But not limited to this.

The following exemplifies in detail how the article transport system 1 of this embodiment implements an article transport method. In this embodiment, the object transportation method includes a modeling process and a transportation process located after the modeling process. Specifically, the modeling process is performed by the article transport system 1 on a modeling object (not shown in the figure) that is substantially identical to the objects 5, and the transport process is the article transport system. 1 The process of actually transporting each item 5 to the position to be processed.

Referring to FIG. 3 and referring to FIG. 1, the following first describes the modeling process of the object transportation method.

First, in step S11, the processing unit 14 controls the robot arm 121 to drive the holding module 122 to the shooting position, and controls the holding module 122 to hold the modeling object so that the modeling object It is exposed within a shooting range of the depth camera 11. It is added that, in this step, the position and angle of the modeling object held by the holding module 122 can be manually adjusted manually, so that the modeling object is relative to the depth camera 11. It is displayed at a right angle and is located in the center of the shooting range, but not limited to this.

Referring to FIG. 4, in step S12, the processing unit 14 controls the depth camera 11 to take a picture, and thereby obtains a piece of template data D1 from the depth camera 11 (shown in FIG. 4). In this embodiment, the template data D1 indicates a target posture presented by the modeling object within the shooting range of the depth camera 11. More specifically, the target posture represents the modeling in this embodiment. Use the position and angle of the object in the shooting range.

In this embodiment, the template data D1 includes a plurality of coordinate point data, and each coordinate point data includes a two-dimensional image portion and a three-dimensional point cloud portion. Each two-dimensional image portion includes a red color value, a green color value, and a blue color value, and each three-dimensional point cloud portion includes an X-axis coordinate value, a Y-axis coordinate value, and a Z-axis coordinate value.

More specifically, the two-dimensional image portions of the template data D1 collectively constitute a color two-dimensional target image D11 (that is, a color photo) included in the template data D1, and the two-dimensional target The image D11 has a two-dimensional target portion D111 showing the appearance of the modeling object. On the other hand, the three-dimensional point cloud portions of the template data D1 collectively constitute a three-dimensional point cloud template D12 (that is, a 3D Model) included in the template data D1, and the three-dimensional point cloud template D12 has a The three-dimensional target portion D121 showing the appearance of the modeling object. In other words, the two-dimensional image portions of the template data D1 collectively indicate the target pose presented by the modeling object in a two-dimensional plane, and the three-dimensional point cloud portions of the template data D1 collectively indicate The target pose presented by the modeling object in three-dimensional space is obtained.

Next, in step S13, the processing unit 14 performs a background filtering process on the template data D1, and stores the template data D1 that has completed the background filtering process in the storage unit 13. Specifically, the background filtering process is performed on the 3D point cloud template D12 of the template data D1, and for example, each 3D point cloud portion in the 3D point cloud template D12 with a depth greater than a predetermined threshold is used as noise filtering. In addition, a part of the background in the three-dimensional point cloud template D12 can be filtered out and the three-dimensional target portion D121 can be retained. In addition, the position of the three-dimensional target portion D121 in the three-dimensional point cloud template D12 is used as a target position, and the angle of the three-dimensional target portion D121 in the three-dimensional point cloud template D12 is used as a target angle. It is added that, in an application sense, the template data D1 is used to define that when the holding module 122 holds each object 5 and is located at the shooting position, the objects 5 held by the holding module 122 are in The ideal position and angle in this shooting range.

The above steps S11 to S13 are the modeling process of the article transportation method. Next, referring to FIG. 5 and in conjunction with FIGS. 1 and 4, the following describes the transportation process of the article transportation method, and for convenience of description and understanding, only one of the articles 5 of the article transportation system 1 will be described below. (Hereinafter referred to as "this article 5") to carry out the process.

First, in step S21, the processing unit 14 controls the robot arm 121 to drive the photographing module 123 to a pair of photographing positions corresponding to the positions to be transported, so that the photographing module 123 is aligned with the stack of objects 5 and makes One or more of the stacks of objects 5 located at the uppermost layer are within the shooting range of the camera module 123.

Next, in step S22, the processing unit 14 recognizes the object 5 from one or more of the stack of objects 5 located at the uppermost layer by using the object recognition neural network model M1 and the machine vision of the photographing module 123. .

Next, in step S23, the processing unit 14 controls the robot arm 121 to drive the holding module 122 to the position of the object 5, and controls the holding module 122 to hold the object 5.

Next, in step S24, when the holding module 122 holds the object 5, the processing unit 14 controls the robot arm 121 to drive the holding module 122 to the shooting position, so that the object 5 follows the The holding module 122 is driven together, and the object 5 is exposed within the shooting range of the depth camera 11 (that is, the situation shown in FIG. 2).

Next, in step S25, the processing unit 14 controls the depth camera 11 to take a picture, and thereby obtains a piece of reference data D2 from the depth camera 11 (shown in FIG. 4). In this embodiment, the reference material D2 indicates an initial posture presented by the object 5 within the shooting range of the depth camera 11. More specifically, the initial posture represents the object 5 in this embodiment. An initial position and an initial angle presented in the shooting range.

Similar to the template data D1, the reference data D2 also contains multiple coordinate point data, and each coordinate point data of the reference data D2 also includes a two-dimensional image part and a three-dimensional point cloud part, and each two-dimensional image part It includes a red color value, a green color value, and a blue color value, and each three-dimensional point cloud part includes an X-axis coordinate value, a Y-axis coordinate value, and a Z-axis coordinate value. In addition, the two-dimensional image portions of the reference material D2 collectively constitute a two-dimensional reference image D21 (that is, another color photograph) included in the reference material D2 and is a color, and the two-dimensional reference image D21 It has a two-dimensional key portion D211 showing the appearance of the object 5. On the other hand, the three-dimensional point cloud portions of the reference material D2 together constitute a three-dimensional point cloud model D22 (that is, another 3D Model) included in the reference material D2, and the three-dimensional point cloud model D22 has A three-dimensional key portion D221 showing the appearance of the object 5. In other words, the two-dimensional image portions of the reference material D2 collectively indicate the initial pose presented by the object 5 in a two-dimensional plane, and the three-dimensional point cloud portions of the reference material D2 collectively indicate the This initial posture of the object 5 in the three-dimensional space. In particular, the reference data D2 and the template data D1 are two data of the same nature but independent of each other.

Next, in step S26, the processing unit 14 performs the background filtering process on the reference material D2. Similar to the foregoing, the background filtering process is performed on the three-dimensional point cloud model D22 of the reference material D2, and for example, each three-dimensional point cloud portion in the three-dimensional point cloud model D22 whose depth is greater than the predetermined threshold is used as noise. Filtering, thereby filtering out a part of the background in the three-dimensional point cloud model D22 while retaining the three-dimensional key portion D221.

Next, in step S27, the processing unit 14 compares the reference data D2 and the template data D1 that have completed the background filtering process, and generates a piece of correction data according to the comparison result. In this embodiment, the correction data includes a displacement correction data and a rotation correction data, and the displacement correction data corresponds to a position offset of the object 5 in the three-dimensional space between the initial posture and the target posture (that is, Offset in distance), and the rotation correction data corresponds to the angular offset (ie, the offset in the direction) of the object 5 in the three-dimensional space between the initial attitude and the target attitude. More specifically, the displacement correction data includes, for example, a 3D correction displacement direction, a 3D correction displacement distance, a 3D correction rotation direction, and a 3D correction rotation angle, but is not limited thereto.

Further referring to FIG. 6, the following describes in detail how the processing unit 14 in this embodiment compares the reference data D2 with the template data D1.

First, in a sub-step S271 shown in FIG. 6, the instance segmentation neural network model M2 is stored by the storage unit 13, and the processing unit 14 uses the instance segmentation technique to the two-dimensional reference to the reference material D2. Image D21 (that is, the two-dimensional image portions of the reference material D2) is used for image recognition, and the two-dimensional key portion D211 (that is, identified from the two-dimensional reference image D21) is identified from the two-dimensional reference image D21. The appearance of the object 5).

Next, in sub-step S272, the neural network model M2 is segmented by the example, and the processing unit 14 further estimates the three-dimensional key portion D221 in the three-dimensional point cloud model according to the two-dimensional key portion D211 of the reference material D2. The position and angle in D22, and produce a pair of estimation results corresponding to the initial attitude. In other words, the processing unit 14 estimates the initial pose of the object 5 in the three-dimensional space based on the color photograph in the reference material D2. It is added that during the training of the example segmentation neural network model M2, the example segmentation neural network model M2 will perform deep learning based on multiple photos of the object 5, and these photos will be divided into multiple photos. The appearance of the object 5 is presented at different angles. Therefore, by segmenting the neural network model M2 through this example, the processing unit 14 can estimate the approximate position of the object 5 in the three-dimensional space based on the photo of the object 5 and angle. In this embodiment, the estimation result includes, for example, an estimated position and an estimated angle, but is not limited thereto.

Next, in sub-step S273, the processing unit 14 compares the estimated position and the estimated angle of the estimated result with the target position and the target angle, respectively, and generates an estimated offset result, and then according to the estimated offset, The result is a pair of preliminary matches that should estimate the offset results. Specifically, the estimated offset result indicates an estimated offset degree of the object 5 in the three-dimensional space between the initial posture and the target posture, and the preliminary matching data is equivalent to the estimated offset. An estimated correction parameter used as a reference for correcting the estimated offset degree in a three-dimensional space. In more detail, the preliminary matching data in this embodiment includes, for example, a preliminary matching displacement direction, a preliminary matching displacement distance, a preliminary matching rotation direction, and a preliminary matching rotation angle.

Next, in sub-step S274, the processing unit 14 performs a preliminary matching process on the three-dimensional point cloud model D22 of the reference material D2 according to the preliminary matching data. More specifically, in the preliminary matching process, the processing unit 14 performs a displacement in a three-dimensional space on the three-dimensional point cloud model D22 of the reference material D2 according to the preliminary matching displacement direction and the preliminary matching displacement distance. The processing unit 14 further rotates the three-dimensional point cloud model D22 of the reference material D2 in the three-dimensional space according to the preliminary matching rotation direction and the preliminary matching rotation angle. Specifically, the preliminary matching displacement direction and the preliminary matching displacement distance are used to correct a difference between the estimated position and the target position of the three-dimensional key portion D221, and the preliminary matching rotation direction and the preliminary matching rotation angle are It is used to correct the difference between the estimated angle and the target angle of the three-dimensional key portion D221, so that the three-dimensional key portion D221 can be adjusted to a posture closer to the three-dimensional target portion D121.

Next, in sub-step S275, the processing unit 14 performs three-dimensional matching (3D Matching) with the three-dimensional point cloud model D22 after the preliminary matching process and the three-dimensional point cloud template D12 of the template data D1, and according to the result of the three-dimensional matching Generate the calibration data. More specifically, the processing unit 14 generates the correction data according to the three-dimensional matching between the three-dimensional key portion D221 and the three-dimensional target portion D121. It is worth mentioning that since the processing unit 14 has performed the preliminary matching processing on the three-dimensional point cloud model D22 in the sub-step S274, the processing unit 14 is equivalent to using the estimated bias in the sub-step S275. The three-dimensional matching is performed on the three-dimensional key portion D221 and the three-dimensional target portion D121 based on the shift result. By performing the preliminary matching process on the three-dimensional point cloud model D22, the difference in position and angle between the three-dimensional key portion D221 and the three-dimensional target portion D121 can be reduced in advance. In this way, it is possible to reduce The three-dimensional matching between the three-dimensional key portion D221 and the three-dimensional target portion D121 is completed at a faster speed. In addition, since the technical details of the three-dimensional matching itself are not the focus of this patent specification, they are not described in detail here.

Next, in step S28 shown in FIG. 5, the processing unit 14 controls the movement of the robot arm 121 according to the calibration data and the predetermined path R, so that the robot arm 121 will hold the holding module 122 holding the object 5. Drive to a pair of correction positions corresponding to the correction data and the predetermined path R, specifically, the correction position is that the holding module 122 controls the robot arm 121 in the processing unit 14 according to the correction data and the predetermined path R to complete the movement Where you are. In addition, in a case where the holding module 122 has been driven to the correction position by the robot arm 121, the processing unit 14 controls the holding module 122 to release the holding of the object 5 to complete the transportation of the object 5.

Specifically, by controlling the movement of the robot arm 121 according to the correction data by the processing unit 14, it is possible to correct the offset of the object 5 in the three-dimensional space between the initial posture and the target posture. On the other hand, by controlling the movement of the robot arm 121 according to the predetermined path R by the processing unit 14, the holding module 122 can be driven to the predetermined position by the robot arm 121. Moreover, in this embodiment, preferably, the processing unit 14 controls the movement of the robot arm 121 according to the calibration data and the predetermined path R at the same time. However, in other embodiments, the processing unit 14 may also be the first The movement of the robot arm 121 is controlled according to the calibration data, and the movement of the robot arm 121 is controlled according to the predetermined path R, but not limited to this embodiment.

The above steps S21 to S28 are the transportation process of the article transportation method.

It is further explained that if the article transport system 1 transports a total of N articles 5 (N is an integer greater than 1), it is equivalent to performing N transport processes, and in the N transport processes, the holding mold The contact position of each object 5 held by the group 122 in step S23 may be different. Therefore, the initial posture of each object 5 when held by the holding module 122 is different, that is, if the processing unit 14 only According to the predetermined path R, the robot arm 121 is controlled to drive the holding module 122 to the predetermined position and release the holding of the object 5, so that the actual position where the object 5 is placed may have a significant deviation from the position to be processed.

In this embodiment, the processing unit 14 generates corresponding correction data according to the initial posture presented by each object 5 in step S25. Therefore, the correction data can be used to control the robot arm 121 to move each object 5 from Various initial attitudes are corrected to the target attitude uniformly. Moreover, in step S28, since the processing unit 14 controls the robot arm 121 to drive the holding module 122 to the correction position according to the calibration data and the predetermined path R, it is equivalent to aligning the objects 5 according to the initial posture of each object 5. The predetermined path R is corrected (that is, the predetermined position is corrected). In this way, each object 5 can be more accurately placed at the position to be processed. Specifically, this embodiment can at least reduce the attitude tolerance of the objects 5 when they are placed in the to-be-processed position to within 1.5 mm, which is beneficial to various subsequent automatic processing programs.

The above is the description of the first embodiment of the article transport system 1 of the present invention.

The article transport system 1 of the present invention also has a second embodiment. The differences between the second embodiment (hereinafter referred to as "this embodiment") and the first embodiment will be described below.

In this embodiment, the storage unit 13 stores the predetermined path R, the object recognition-type neural network model M1, and a pair of feature recognition-type neural network models corresponding to the appearance of the object 5 (not shown in the figure). ). The feature recognition neural network model is trained in a deep learning manner, and by using the feature recognition neural network model, the processing unit 14 can recognize from a two-dimensional image and / or a three-dimensional point cloud. One or more characteristic parts of the appearance of the object 5 are defined in advance.

Moreover, in step S12 of the modeling process of this embodiment, the three-dimensional point cloud template D12 of the template data D1 has a plurality of target feature parts, and the target feature parts are all located in the three-dimensional target part D121. On the other hand, in step S25 of the transportation process, the three-dimensional point cloud model D22 of the reference material D2 has a plurality of reference feature parts corresponding to the target feature parts, and the reference feature parts are all located on the three-dimensional key. Section D221.

In addition, in step S27 of the transportation process, the processing unit 14 compares the reference data D2 with the template data D1 in a manner different from that in the first embodiment.

Specifically, in this embodiment, by using the feature recognition neural network model, the processing unit 14 is in a three-dimensional space based on each reference feature part and the target feature part corresponding to the reference feature part. The relative position of the generated calibration data. In other words, in this embodiment, the processing unit 14 compares the object 5 between the initial posture and the target posture according to the relative positions of the target characteristic parts and the reference characteristic parts with each other. The offset in the three-dimensional space further generates the correction data. Therefore, this embodiment is not a three-dimensional matching between the three-dimensional key portion D221 and the three-dimensional target portion D121 as in the first embodiment. However, the object transport system 1 of this embodiment also corrects the offset of each object 5 in the three-dimensional space between the initial posture and the target posture by comparing the template data D1 with the reference data D2. Therefore, the same technical effect as that of the first embodiment can be achieved.

In addition, in another embodiment, the object transport system 1 may include a plurality of depth cameras 11, and the shooting lenses 111 of the depth cameras 11 are respectively aligned with the shooting position from different directions. For example, the number of the depth cameras 11 may be, for example, three, and the three shooting lenses 111 of the three depth cameras 11 are respectively aligned at the shooting positions from the front, the left, and the right. Moreover, in the modeling process, when the holding module 122 holds the modeling object and is driven to the shooting position by the robot arm 121, the processing unit 14 controls the three depth cameras 11 respectively. The modeling object is photographed, thereby obtaining three three-dimensional point cloud templates D12 from the three depth cameras 11 respectively. Then, the processing unit 14 merges the three three-dimensional point cloud templates D12 into a stitching three-dimensional point cloud template in a three-dimensional stitching manner, and stores the stitched three-dimensional point cloud template as template data in the storage unit. 13. Then, in the transportation process, the processing unit 14 also controls the three depth cameras 11 to photograph the object 5 respectively, thereby obtaining three three-dimensional point cloud models D22 from the three depth cameras 11, and then, The unit 14 merges the three three-dimensional point cloud models D22 into one stitching three-dimensional point cloud model in a three-dimensional stitching manner. Then, the processing unit 14 compares the stitching three-dimensional point cloud model with the template data (that is, the stitching three-dimensional point cloud template) as the reference material, and the comparison method may be, for example, in the first embodiment. The three-dimensional matching (3D Matching) or the feature part comparison described in the second embodiment may also be adopted. By merging the three-dimensional point cloud templates D12 and merging the three-dimensional point cloud models D22, the three-dimensional point cloud used for comparison can be more complete, and the accuracy of comparison and correction can be further improved.

In summary, in each embodiment of the object transport system 1 of the present invention, the object transport system 1 can generate the correction according to the offset of the object in the three-dimensional space between the initial attitude and the target attitude. Data, and controlling the movement of the robot arm 121 according to the correction data and the predetermined path R, thereby being able to correct the offset of the object 5 in the three-dimensional space between the initial posture and the target posture. The object transport system 1 enables the objects 5 to be placed in the to-be-processed position with a more uniform attitude, which is beneficial to subsequent automatic processing programs (especially automatic processing programs that require high accuracy). Therefore, the object The transportation system 1 can indeed improve the inconvenience of the prior art, and indeed can achieve the purpose of the invention.

However, the above are only examples of the present invention. When the scope of implementation of the present invention cannot be limited in this way, any simple equivalent changes and modifications made in accordance with the scope of the patent application and the content of the patent specification of the present invention are still Within the scope of the invention patent.

1‧‧‧ Object Delivery System
11‧‧‧ Depth Camera
111‧‧‧ shooting lens
12‧‧‧ Shipping Unit
121‧‧‧ robot arm
122‧‧‧ holding module
123‧‧‧Photographic Module
124‧‧‧Fixed end
125‧‧‧ free end
126‧‧‧movable joint
13‧‧‧Storage unit
14‧‧‧ processing unit
R‧‧‧ scheduled route
M1‧‧‧ object recognition neural network model
M2‧‧‧ instance segmentation neural network model
5‧‧‧ objects
D1‧‧‧Template Information
D11‧‧‧Two-dimensional target image
D111‧‧‧Two-dimensional target part
D12‧‧‧3D point cloud template
D121‧‧‧Three-dimensional target part
D2‧‧‧Reference
D21‧‧‧Two-dimensional reference image
D211‧‧‧Two-dimensional key part
D22‧‧‧3D point cloud model
D221‧‧‧Three-dimensional key part
S11 ~ S13‧‧‧‧steps
S21 ~ S28‧‧‧‧step
S271 ~ S275‧‧‧Sub-step

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which:
FIG. 1 is a schematic block diagram illustrating a first embodiment of an article transport system according to the present invention;
FIG. 2 is a schematic diagram exemplarily illustrating that a robot arm according to the first embodiment holds an object and is photographed by a depth camera;
FIG. 3 is a flowchart exemplarily illustrating how the first embodiment implements a modeling process in an article transport method; FIG.
FIG. 4 is a schematic diagram exemplarily showing a template data and a reference material generated in the object transportation method;
FIG. 5 is a flowchart exemplarily illustrating how the first embodiment implements a transportation process in the article transportation method; and FIG. 6 is a flowchart exemplarily illustrating how a processing unit compares reference materials and the Template information.

Claims (10)

An object conveying method is implemented by an object conveying system including an depth camera, a conveying unit adapted to move the object, and a processing unit. The conveying unit includes a movable and suitable for holding The object holding module; the object transportation method includes:
(A) In the case where the holding module holds the object, the processing unit controls the depth camera to take a picture to obtain a reference, which indicates that the object appears within a shooting range of the depth camera. An initial posture
(B) The processing unit compares the reference data with a pre-stored template data to generate a piece of correction data, the template data indicates a target posture presented by another object within the shooting range, and the correction data Related to the offset of the object in the three-dimensional space between the initial pose and the target pose; and
(C) The processing unit controls the holding module to be driven to a calibration position according to the calibration data and a pre-stored predetermined path to release the holding of the object. The method for transporting objects according to claim 1, wherein the reference data and the template data each include a plurality of coordinate point data, and each coordinate point data includes a two-dimensional image portion and a three-dimensional point cloud portion, and the step ( B) Contains:
(b1) the processing unit uses the instance segmentation technology to identify the initial posture presented by the object based on the two-dimensional image portions of the reference material;
(b2) the processing unit generates an estimated offset result based on a difference in the two-dimensional plane between the initial attitude and the target attitude; and
(b3) The processing unit uses the estimated offset result as a reference to perform three-dimensional matching on the three-dimensional point cloud portions of the reference data and the three-dimensional point cloud portions of the template data, and generates the Correct the data. The object transport method according to claim 2, wherein in step (b2), the estimated offset result indicates an estimated offset degree of the object in the three-dimensional space between the initial posture and the target posture. And, in step (b3), the processing unit first generates a pair of preliminary matching data that should be the estimated offset result according to the estimated offset result, and then according to the preliminary matching data to the three-dimensional reference data. The point cloud part is subjected to a preliminary matching process, and then the processing unit performs three-dimensional matching on the three-dimensional point cloud part of the reference material and the three-dimensional point cloud part of the template data after the preliminary matching process. The preliminary matching process is to perform displacement and rotation of the three-dimensional point cloud portions of the reference material in three-dimensional space according to the preliminary matching data, and the preliminary matching data includes a preliminary matching displacement direction, a preliminary matching displacement distance, and a preliminary matching. Rotation direction and a preliminary matching rotation angle. The article transportation method according to claim 1, wherein in step (A), the reference material has a plurality of reference feature parts corresponding to the object, and in step (B), the template material has a plurality of Corresponding to target feature parts of the reference feature parts, and the processing unit generates the correction data according to a relative position in the three-dimensional space between each reference feature part and the target feature part corresponding to the reference feature part. The object transport method according to claim 1, wherein in step (B), the correction data includes a displacement correction data and a rotation correction data, and the displacement correction data corresponds to the object in the initial posture and the target posture The position is offset in three-dimensional space, and the rotation correction data corresponds to the angular offset of the object in the three-dimensional space between the initial posture and the target posture. The article transportation method according to claim 1, wherein the article transportation system further includes a storage unit, and the article transportation method further includes before the step (A):
(D) In the case where the holding module holds another object, the processing unit controls the holding module to be driven to a pair of shooting positions corresponding to the depth camera, and controls the depth camera to shoot to obtain the template data;
(E) the processing unit stores the template data in the storage unit; and
(F) In the case where the holding module holds the object, the processing unit controls the holding module to be driven to the shooting position, and executes step (A). The object transport method according to claim 6, wherein the storage unit stores an object recognition neural network model trained in a deep learning manner, and the transport unit further includes a robot arm for driving the holding module. , And a photographic module that can be driven by the robotic arm and corresponds to the position of the holding module; the object transportation method further includes a step between (E) and (F): (G) the processing unit borrows The object recognition-type neural network model and the machine vision of the photographic module identify the object from other objects, and control the robotic arm to drive the holding module to the position of the object, and control the holding mold The group holds the object. The object transport method according to claim 1, wherein the object transport system includes a plurality of depth cameras, and a plurality of shooting lenses of the depth cameras are respectively aligned to a shooting position from a plurality of different directions;
The object transportation method further includes before the step (A): (H) In a case where the holding module holds another object and is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain multiple Three-dimensional point cloud template, and merge the three-dimensional point cloud templates into the template data by three-dimensional stitching and store the template data;
In step (A), when the holding module is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain a plurality of three-dimensional point cloud models, and uses the three-dimensional point cloud models to Three-dimensional stitching is incorporated into this reference. An object transport system is suitable for transporting an object. The object transport system includes:
A depth camera
A transport unit adapted to move the object and including a holding module which can be driven and adapted to hold the object; and a processing unit electrically connected to the depth camera and the transport unit;
Wherein, in the case that the object is held by the holding module, the processing unit controls the depth camera to take a picture to obtain a reference material, the reference material indicates that an object appears in a shooting range of the depth camera. Initial posture, the processing unit compares the reference data with a pre-stored template data to generate a piece of correction data, the template data indicates a target posture presented by another object within the shooting range, and the correction data In relation to the offset of the object in the three-dimensional space between the initial posture and the target posture, the processing unit controls the holding module to be driven to a correction position according to the correction data and a pre-stored predetermined path, and is released. Hold the object. The object transport system according to claim 9 includes a plurality of depth cameras, and the plurality of shooting lenses of the depth cameras are respectively aligned to a shooting position from a plurality of different directions, and the other object is held in the holding module. In the case of being located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain a plurality of three-dimensional point cloud templates, and merges the three-dimensional point cloud templates into the template data and stores the Template data. In the case where the holding module is located at the shooting position, the processing unit controls the depth cameras to shoot separately to obtain multiple three-dimensional point cloud models, and the three-dimensional point cloud models are three-dimensionally stitched. Combined into that reference.