JP6924448B2 - Picking system, picking method, and program - Google Patents

Description

The present invention relates to a picking system, a picking method, and a program for picking a workpiece.

Industrial robots have been introduced to automate operations in factories such as manufacturing factories and food factories. One of the major industrial robots is a robot that picks (grabs) a workpiece, transports it to a desired location, and places it (places it). Here, in Patent Document 1, point cloud data having three-dimensional data of the work is generated by 3D laser measurement technology, position information and normal information of the work are calculated using the point cloud data, and based on these information. An object recognition system that controls a robot is disclosed. By adopting object recognition based on the three-dimensional data of the work obtained by the 3D laser measurement technology instead of the object recognition based on the model matching using the measurement data, it is possible to recognize the atypical work.
Patent Document 1 Japanese Unexamined Patent Publication No. 2011-167815

However, in the above-mentioned object recognition system, since the workpieces can take an arbitrary shape when the workpieces are flexible and can be oriented in an arbitrary direction, the purpose is particularly among them when a plurality of workpieces are overlapped. It is expected that it will be difficult to calculate the position information and normal information of the work for identifying the work and picking it stably.

In the first aspect of the present invention, there is a picking system for picking a work, an imaging unit that photographs the work to obtain an image and a depth map, and a work is specified based on the image, and the depth map is specified. Provided is a picking system including an analysis unit that determines the position and orientation of the work based on the work portion corresponding to the work, and a control unit that drives and controls the robot arm based on the position and orientation of the work.

In the second aspect of the present invention, there is a picking method for picking a work, in which the imaging unit captures the work to obtain an image and a depth map, and the analysis unit identifies the work based on the image. , The stage of determining the position and orientation of the work based on the work portion corresponding to the specified work in the depth map, and the stage of driving and controlling the robot arm based on the position and orientation of the work by the control unit. A picking method is provided.

In the third aspect of the present invention, a program for causing a computer to execute the picking method of the second aspect is provided.

The outline of the above invention does not list all the features of the present invention. Sub-combinations of these feature groups can also be inventions.

The configuration of the robot according to this embodiment is shown. The configuration of the robot hand is shown. The functional configuration of the picking system according to this embodiment is shown. The configuration of the work separating device is shown. The arrangement of robots and containers in the picking system is shown. Indicates the work housed in the compartment of the container. The result of identifying the work in the container by image recognition is shown. An example of the image and the depth map obtained by the picking system is shown. An example of the work specified from the image and the work part in the depth map cut out by using this as a mask is shown. An example of point cloud data obtained by reconstructing the work part in the depth map is shown. An example of data points acquired to determine the work orientation from the point cloud data is shown. An example of the orientation of the work determined using the three data points is shown. An example of the adsorption point on the work surface determined from the point cloud data is shown. An example of suctioning a workpiece by a robot hand is shown. The flow of the picking method according to this embodiment is shown. An example of the configuration of the computer according to this embodiment is shown.

Hereinafter, the present invention will be described through embodiments of the invention, but the following embodiments do not limit the inventions that fall within the scope of the claims. Also, not all combinations of features described in the embodiments are essential to the means of solving the invention.

1A, 1B, and 1C show the configuration of the robot 101 according to the present embodiment, the detailed configuration of the robot hand 130 of the robot 101, and the functional configuration of the picking system 100 according to the present embodiment, respectively. The picking system 100 is a system that picks (also referred to as picking) a work 90, transports the work 90 and places it at a target location (also referred to as bracing), and is a robot 101, a control device 150, and a work separating device. 200 is provided. The work 90 handled in the present embodiment is, for example, a flexible and amorphous bag-shaped container containing a liquid substance, a powdery substance, or a granular substance. The work 90 cannot be stacked to protect the contents, and is arranged upright in the container compartment. Since it is flexible, the surface can face any direction, and the contents can be oriented when the work 90 is erected. The contents are protected by picking the upper side of the container to concentrate on the lower side of the container.

The robot 101 is a device for picking and placing a work 90, and includes a main body 110, a robot arm 120, and a robot hand 130.

The main body 110 is a portion that supports the robot arm 120 on a pedestal or a floor surface (not shown), and includes a base 111, a driving device 140, and a communication device (not shown). The base 111 rotatably supports the main body 110. As a result, the main body 110 rotates on the upper surface of the base 111 with respect to the central axis parallel to the vertical direction of the drawing and changes its orientation. The drive device 140 may employ, for example, an electric motor, which drives each part of the robot arm 120. As the communication device, for example, a wireless communication device can be adopted, whereby the control signal transmitted from the control device 150 is received and transmitted to the drive device 140, the drive device in the fixing member 134, the suction pad 135, and the like. ..

The robot arm 120 is a portion that drives the robot hand 130, and includes the first to third arms 121 to 123. The first arm 121 is supported by the upper end of the main body 110 and rotates in the left-right direction in the drawing. The second arm 122 is supported by the tip of the first arm 121 and rotates in the left-right direction in the drawing. The third arm 123 supports the robot hand 130 at its tip and is rotatably supported by the tip of the second arm 122.

The robot hand 130 is a portion that grips the work 90, and has a base 131, a shaft 132, an image pickup device 133, a fixing member 134, and a suction pad 135.

The base portion 131 is a portion that supports each portion of the robot hand 130, and is supported by the tip of the third arm 123.

The shaft 132 is a shaft-shaped member that supports the image pickup apparatus 133 and the suction pad 135, and the upper end thereof is supported by the base 131.

As the image pickup apparatus 133, for example, a stereo camera can be adopted. The image pickup apparatus 133 is fixed downward to the shaft 132 via the support member 133a, whereby the work 90 can be imaged to obtain an image thereof and a depth map. The image is an RGB image as an example. The depth map is a map relating to the distance in the depth direction with respect to the position of the image pickup apparatus 133.

The fixing member 134 is fixed to the lower end of the shaft 132 and supports the support member 135a in the horizontal direction. The fixing member 134 has a driving device (not shown) such as an electric motor, whereby the supporting member 135a is rotated.

The suction pad 135 is connected to a compressor (not shown) and sucks and holds the work 90 by sucking air. This makes it possible to pick an amorphous work 90. As an example, one suction pad 135 is fixed to both ends of the support member 135a, and the support member 135a can be rotated by a driving device in the fixing member 134 to change the inclination in the vertical direction. The number of suction pads 135 is not limited to two, and may be any number of one or three or more depending on the size and weight of the work 90.

The suction pad 135 includes a barometric pressure sensor and the like, and measures the internal barometric pressure when air is sucked by the compressor. When the surface of the work 90 is sucked by the suction pad 135 and gripped, a low internal air pressure is measured, and when the work 90 fails to be sucked, a high internal pressure is measured, so that the work is measured for each of the two suction pads 135. It is possible to detect whether the gripping of the 90 is successful or unsuccessful. The result is transmitted to the control unit 154.

The control device 150 is a device that controls the robot 101, and is mounted by an arbitrary computer device. The control device 150 includes a processing device (CPU and / or GPU) and a communication device. The processing device (not shown) causes the control device 150 to express the control function of the robot 101 by executing the control program. The control program is stored in, for example, a ROM (not shown), read by a processing device, and expanded in a RAM to be activated. The communication device (not shown) is, for example, a device that wirelessly communicates with the robot 101. The robot 101 is controlled by transmitting a control signal to the robot 101 by the communication device. The control device 150 expresses the imaging unit 151, the analysis unit 152, the learning unit 153, and the control unit 154.

The imaging unit 151 photographs the work 90 using the imaging device 133. This makes it possible to obtain an image of the work 90 and a depth map (in the depth direction).

The analysis unit 152 identifies the work 90 based on the image, and determines the position and orientation of the work 90 based on the work portion corresponding to the specified work 90 in the depth map. The method of determining the position and orientation of the work 90 will be described later. Here, the analysis unit 152 may specify the work based on the image by machine learning by the learning unit 153.

The learning unit 153 generates a machine learning model adopted by the analysis unit 152 by using the correct image in which the work 90 has been specified. As a machine learning model, for example, deep learning can be adopted. As the learning image, for example, an image of 100 workpieces 90 may be used. As the correct image, an image in which the work 90 is specified may be used, for example, by performing polygon processing on several images from the learning images using arbitrary image processing software. Specifically, using image processing software, vertices are manually created so as to surround the work shown in the image, and polygons are created based on the vertices. As the image processing software for polygon processing, for example, VGG Image Annotator (VIA) can be used. This enables the analysis unit 152 to perform high-speed processing of image recognition of the work 90 that can be deformed in an indefinite form. Machine learning may be performed so as to specify the inner wall and the partition of the container.

The control unit 154 drives and controls the robot arm 120 based on the position and orientation of the work 90 determined by the analysis unit 152. By transmitting a control signal to the robot 101, the control unit 154 changes the direction of the main body 110 by the driving device 140 of the robot 101, extends the robot arm 120 back and forth, and uses the driving device in the fixing member 134 to push the suction pad 135. Tilt up and down to turn the suction pad 135 on and off to pick and place the work 90.

The control unit 154 determines the driving speed of the robot arm 120 according to the number of suction pads 135 sucked on the work 90 among the plurality of suction pads 135. When the number of suction pads 135 adsorbed on the work 90 is large, for example, when both of the two suction pads 135 are adsorbed on the work 90, the driving speed of the robot arm 120 is high, and when the number is small, for example, one of the two suction pads 135. When only the work is adsorbed on the work 90, it is lowered so that the work can be stably conveyed at high speed.

FIG. 2 shows the configuration of the work separating device 200. The work separating device 200 is set with respect to the container 70. Container 70, as an example, the longitudinal direction 4 (drawing oblique direction) 3 and the lateral direction (horizontal direction in the drawing), is partitioned into a total of 12 compartments by a partition 79, the work in the compartment S _a of them 90, the workpiece 91 is accommodated toward the surface lateral direction in the compartment S _b. The work separating device 200 is a device that brings at least the upper portion of the work 90 in one direction to separate it from another work, and includes a set of support drive devices 210 and a rod-shaped member 220.

A set of support device driving devices 210 are devices for driving the rod-shaped member 220 in the lateral direction of the container, and are arranged on one side and the other side in the longitudinal direction of the container 70, respectively. Each support drive device 210 includes a main body 211, a table 212, and a support 213.

The main body 211 supports the table 212 on its upper surface, and drives the table 212 in the uniaxial direction by rotating a ball screw passing through a guide lock fixed to the bottom of the table 212 by an electric motor (not shown).

The table 212 is a member that supports the support 213 on the main body 211 and moves in the uniaxial direction.

The support tool 213 is a member that stands on the table 212 and supports one end of the rod-shaped member 220. The support 213 has a shape that extends upward, bends inward toward the inside of the container 70, and extends downward along the inner wall of the container 70, and one end of the rod-shaped member 220 is fixed to the tip thereof. The tip of the support tool 213 supports the rod-shaped member 220 and moves up and down by a driving device (not shown).

Both ends of the rod-shaped member 220 are fixed to the respective supports 213 of the set of support drive devices 210, and the rod-shaped members 220 are supported above the partition 79 in the container 70. The rod-shaped member 220 may be formed of a material such as metal or plastic, or may be a wire instead.

In the work separating device 200, the rod-shaped member 220 is raised by the support tool 213, the support tool 213 is driven by the main body 211 in the left-right direction in the drawing, the rod-shaped member 220 is lowered and arranged between the works 90 and 91, and is supported by the main body 211. By driving the tool 213 to the left side of the drawing, at least the upper part of the work 90 is moved to the left side of the drawing and separated from the other work 91. As a result, a space can be provided between the works 90 and 91 so that the surface of the work 90 or the work 91 can be imaged, and a space for inserting the suction pad 135 of the robot hand 130 can be secured. ..

The work separating device 200 may not only move the rod-shaped member 220 to move at least the upper part of the work 90 to the left, but also move at least the upper part of the work 91 to the right. As a result, the upper part of the work 90 and the upper part of the work 91 are further separated from each other, so that the possibility that the work 91 is reflected when the work 90 is imaged can be reduced.

FIG. 3 shows the arrangement of the robot 101, the containers 70, 80, and the QR code (registered trademark) reader 60 in the picking system 100 according to the present embodiment. In the top view, the container 70 is arranged on the left side of the robot 101, the container 80 is arranged on the upper side, and the QR code (registered trademark) reader 60 is arranged on the upper left side.

The container 70 is a container for accommodating the works to be rearranged, and the inside thereof is divided into 12 sections 71 as an example by a partition 79. It is assumed that the work is cluttered in 12 compartments 71. The work separating device 200 is installed in the container 70.

The container 80 is a place container, and the inside thereof is divided into 12 sections as an example by a partition 89. Each work is housed in a designated compartment of the 12 compartments 81.

The QR code (registered trademark) reader 60 is a device that reads a QR code (registered trademark). A QR code (registered trademark) that records the contents, the amount of contents, etc. is affixed to the surface of each work. The QR code (registered trademark) of each work is read by the QR code (registered trademark) reader 60, and the section 81 of the container 80 to be accommodated is determined from the record.

The identification of the work by the analysis unit 152 will be described.

FIG. 4A shows the workpieces 91 to 97 housed in the container 70. The inner wall of the container 70 is omitted. In the compartment 71 of the container 70, three workpieces 91, 95, 96 are housed facing the right side of the drawing and leaning against the left inner wall or the partition (not shown) of the container 70 while overlapping the upper parts of each other. .. Here, the work 91 is on top, and the works 95 and 96 are on top of it. In the compartments 72 and 73, one work 92 and 93, respectively, are housed facing downward in the drawing and leaning toward the upper inner wall or partition (not shown) of the container 70 at the upper portion. In the compartment 74, the two workpieces 94 and 97 are housed facing the left side of the drawing and leaning toward the right inner wall or the partition (not shown) of the container 70 while overlapping the upper parts of each other. Here, the work 94 is on top, and the work 97 is on top of it. As an example, the accommodation state of these works 91 to 97 is imaged from above by the imaging unit 151, and as a result, an image and a depth map are obtained.

FIG. 4B shows the identification results of the works 91 to 97 in the container 70 by image recognition. The analysis unit 152 identifies the individual works 91 to 97 by recognizing the images of the works 91 to 97. Specifically, the analysis unit 152 identifies the works 91 to 97 based on the image by using the machine learning model generated from the correct image in which the work has been specified. As a method for specifying, for example, semantic segmentation can be adopted, and in this example, instance segmentation in particular can be adopted. In instance segmentation, each pixel of an image is assigned to an object class, and the shape of each object is captured to identify individual works. In short, the analysis unit 152 identifies the works 91 to 97 by recognizing the pixels occupied by the works 91 to 97 in the image.

In this example, among the works 91 to 93 in the section 71, the work 91 capable of recognizing almost the entire surface 91a is specified. Further, the workpieces 92 and 93 capable of recognizing almost the entire surface 92a and 93a are specified in the compartments 72 and 73. Further, among the works 94 and 97 in the compartment 74, the work 94 capable of recognizing almost the entire surface 94a has been specified. By instance segmentation, a work that can recognize almost the entire surface from above the container 70 can be specified as a work stacked on top. The workpieces 95 to 97 that can recognize only a part of the surface are specified as workpieces stacked underneath, and a sufficiently wide area on the surface that can be attracted and held by the suction pad 135 is exposed upward. Since there is no work, it is excluded from the work to be picked.

Further, the analysis unit 152 may identify the work by Object Contour Detection instead of semantic segmentation.

Further, the analysis unit may identify the work by a rule-based image recognition method instead of machine learning.

It should be noted that it is usually not possible to identify overlapping workpieces from the depth map. As described above, individual works can be identified by image boundary recognition, the target work can be specified from them, and the work portion corresponding to the target work can be specified from the depth map based on the result. ..

The calculation of the position and orientation of the work by the analysis unit 152 will be described.

FIG. 5A shows an example of an image (left figure) and a depth map (right figure) of the work in the container 70 by the imaging unit 151 in the picking system 100. The image may be an RGB image. In this example, one work 91 is housed in one section 71 partitioned by the partition 79 of the container 70, and one work is accommodated in each of the other surrounding sections. The analysis unit 152 applies the above-mentioned image recognition to the image to identify the work 91 and other works in the container 70 from the partition 79 and the like. The analysis unit 152 identifies the work 91 to be picked from the identified works.

FIG. 5B shows an example of the work 91 (left figure) identified from the image and the work portion 91D (right figure) in the depth map. The analysis unit 152 extracts the specified work 91 from the image and uses this as a mask to cut out the corresponding work portion 91D from the depth map.

FIG. 5C shows an example of the point cloud data 91P obtained by reconstructing the work portion 91D in the depth map. The analysis unit 152 converts the depth map into a three-dimensional point cloud based on the position of the image pickup device 133, that is, the position information in the depth direction with reference to the image pickup device 133 into the position information in the three-dimensional space. As a result, the three-dimensional position information of the spatially-referenced work can be obtained from the depth map.

The analysis unit 152 may downsample the point cloud data 91P as appropriate. The sampling rate may be, for example, 1/10. By reducing the amount of data in the point cloud data 91P, it is possible to speed up the analysis of the position and orientation of the work 91. Further, the analysis unit 152 may perform static outliner processing (statistical outlier processing) on the point cloud data 91P. By removing the outliers from the point cloud data 91P, the position and orientation of the work 91 can be accurately determined. The depth map before reconstruction may be subjected to processing for downsampling and removing outliers.

FIG. 5D shows an example of data points P1 to P3 acquired from the point cloud data 91P to determine the orientation of the work 91. First, the analysis unit 152 determines a rectangular area S including all or most of the point cloud data 91P reconstructed from the work portion 91D. Then, the analysis unit 152, three rectangular zones S vertically, horizontally three, is divided into a total of nine areas _S 1 to S _9. Here, it may be divided at equal intervals in the vertical direction or at equal intervals in the horizontal direction. However, it is assumed that determined greater than the respective minimum width of at least the center of the area S ₅ vertical and horizontal directions of width. Then, the analysis unit 152 selects the three zones which are not adjacent to each other nine areas _S 1 to S _9. As an example, to select an area _{S 1, S 3, S 8} in the reverse triangular arrangement. Alternatively, the area _{S 2, S 7, S 9} in triangular arrangement, the area _S 3 in the left triangular _arrangement, S 4, _{S 9,} may select an area _{S 1, S 6, S 7} in right triangular arrangement. Then, the analysis unit 152 obtains the three zones _{S 1,} S 3, each one data point P1~P3 from _{S 8.} The data points P1 to P3 may be randomly selected from the data points in each area, or the closest data point may be selected from the predetermined points for each area.

FIG. 5E shows an example of the orientation of the work 91 determined by using the three data points P1 to P3. The analysis unit 152 calculates the normal vector Vn of the triangular area T defined by the three data points P1 to P3 using the three selected data points P1 to P3. The orientation of the surface of the work 91 is determined from the normal vector Vn.

Thus, the orientation of the workpiece 91, i.e. the three data points P1~P3 for calculating the normal vector Vn, three zones which are not adjacent to each other among the area _S 1 to S ₉ that 9 divides the work portion 91D By separating the data points P1 to P3 from the data points P1 to P3, it is possible to determine the overall orientation of the surface of the work 91 without being affected by local unevenness in the case of the flexible work 91.

The analysis unit 152 may change the three areas when the data points P1 to P3 for calculating the normal vector cannot be acquired. In such a case, only the area where the decision point could not be obtained may be changed to another area. However, the three areas shall not be adjacent to each other. Alternatively, the area _{S 1,} S 3, _{S 8} in the reverse triangular arrangement, area _{S 2,} S 7, _{S 9} in triangular arrangement, the area _S 3 in the left triangular _arrangement, S 4, _{S 9,} areas in the right triangle configuration S _It may be changed to any of 1, S ₆ and S _7. As a result, for example, even if the depth map cannot be partially obtained for the irregular work 91 and the decision point cannot be obtained, the decision point can be obtained by changing the area to obtain the decision point on the surface of the work. It is possible to determine the overall orientation.

Further, the analysis unit 152 may divide the rectangular area S into 9 substantially evenly. By dividing the rectangular area S into 9 substantially evenly, a triangular area T having a size larger than a predetermined value can be obtained, and a highly robust normal vector can be calculated.

In the above description, the analysis unit 152 divides the rectangular area S into a total of nine areas S _{1 to} S ₉ , but the number of divisions of the rectangular area S is not limited to nine. The analysis unit 152 may divide the work portion 91D into seven or more sections.

FIG. 5F shows an example of the suction point (that is, the position determination point) Pa on the work surface determined from the point cloud data 91P. The analysis unit 152 determines the direction along the surface of the work 91 (the direction in which the vertical downward component of the unit length vector that defines the direction in the plane of the triangular area T is substantially maximum) based on the orientation of the work 91. Then, the suction point Pa (this position is called the position of the work 91) in the work portion 91D is determined based on the direction. More specifically, the tangent vector Vt is determined from the normal vector Vn so as to determine the suction point Pa on the upper side of the work 91 where the contents are not concentrated according to the size of the suction pad 135, and the upper end of the rectangular area S is determined. The position of the distance d downward along the tangent vector Vt from the center is defined as the suction point Pa. Here, the distance d is set to be larger than the size of the suction pad 135, for example, 3 cm.

FIG. 6 shows an example of suction of the work 91 by the robot hand 130 (suction pad 135). The control unit 154 receives the calculation result of the position and orientation of the work 91 from the analysis unit 152, controls the robot arm 120, and sets the direction of the suction pad 135 to the direction of the work 91 (that is, the direction of the normal vector Vn). Together, the suction pad 135 is driven toward the suction point Pa on the surface of the work 91. By matching the orientation of the suction pad 135 with the orientation of the work 91, it is possible to suck the work surface and pick it stably.

FIG. 7 shows the flow of the picking method according to the present embodiment.

In step S102, the control unit 154 moves the robot hand 130 above the container 70. Alternatively, it may move above one of the plurality of compartments 71 in the container 70.

In step S104, the imaging unit 151 photographs the work in the container 70 to obtain an image and a depth map.

In step S106, the control unit 154 analyzes the image and determines whether or not the work 90 can be seen, that is, whether or not the work 90 is included in the image. If the work 90 is visible, proceed to the next step. If you cannot see it, end the flow. Alternatively, returning to step S102, the control unit 154 may move the robot hand 130 above the next compartment 71 of the plurality of compartments 71 in the container 70.

Although the work 90 can be recognized, for example, when it is determined that a sufficiently wide surface area that can be sucked and held by the suction pad 135 is not exposed upward because it overlaps with another work 91. The control unit 154 may separate the work 90 from the other work 91 by moving at least the upper portion of the work 90 in one direction by the work separating device 200. As a result, a space can be provided between the work 91 and the surface of the work 90, and a space for inserting the suction pad 135 of the robot hand 130 can be secured. After separating the work, the process returns to step S104.

In step S108, the analysis unit 152 identifies the work 90 based on the image, and calculates the position and orientation of the work 90 based on the work portion 91D corresponding to the specified work 90 in the depth map. Details are as described above.

As described with reference to FIGS. 4A and 4B, the analysis unit 152 identifies the individual works 91 to 97 by performing image recognition (particularly, boundary recognition) on the images of the works 91 to 97. Next, the analysis unit 152 identifies the work 91 to be picked from the works identified in the image of FIG. 5A (left figure). Next, the analysis unit 152 extracts the specified work 91 from the image of FIG. 5B (left figure), uses this as a mask, and cuts out the corresponding work portion 91D from the depth map (right figure). Next, as shown in FIG. 5C, the analysis unit 152 converts the depth map (work portion 91D) into a three-dimensional point cloud based on the position of the image pickup apparatus 133. Here, the analysis unit 152 may downsample the point cloud data 91P as appropriate. Further, the analysis unit 152 may perform static outliner processing (statistical outlier processing) on the point cloud data 91P. In addition, downsampling and outlier processing may be performed on the depth map before reconstruction.

Next, as shown in FIG. 5D, the analysis unit 152 determines a rectangular area S including all or most of the point cloud data 91P reconstructed from the work portion 91D, and three rectangular areas S up and down. left three, divided into a total of nine areas _S 1 to S _9, three areas from among those not adjacent to one another, for example, three zones _{S 1,} S 3, each one data point P1~P3 from _{S 8} get. Next, as shown in FIG. 5E, the analysis unit 152 calculates the normal vector Vn of the triangular area T defined by the three data points P1 to P3 using the three selected data points P1 to P3. The orientation of the surface of the work 91 is determined from the normal vector Vn. The analysis unit 152 may change the three areas when the data points P1 to P3 for calculating the normal vector Vn cannot be acquired. Next, as shown in FIG. 5F, the analysis unit 152 performs a vertical downward component of a vector having a unit length that defines the direction along the surface of the work 91 (the direction in the plane of the triangular area T) based on the direction of the work 91. The direction in which is substantially maximum) is determined, and the suction point Pa (this position is referred to as the position of the work 91) in the work portion 91D is determined based on that direction.

In step S110, the control unit 154 moves the robot hand 130 above the target work 91 based on the position of the work 91 calculated in step S108.

In step S112, the control unit 154 turns on the suction pad 135.

In step S114, as shown in FIG. 6, the control unit 154 controls the robot arm 120 based on the position and orientation of the work 91 calculated in step S108, and sets the orientation of the suction pad 135 to the orientation of the work 91. The suction pad 135 is driven toward the suction point Pa on the surface of the work 91 in accordance with (that is, the direction of the normal vector Vn). By aligning the direction of the suction pad 135 with the direction of the work 91, the surface of the work is sucked and picked stably.

In step S116, the control unit 154 determines whether or not the suction pad 135 has succeeded in sucking the work 91. If successful, the process proceeds to step S118. If it fails, the process returns to step S104. If it fails a plurality of times, the flow may be terminated.

In step S118, the control unit 154 determines the number of suction pads 135 sucked on the work 91 among the plurality of suction pads 135. When both of the two suction pads 135 are sucked to the work 91, the process proceeds to step S120, and the control unit 154 increases the drive speed of the robot arm 120 to convey the work 91 at high speed, and only one of the two suction pads 135 is sucked. When is attracted to the work 90, the process proceeds to step S122, the driving speed of the robot arm 120 is lowered, and the work 91 is stably conveyed. Since the work 91 is flexible, the work 91 can be stably conveyed even when suction by one of the suction pads 135 fails.

In step S124, the control unit 154 conveys the work 91 to the QR code (registered trademark) reader 60, and uses this to read the QR code (registered trademark) attached to the surface of the work. The control unit 154 determines the compartment 81 of the container 80 to be accommodated from the record by the QR code (registered trademark).

In step S126, the control unit 154 conveys the work 91 to the target section 81.

In step S128, the control unit 154 turns off the suction pad 135 and places the work 91 in the compartment 81.

The control unit 154 returns to step S102 and repeats the above steps. The control unit 154 transports all the works in the container 70 to the container 80, and ends the flow when the works cannot be confirmed in the container 70 in step S106.

Prior to the flow of the picking method described above, the learning unit 153 generates a machine learning model using the correct image in which the work has been specified. However, the analysis unit 152 may specify the work based on the image by a rule-based image recognition method instead of machine learning.

As described above, the picking system 100 according to the present embodiment corresponds to the imaging unit 151 that captures the work and obtains an image and a depth map, specifies the work based on the image, and corresponds to the specified work in the depth map. It includes an analysis unit 152 that determines the position and orientation of the work based on the work portion, and a control unit 154 that drives and controls the robot arm 120 based on the position and orientation of the work. The analysis unit 152 identifies the work by image recognition, determines the position and orientation of the work based on the portion of the depth map corresponding to the specified work, and controls the robot arm 120 by the control unit 154 to control the work. By driving the work to that position in accordance with the direction of the robot arm 120, even an irregular workpiece can be stably picked by the robot arm 120.

In the picking system 100 according to the present embodiment, the suction pad 135 is used for the robot hand 130 for holding the work, but the present invention is not limited to this, and a hand having a configuration for sandwiching the upper part of the work may be used.

Further, in the picking system 100 according to the present embodiment, the control device 150 is configured to include the learning unit 153, but the configuration of the control device 150 is not limited to this. The control device 150 may be a device that stores a machine learning model that has been learned in advance, as long as the analysis unit 152 can identify the work. In this case, the control device 150 does not include the learning unit 153. Further, the learning unit 153 may exist as another device and may be connected via a network.

Various embodiments of the present invention may be described with reference to flowcharts and block diagrams, wherein the block is (1) a stage of the process in which the operation is performed or (2) a device responsible for performing the operation. May represent a section of. Specific stages and sections are implemented by dedicated circuits, programmable circuits supplied with computer-readable instructions stored on a computer-readable medium, and / or processors supplied with computer-readable instructions stored on a computer-readable medium. You can. Dedicated circuits may include digital and / or analog hardware circuits, and may include integrated circuits (ICs) and / or discrete circuits. Programmable circuits are memory elements such as logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, flip-flops, registers, field programmable gate arrays (FPGA), programmable logic arrays (PLA), etc. May include reconfigurable hardware circuits, including, etc.

The computer readable medium may include any tangible device capable of storing instructions executed by the appropriate device, so that the computer readable medium having the instructions stored therein is specified in a flowchart or block diagram. It will be equipped with a product that contains instructions that can be executed to create means for performing the operation. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media include floppy (registered trademark) disks, diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), Electrically erasable programmable read-only memory (EEPROM), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), Blu-ray (RTM) disc, memory stick, integrated A circuit card or the like may be included.

Computer-readable instructions are assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object-oriented programming such as Smalltalk, JAVA®, C ++, etc. Contains either source code or object code written in any combination of one or more programming languages, including languages and traditional procedural programming languages such as the "C" programming language or similar programming languages. good.

Computer-readable instructions are applied locally to a general purpose computer, a special purpose computer, or the processor or programmable circuit of another programmable data processing unit, or to a wide area network (WAN) such as the local area network (LAN), the Internet, etc. ) May be executed to create a means for performing the operation specified in the flowchart or block diagram. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers and the like.

FIG. 8 shows an example of a computer 2200 in which a plurality of aspects of the present invention may be embodied in whole or in part. The program installed on the computer 2200 can cause the computer 2200 to function as an operation associated with the device according to an embodiment of the present invention or as one or more sections of the device, or the operation or the one or more. Sections can be run and / or the computer 2200 can be run a process according to an embodiment of the invention or a stage of such process. Such a program may be run by the CPU 2212 to cause the computer 2200 to perform certain operations associated with some or all of the blocks in the flowcharts and block diagrams described herein.

The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are interconnected by a host controller 2210. The computer 2200 also includes input / output units such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via the input / output controller 2220. There is. The computer also includes legacy input / output units such as the ROM 2230 and keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

The CPU 2212 operates according to the programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphic controller 2216 acquires the image data generated by the CPU 2212 in a frame buffer or the like provided in the RAM 2214 or itself so that the image data is displayed on the display device 2218.

The communication interface 2222 communicates with other electronic devices via the network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads the program or data from the DVD-ROM 2201 and provides the program or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from the IC card and / or writes programs and data to the IC card.

The ROM 2230 contains a boot program or the like executed by the computer 2200 at the time of activation, and / or a program depending on the hardware of the computer 2200. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via a parallel port, serial port, keyboard port, mouse port, and the like.

The program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from a computer-readable medium, installed on a hard disk drive 2224, RAM 2214, or ROM 2230, which is also an example of a computer-readable medium, and executed by the CPU 2212. The information processing described in these programs is read by the computer 2200 and provides a link between the program and the various types of hardware resources described above. The device or method may be configured to perform manipulation or processing of information in accordance with the use of computer 2200.

For example, when communication is executed between the computer 2200 and an external device, the CPU 2212 executes a communication program loaded in the RAM 2214, and performs communication processing on the communication interface 2222 based on the processing described in the communication program. You may order. Under the control of the CPU 2212, the communication interface 2222 reads and reads transmission data stored in a transmission buffer processing area provided in a recording medium such as a RAM 2214, a hard disk drive 2224, a DVD-ROM 2201, or an IC card. The data is transmitted to the network, or the received data received from the network is written to the reception buffer processing area or the like provided on the recording medium.

Further, the CPU 2212 causes the RAM 2214 to read all or necessary parts of a file or database stored in an external recording medium such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM2201), or an IC card. Various types of processing may be performed on the data on the RAM 2214. The CPU 2212 then writes back the processed data to an external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored on the recording medium and processed. The CPU 2212 describes various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, and information retrieval described in various parts of the present disclosure with respect to the data read from the RAM 2214. Various types of processing may be performed, including / replacement, etc., and the results are written back to RAM 2214. Further, the CPU 2212 may search for information in a file, a database, or the like in the recording medium. For example, when a plurality of entries each having an attribute value of the first attribute associated with the attribute value of the second attribute are stored in the recording medium, the CPU 2212 specifies the attribute value of the first attribute. Search for an entry that matches the condition from the plurality of entries, read the attribute value of the second attribute stored in the entry, and associate it with the first attribute that satisfies the predetermined condition. The attribute value of the second attribute obtained may be acquired.

The program or software module described above may be stored on or near a computer 2200 on a computer-readable medium. Also, a recording medium such as a hard disk or RAM provided in a dedicated communication network or a server system connected to the Internet can be used as a computer readable medium, thereby providing the program to the computer 2200 over the network. do.

Although the present invention has been described above using the embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that such modified or improved forms may also be included in the technical scope of the present invention.

The order of execution of operations, procedures, steps, steps, etc. in the devices, systems, programs, and methods shown in the claims, specification, and drawings is particularly "before" and "prior to". It should be noted that it can be realized in any order unless the output of the previous process is used in the subsequent process. Even if the scope of claims, the specification, and the operation flow in the drawings are explained using "first", "next", etc. for convenience, it means that it is essential to carry out in this order. It's not a thing.

60 ... QR code (registered trademark) reader, 70, 80 ... container, 71-74 ... compartment, 79 ... partition, 80 ... container, 81 ... compartment, 89 ... partition, 90-97 ... work, 91a-94a ... surface, 91D ... Work part, 91P ... Point group data, 100 ... Picking system, 101 ... Robot, 110 ... Main body, 111 ... Base, 120 ... Robot arm, 130 ... Robot hand, 131 ... Base, 132 ... Shaft, 133 ... Imaging device , 133a ... Support member, 134 ... Fixing member, 135 ... Suction pad, 135a ... Support member, 140 ... Drive device, 150 ... Control device, 151 ... Imaging unit, 152 ... Analysis unit, 153 ... Learning unit, 154 ... Control unit , 200 ... Work separation device, 210 ... Support drive device, 211 ... Main body, 212 ... Table, 213 ... Support, 220 ... Rod-shaped member, 2200 ... Computer, 2201 ... DVD-ROM, 2210 ... Host controller, 2214 ... RAM , 2216 ... Graphic controller, 2218 ... Display device, 2220 ... Input / output controller, 2222 ... Communication interface, 2224 ... Hard disk drive, 2226 ... DVD-ROM drive, 2240 ... Input / output chip, 2242 ... Keyboard, P1 to P3 ... Data point, Pa ... adsorption point (position determination point), S ... rectangular area, S _{1 to} S ₉ ... area, S _a , S _b ... section, T ... triangular area, Vn ... normal vector, Vt ... tangent vector.

Claims (18)

It is a picking system that picks workpieces.
An imaging unit that captures a workpiece to obtain an image and a depth map,
It is an analysis unit that identifies the work based on the image and determines the position and orientation of the work based on the work portion corresponding to the specified work in the depth map, and divides the work portion into a plurality of sections. The normal vector of the surface of the work is divided into three sections, which are not adjacent to each other, and a triangular area having a size equal to or larger than a predetermined value defined by three points obtained from each of the three sections is used. The analysis unit and the analysis unit, which determine the orientation by calculating
A control unit that drives and controls the robot arm based on the position and orientation of the work,
A picking system equipped with. The picking system according to claim 1, wherein the analysis unit divides the work portion into seven or more sections. The picking system according to claim 1 or 2, wherein the analysis unit changes the three sections when a point for calculating a normal vector on the surface of the work cannot be obtained. The picking according to any one of claims 1 to 3, wherein the analysis unit analyzes a region on the surface of the work based on the image and identifies the work on the front side with respect to the imaging device of the imaging unit. system. The picking system according to any one of claims 1 to 4, wherein the analysis unit reconstructs the depth map into a three-dimensional point cloud based on the position of the imaging device of the imaging unit. The picking system according to any one of claims 1 to 5, wherein the work is a bag-shaped container. The analysis unit determines a direction along the surface of the work, determines a position-determining point in the work portion based on the direction, and determines the position of the work from the position of the position-determining point. The picking system according to any one of items 1 to 6. The picking system according to any one of claims 1 to 7, wherein the analysis unit identifies the work based on the image by using a machine learning model generated from a correct image in which the work has been specified. The picking system according to any one of claims 1 to 8, wherein the robot arm has at least one suction pad. The picking system according to claim 9, wherein the control unit aligns the direction of the suction pad with the direction of the work when driving and controlling the robot arm. The robot arm has a plurality of suction pads and has a plurality of suction pads.
The picking system according to claim 9 or 10, wherein the control unit determines the driving speed of the robot arm according to the number of suction pads sucked on the work among the plurality of suction pads. The picking system according to any one of claims 1 to 11, further comprising a work separating device for moving at least the upper portion of the work in one direction to separate it from another work. The picking system according to any one of claims 1 to 12, wherein the imaging unit photographs the work using an imaging device attached to the robot arm. It is a picking system that picks workpieces.
An imaging unit that captures a workpiece to obtain an image and a depth map,
Identifying the work based on the image, and an analysis unit for determining the position and orientation of the workpiece on the basis of the work portion corresponding to the specified work of the depth map, the vertical and the central region The work portion is divided into nine parts vertically and horizontally so that the width in the left-right direction is larger than the minimum width, three areas that are not adjacent to each other are selected, and the work is used by using three points obtained from each of the three areas. The analysis unit, which determines the direction by calculating the normal vector of the surface of the
A control unit that drives and controls the robot arm based on the position and orientation of the work,
With
The imaging unit is a picking system that photographs the work using an imaging device attached to the robot arm. It is a picking method for picking a work.
At the stage where the work is photographed by the imaging unit to obtain an image and a depth map,
The analysis unit identifies the work based on the image, and determines the position and orientation of the work based on the work portion corresponding to the specified work in the depth map. Divide into a plurality of sections, select three sections that are not adjacent to each other, and use a triangular area having a size equal to or larger than a predetermined value defined by three points obtained from each of the three sections on the surface of the work. The step of determining the direction by calculating the normal vector, and the step of determining the direction,
A stage in which the control unit drives and controls the robot arm based on the position and orientation of the work, and
A picking method that includes. It is a picking method for picking a work.
At the stage where the work is photographed by the imaging unit to obtain an image and a depth map,
The analysis unit identifies the work based on the image, and determines the position and orientation of the work based on the work portion corresponding to the specified work in the depth map . The work portion is divided into nine parts vertically and horizontally so that the widths in the vertical and horizontal directions are larger than the minimum width , three areas that are not adjacent to each other are selected, and three points obtained from each of the three areas are used. The direction is determined by calculating the normal vector of the surface of the work, and the determination step and
A stage in which the control unit drives and controls the robot arm based on the position and orientation of the work, and
A picking method that includes. The work is a bag-shaped container,
The picking method according to claim 16. A program that causes a computer to execute the picking method according to claim 16 or 17.

Images