JP7136554B2 - Grasping device, learning device, program, grasping system, and learning method - Google Patents

Description

The present invention relates to a grasping device, a learning device, a trained model, a grasping system, a determination method, and a learning method.

Techniques for gripping an object using machine learning are known. For example, in Non-Patent Document 1, a gripping device generates gripping pose candidates such that a gripper and an object point group are in contact and the object point group is sufficiently between the grippers. Then, this grasping apparatus uses a CNN (Convolutional Neural Network) to determine whether the generated grasping posture candidate is suitable for grasping.

Marcus Gualtieri, Andreas ten Pas, Kate Saenko, and Robert Platt, “High precision grasp pose detection in dense clutter”, arXiv preprint arXiv:1603.01564, 2016.

When machine learning using image data is used to determine the gripping position of a folded and stacked flexible object, it may be difficult to determine the appropriate gripping position. The inventors have found that a regular lamination pattern shape appears on the side surface of the soft object in the above state, and there is no sufficient difference between the image of the part suitable for gripping and the image of the part not suitable for gripping. It was found that it is difficult to determine the gripping position.

One aspect of the present invention for achieving the above object is an image data acquisition unit that acquires image data of an image of an object, a grip unit that grips the object, a control unit that controls the operation of the grip unit, and Using the image data obtained by the image data acquisition unit, in which the flexible object in a folded and stacked state is imaged, whether or not the part of the flexible object shown in the image data is suitable for gripping. a gripping position determination unit that determines whether the top surface of the flexible object on the uppermost stage is deformed by the gripping position determination unit; The gripping device determines whether or not the flexible object is suitable for gripping, using the image data obtained by imaging the soft object with the deformed end of the soft object.
In this gripping device, the edge of the upper surface of the flexible object is deformed by the gripping portion. Since the mode of deformation differs between the ends that are combined and the ends that are not such ends, there is a sufficient difference between the image of the part suitable for grasping and the image of the part not suitable for grasping. Therefore, according to this gripping device, it is possible to determine an appropriate gripping position.

In the above aspect, the image data obtained by the image data obtaining unit, in which the soft object is imaged before the edge of the upper surface is deformed, is used for the deformation of the edge of the upper surface. A contact position determination unit that determines a contact position of the flexible object with the grip portion, and the control unit controls the contact position determined by the contact position determination unit to deform the end of the upper surface. , the grip may be controlled to displace the flexible object.
According to such a configuration, it is possible to autonomously perform control for deforming the edge of the upper surface of the flexible object.

In the aspect described above, the control section may control the gripping section such that the gripping section pushes the soft object, thereby deforming the edge of the upper surface.
According to such a configuration, the end of the upper surface can be easily deformed.

Further, one aspect of the present invention for achieving the above object is a training data acquisition unit that acquires training data including image data of an object, and a machine using the training data acquired by the training data acquisition unit. a learning calculation unit that performs a learning calculation, wherein the image data is a flexible object in a folded and stacked state, and the edge of the upper surface of the flexible object is deformed from a regular lamination pattern shape. including first image data and second image data in which a flexible object is shown, wherein the first image data is image data in which the edge of the upper surface is an edge gathered into one by folding, The second image data is image data in which the edge of the upper surface is an edge that is not combined into one by folding, and the learning calculation unit uses the training data to obtain an image suitable for gripping a soft object. It is a learning device that performs calculations for learning body parts.
In this learning device, learning is performed using an image of a flexible object deformed from a regular lamination pattern shape. Since the mode of deformation is different between the edge that is combined into one and the edge that is not such an edge, the first image data and the second image data that have a sufficient difference for use in learning are used. It is possible to learn Therefore, according to this learning device, it is possible to learn an appropriate gripping position.

Further, one aspect of the present invention for achieving the above object is a computer for functioning to output a quantified value of the propriety of gripping a part of an object based on image data obtained by imaging the object. A trained model, wherein weighting coefficients of a neural network that constitutes the trained model are learned using training data including image data of an image of an object, and the image data included in the training data are: First image data and second image data showing a flexible object in a state of being folded and stacked, wherein the edge of the upper surface of the flexible object is deformed from a regular lamination pattern shape. wherein the first image data is image data in which the edge of the top surface is an edge folded into one, and the second image data is image data in which the edge of the top surface is folded into one and the trained model is input to the input layer of the neural network. It is a learned model that causes a computer to function so as to execute calculations according to the image data and output from the output layer of the neural network whether or not the part of the soft object shown in the image data is suitable for gripping.
This model is trained using images of flexible objects deformed from regular lamination patterns. Since the mode of deformation is different between the edge that is combined into one and the edge that is not such an edge, there is a sufficient difference between the first image data and the second image data for use in learning. be. Therefore, according to this trained model, it is possible to determine an appropriate gripping position.

Further, one aspect of the present invention for achieving the above objects includes an imaging device that captures an image of a surrounding object, an image data acquisition unit that acquires image data from the imaging device, a gripping unit that grips the object, and Using the image data acquired by the image data acquiring unit and obtained by the image data acquiring unit, the flexible object shown in the image data is obtained by using the a gripping position determination unit that determines whether or not the portion of is suitable for gripping, wherein the control unit controls the gripping unit so as to deform an end of the upper surface of the flexible object; The gripping position determination unit is a gripping system that determines whether or not the flexible object is suitable for gripping, using the image data obtained by imaging the soft object with the deformed end of the upper surface.
In this gripping system, the edges of the top surface of the flexible object are deformed by the gripping part. Since the mode of deformation differs between the ends that are combined and the ends that are not such ends, there is a sufficient difference between the image of the part suitable for grasping and the image of the part not suitable for grasping. Therefore, according to this gripping system, it is possible to determine an appropriate gripping position.

Further, one aspect of the present invention for achieving the above object is to control the gripping section so as to deform the ends of the upper surfaces of a plurality of folded and stacked flexible objects, and to provide the above-described soft objects whose ends have been deformed. This is a determination method for determining whether or not a part of the soft object shown in the image data is suitable for gripping, using image data obtained by imaging the soft object.
In this determination method, the end of the upper surface of the flexible object is deformed by the gripping portion. Since the mode of deformation differs between the ends that are combined and the ends that are not such ends, there is a sufficient difference between the image of the part suitable for grasping and the image of the part not suitable for grasping. Therefore, according to this determination method, it is possible to determine an appropriate gripping position.

Further, according to one aspect of the present invention for achieving the above object, training data including image data of an object is acquired, and a part of a flexible object suitable for gripping is determined using the acquired training data. The image data is a flexible object in a folded and stacked state, and the flexible object in a state where the edge of the upper surface of the flexible object is deformed from the regular lamination pattern shape. wherein the first image data is image data in which the edge of the upper surface is an edge that is folded into one; and the second image The data is a learning method in which the edges of the top surface are image data that are edges that have not been folded together.
In this learning method, learning is performed using an image of a flexible object deformed from a regular lamination pattern shape. Since the mode of deformation is different between the edge that is combined into one and the edge that is not such an edge, the first image data and the second image data that have a sufficient difference for use in learning are used. It is possible to learn Therefore, according to this learning method, it is possible to learn an appropriate gripping position.

According to the present invention, there are provided a gripping device, a learning device, a trained model, a gripping system, a determination method, and a learning method that can determine an appropriate gripping position for a folded and stacked flexible object. can be done.

FIG. 3 is a schematic diagram showing an example of the appearance of a folded flexible object; FIG. 10 is a diagram showing flexible objects in a folded and stacked state; FIG. 10 is a diagram showing flexible objects in a folded and stacked state; FIG. 10 is a diagram showing flexible objects in a folded and stacked state; 1 is a block diagram showing a schematic configuration of a grasping device according to an embodiment; FIG. 2 is a block diagram showing a schematic configuration of a robot arm; FIG. 6 is a flow chart showing the flow of learning processing by the gripping device according to the embodiment. 4 is a flow chart showing the flow of processing when the gripping device according to the embodiment grips a gripping target. FIG. 7 is a flowchart showing details of S21 (determination of contact position) shown in FIG. 6; FIG. FIG. 10 is a diagram showing how a flexible object is pushed by an end effector; 1 is a schematic diagram showing a gripping system including an imaging device, a server, and a gripping device; FIG.

The gripping device 1 according to the embodiment grips soft objects such as cloth products and paper products that easily change their shape. In particular, the gripping device 1 grips the flexible object stacked on the uppermost level among the plurality of folded and stacked flexible objects. At this time, depending on the part gripped by the gripping device 1, the appearance of the flexible object changes from the folded state to the unfolded state. In other words, if the gripping position is not appropriate, the folded state cannot be lifted.

FIG. 1 is a schematic diagram showing an example of the appearance of a folded soft object 90. As shown in FIG. Note that FIG. 1 shows the appearance of a flexible object 90 folded in four as an example. As shown in FIG. 1, the edges of the four sides of flexible object 90 have different characteristics. Edge 91 is the edge that is folded together, while the other edge is the edge that is not folded together. In other words, the edge 91 is the edge where one crease appears in appearance, whereas the other edge is the edge where a plurality of creases appear or no crease appears. In other words, the edge 91 is the edge where one edge appears in appearance, while the other edge has two or more edges (in the case of the four-fold as in FIG. 1, specifically two or four edges). is the edge where appears. In other words, end 91 is a closed end while the ends other than end 91 are open ends.

When the gripping position is near the edge 91, it is relatively easy to lift the flexible object 90 while maintaining the folded state. It is relatively difficult to maintain and lift flexible object 90 . For example, when an end effector is inserted from the lateral side of the flexible object 90 to grip the upper and lower sides of the folded flexible object, if the gripping position is near the end 91, the flexible object is folded into a plurality of sheets. Although it is easy to grasp the entire sheet group, if the grasping position is near an end other than the end 91, there is a risk that only a part of the sheet group will be grasped. If only a part of the sheet group is grabbed and lifted, the folding will unfold. Thus, in order to lift the flexible object while maintaining the folded state, it is necessary to select an appropriate gripping position. In the present embodiment, machine learning using images is used to identify whether or not the part of the soft object shown in the image is at an appropriate grip position, that is, near the end that is folded into one. When performing identification by such a method, there are the following problems.

2A-2C are diagrams showing flexible objects in a folded and stacked state. 2A-2C specifically show how a plurality of folded cloth towels 92 are stacked. Here, in FIG. 2A, markings 92a indicate the ends folded together, and in FIGS. end) are shown. A similar regular lamination pattern appears in all of FIGS. 2A-2C. Therefore, it is difficult to distinguish between the edge shown in FIG. 2A and the edge shown in FIG. 2B or 2C. Therefore, it is difficult to recognize an appropriate gripping position. In other words, it is not easy for the gripping device to lift the flexible object stacked on the top while maintaining the folded state. Therefore, in the present embodiment, the following technique for lifting a folded flexible object while maintaining the folded state is disclosed. It should be noted that in the following embodiments, the flexible object is folded to have a united end. For example, the flexible object may be folded in half by repeating one or more folds such as two folds, four folds, and eight folds. In this case, when the n+1 fold is made, the fold may be perpendicular to or parallel to the fold made in the n fold. It should be noted that the method of folding the soft object described here is merely an example, and the present invention is not limited to these.

Embodiments will be described below with reference to the drawings.
FIG. 3 is a block diagram showing a schematic configuration of the gripping device 1 according to the embodiment. The gripping device 1 has the configuration shown in FIG. and a judgment process for judging about. When grasping the grasping device 1 as a device that performs learning processing, the grasping device 1 can also be referred to as a learning device.

The gripping device 1 includes an imaging device 10, a robot arm 11, a control unit 12, a training data acquisition unit 13, a learning calculation unit 14, a model storage unit 15, an image data acquisition unit 16, and a contact position determination unit. 17 and a gripping position determination unit 18 .

The imaging device 10 is a device that captures an image of a surrounding object, and more specifically, a device that generates three-dimensional image information in the present embodiment. Any configuration can be adopted as a specific configuration of the imaging device 10 . For example, the imaging device 10 may be a stereo camera. In addition, the imaging device 10 may include a time-of-flight distance sensor such as a laser range finder, or a light coding distance sensor that measures distance by projecting a dot pattern. may As described above, the specific configuration of the imaging device 10 is arbitrary, and is not limited to a configuration based on any technology.

The robot arm 11 is a specific example of a gripper that grips an object. As shown in FIG. 4, the robot arm 11 includes, for example, a plurality of links 111, joints (wrist joints, elbow joints, shoulder joints, etc.) 112 that rotatably connect the links 111, and gripping objects. and an end effector 113 for

Each joint portion 112 includes a rotation sensor 114 such as an encoder for detecting rotation information of each joint portion 112, an actuator 115 such as a servomotor for driving each joint portion 112, and an operation force of each joint portion 112. A force sensor 116 is provided. The force sensor 116 is, for example, a torque sensor that detects torque of each joint 112 . Each joint 112 is provided with a reduction mechanism. The end effector 113 is composed of, for example, a plurality of finger portions, and can grip an object by pinching it with the finger portions. The end effector 113 is provided with an actuator 117 that drives the end effector 113 and a force sensor 118 that detects the operating force of the end effector 113 .

The control unit 12 controls the motion of the robot arm 11 . The control unit 12 calculates the trajectory of the robot arm 11 based on the processing result by the contact position determination unit 17 or the processing result by the gripping position determination unit 18 . Also, the control unit 12 moves the robot arm 11 according to the calculated trajectory. Specifically, for example, the control unit 12 feedback-controls the robot arm 11 by controlling the actuators 115 and 117 based on the outputs from the rotation sensor 114 and the force sensors 116 and 118 .

The training data acquisition unit 13 acquires training data including image data obtained by imaging an object. The training data acquisition unit 13 may read and acquire training data stored in advance in a storage device such as a memory of the gripping device 1, or may acquire training data transmitted from another device. The learning calculation unit 14 performs machine learning calculations using the training data acquired by the training data acquisition unit 13 . The image data included in the training data is, for example, two-dimensional image data, but may be three-dimensional image data.

Here, the training data used in this embodiment will be described. In the present embodiment, the image data included in the training data is image data showing flexible objects in a folded and stacked state. This is image data showing a flexible object in a state where it is stretched. More specifically, in the present embodiment, image data is used in which the edge of the upper surface of the topmost flexible object is pushed from the side. Image data used as training data includes the following two types of image data as such image data. One of these two types of image data is called first image data, and the other is called second image data.

The first image data is image data in which the deformed edges being imaged are edges that have been folded together. In other words, the first image data is image data in which deformation due to pressure from the side direction is occurring at the end that has been combined into one by folding. In the training data, the first image data is associated with a label indicating that it is an image of a region suitable for grasping. The second image data is image data in which the deformed edges being imaged are edges that are not brought together by folding. In other words, the second image data is image data in which deformation due to pressure from the side direction occurs at the ends other than the ends that are combined into one by folding. In the training data, the second image data is associated with a label indicating that it is an image of a part unsuitable for grasping. Note that the first image data and the second data may be partial image data cut out from the image data.

The learning calculation unit 14 uses such training data to perform calculations for learning parts of a flexible object that are suitable for gripping. In this embodiment, a CNN (Convolutional Neural Network) is used as a machine learning model. More specifically, CNN YOLO v2 is used as a machine learning model, which is capable of outputting the appropriateness of gripping along with the coordinate position in the image of a part suitable for gripping or not suitable for gripping. It should be noted that another neural network capable of outputting a quantified value of the propriety of gripping a part of the object based on image data obtained by imaging the object may be used, or another machine learning model may be used. may be used. In the present embodiment, the learning calculation unit 14 uses the training data to specifically calculate parameter values such as weighting coefficients of the neural network. The learning calculation unit 14 stores the learned model in the model storage unit 15 after the calculation processing using the training data is completed.

Therefore, in the model storage unit 15, based on the image data obtained by imaging the object, the following learned model for functioning the computer is output as a value quantifying the propriety of gripping the part of the object. is stored. That is, the weighting coefficients of the neural network that constitutes this trained model are learned using training data including image data obtained by imaging an object. Note that this image data includes the above-described first image data and second image data. Then, this trained model performs an operation according to the weighting coefficients on the image data, which is input to the input layer of the neural network and shows the flexible objects in a folded and stacked state. The computer is operated to output from the output layer of the neural network whether or not the part of the soft object shown is suitable for gripping. The input layer may receive image data that does not show the folded and stacked flexible objects. Thus, the learned models stored in the model storage unit 15 are used as program modules.

The image data acquisition unit 16 acquires image data obtained by imaging an object. The image data acquisition unit 16 acquires, for example, image data in which flexible objects in a folded and stacked state are imaged. In the present embodiment, the image data acquisition unit 16 acquires image data captured by the imaging device 10, but image data stored in advance in a storage device such as a memory of the gripping device 1 may be read and acquired. Alternatively, image data transmitted from another device may be obtained. In this embodiment, three-dimensional image data is obtained for controlling the robot arm 11, but the three-dimensional image data is not necessarily required for simply determining an appropriate gripping position. That is, even with two-dimensional image data, it is possible to determine an appropriate gripping position using a trained model.

The contact position determination unit 17 determines a contact position for deforming a soft object, which is a grasped object, by the robot arm 11 (more specifically, the end effector 113). More specifically, the contact position determination unit 17 determines the contact position for deformation of the edge of the upper surface of the piled flexible object. That is, the contact position determination unit 17 determines the contact position of the grasped object with the robot arm 11 when the robot arm 11 imparts displacement to the grasped object. In this embodiment, the displacement is applied by pushing the gripped object with the robot arm 11 (end effector 113). For this reason, in the present embodiment, the contact position determination unit 17 determines the contact position for the robot arm 11 to push the soft object that is the object to be gripped. The contact position determination unit 17 determines the contact position using the image data acquired by the image data acquisition unit 16, which is the image data of the flexible object before the end of the upper surface is deformed by the robot arm 11. do. The details of determining the contact position will be described later using a flowchart.

The grip position determination unit 18 uses the image data acquired by the image data acquisition unit 16 to determine whether or not the part of the flexible object shown in the image data is suitable for gripping. That is, the gripping position determining unit 18 uses image data obtained by capturing images of flexible objects in a folded and stacked state to determine whether a part of the flexible object shown in the image data is a part suitable for gripping. determine whether More specifically, the gripping position determining unit 18 uses image data obtained by capturing an image of a flexible object whose upper surface has been deformed with respect to the edge. Also, at that time, the gripping position determination unit 18 uses the learned model stored in the model storage unit 15 . That is, the gripping position determination unit 18 of the present embodiment applies the above-described CNN, in which parameters including weighting coefficients are set in advance, to the image data acquired by the image data acquisition unit 16, and determines the region and the area suitable for gripping. A region unsuitable for grasping is determined, and a region suitable for grasping is output.

The controller 12 controls the robot arm 11 as described above. In particular, the control unit 12 controls the robot arm 11 to deform the edge of the upper surface of the uppermost soft object when determining the gripping position. Specifically, the control unit 12 controls the robot arm 11 so that the robot arm 11 pushes the flexible object with a predetermined force at the contact position determined by the contact position determining unit 17, thereby deforming the end of the flexible object. . In this embodiment, the control unit 12 controls the end effector 113 of the robot arm 11 to push the edge of the upper surface of the uppermost flexible object, which is the contact position determined by the contact position determination unit 17, from the side direction. do. Since the action for generating the deformation is simply a pushing action, the deformation can be easily realized.

In the present embodiment, the end of the soft object is pushed laterally as described above, but the robot arm 11 may apply displacement to other positions when determining the gripping position. For example, the control unit 12 may control the upper surface of the uppermost soft object so that a position a predetermined distance away from the edge is pushed from above. In this embodiment, the control unit 12 controls to push the soft object at the contact position. The soft object may be displaced by other actions of . For example, the end of the flexible object may be deformed by applying a displacement by another action such as pulling the flexible object at the contact position determined by the contact position determination unit 17 . Note that the image data included in the training data (training data used in the learning calculation unit 14) used to create the learned model is deformed in the same manner as the deformation method by the robot arm 11 that is performed when determining the gripping position. It is image data of a flexible object deformed by the method.

Further, the control unit 12 controls the robot arm 11 so as to grip the flexible object to be gripped, with the region determined by the gripping position determination unit 18 as being suitable for gripping as the gripping position. In the present embodiment, the control unit 12 controls the robot arm 11 so as to grip the soft object by laterally gripping the edge determined to be the area suitable for gripping. A soft object may be gripped by grasping the vicinity of the cut edge from any direction.

The control unit 12, the training data acquisition unit 13, the learning calculation unit 14, the image data acquisition unit 16, the contact position determination unit 17, and the grip position determination unit 18 are stored in a storage device such as a memory included in the grip device 1, for example. A program including various commands executed by the gripping device 1 is executed by a processor. Also, the model storage unit 15 is configured by a storage device such as a memory included in the gripping device 1 . Here, the processor may be a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or other types of processors. The memory may be volatile memory or non-volatile memory.

Next, the operation of the gripping device 1 will be described using a flowchart.
FIG. 5 is a flowchart showing the flow of learning processing by the gripping device 1. As shown in FIG.
In the learning process, first, in step 10 (S10), the training data acquisition unit 13 acquires training data. The training data obtained in step 10 includes the labeled first image data and second image data described above.
Next, in step 11 (S11), the learning calculation unit 14 uses the training data acquired in step 10 to perform predetermined calculation processing for learning CNN for determining a part suitable for grasping.
Next, at step 12 (S12), the model storage unit 15 stores the learned model whose parameter values have been determined upon completion of the arithmetic processing at step 11. FIG.

FIG. 6 is a flow chart showing the flow of processing when the gripping device 1 grips a gripping target.
When gripping is performed, first, in step 20 (S20), the image data acquisition unit 16 acquires image data of an object captured by the imaging device 10. FIG.
Next, in step 21 (S21), the contact position determination unit 17 determines a contact position for deforming the soft object that is the grasped object. In the present embodiment, the contact position determination unit 17 specifically determines the contact position by the process shown in the flowchart of FIG.

Details of the processing of the contact position determining unit 17 will be described below with reference to the flowchart of FIG. 7 .
First, at step 210 (S210), the contact position determination unit 17 detects a plane on which the flexible object, which is the object to be gripped, is placed. That is, the contact position determination unit 17 performs predetermined image analysis processing on the image data acquired by the image data acquisition unit 16 to detect a plane.
Next, at step 211 (S211), the contact position determining unit 17 performs predetermined image analysis processing to detect a mass present in the image above the plane detected at step 210, and cuts it out from the image.
Next, at step 212 ( S<b>212 ), the contact position determining unit 17 performs predetermined edge detection processing on the lump image cut out at step 211 . The edges detected by this edge detection process correspond to the outline of the grasped object (pile up soft objects).
Next, in step 213 (S213), the contact position determination unit 17 determines the edge of the upper surface of the piled up flexible objects as the contact position. Specifically, in order to achieve this, the contact position determination unit 17 determines the position of the edge existing at the highest position (uppermost position) among the edges detected in step 212 as the contact position.

After the contact position is determined as described above, the process proceeds to step 22 (see FIG. 5).
In step 22 (S22), the control unit 12 controls the robot arm 11 so that the contact position determined in step 21 is laterally pressed by the end effector 113 in order to deform the end of the gripped object (see FIG. 8). .
Next, at step 23 ( S<b>23 ), the image data acquisition section 16 acquires image data of the object imaged by the imaging device 10 . Note that the image data acquired in step 23 includes an image of the grasped object deformed in step 22 . A cloth product or the like maintains its deformation due to being pushed even after the end effector 113 is released. Therefore, in step 23, image data captured with the end effector 113 separated from the soft object may be acquired.
At step 24 (S24), the gripping position determination unit 18 performs determination processing regarding a position suitable for gripping. That is, the gripping position determining unit 18 determines, using the learned model, whether or not the part of the object to be gripped shown in the image data acquired in step 23 is suitable for gripping. If a position suitable for gripping is found (Yes in step 25 (S25)), the process proceeds to step 26. FIG. If no position suitable for grasping is found (No at step 25), the process returns to step 20. When the process returns to step 20, for example, the process after step 20 is repeated for image data captured from a direction different from the previous one. As a result, the other ends are similarly subjected to the determination process regarding positions suitable for gripping.
In step 26 (S26), the control unit 12 controls the robot arm 11 so that the end effector 113 laterally grips the position determined to be suitable for gripping in step 24 in order to grip the gripping target.

According to the gripping device 1 configured in this manner, the edge of the upper surface of the flexible object is deformed by the robot arm 11 when determining the gripping position. Edges that are joined together and edges that are not such edges deform differently. In other words, the gripping position determination unit 18 performs determination using images having different modes depending on whether the part is suitable for gripping. Therefore, the gripping position determination unit 18 can determine an appropriate gripping position. Further, the gripping device 1 has the contact position determination section 17 described above. Therefore, the contact position for generating deformation can be determined autonomously. That is, in the gripping device 1, the control unit 12 performs control according to the contact position determined by the contact position determining unit 17, so the gripping device 1 can autonomously deform the flexible object.

It should be noted that the present invention is not limited to the above embodiments, and can be modified as appropriate without departing from the scope of the invention. For example, in the above embodiment, the grasping device 1 includes all the components shown in FIG. 3, but some of them may be included in another device. For example, a system as shown in FIG. 9 may be constructed.

FIG. 9 is a schematic diagram showing a gripping system 2 having an imaging device 10, a server 3, and a gripping device 4. As shown in FIG. In the gripping system 2 shown in FIG. 9, the imaging device 10 is installed in space instead of the gripping device 4 . 3, the server 3 includes the training data acquisition unit 13, the learning calculation unit 14, the model storage unit 15, and the gripping position determination unit 18, and the other components are provided by the gripping device 4. I have. In this case, the control unit 12, the image data acquisition unit 16, and the contact position determination unit 17 are provided with a program including various instructions stored in a storage device such as a memory provided in the gripping device 4. It is realized by being executed by a processor. Further, the training data acquisition unit 13, the learning calculation unit 14, and the gripping position determination unit 18 are configured such that a program including various instructions stored in a storage device such as a memory provided in the server 3 is executed by a processor provided in the server 3. It is realized by being executed. Also, the model storage unit 15 is configured by a storage device such as a memory included in the server 3 . Image data is transmitted and received between the imaging device 10 and the gripping device 4 . That is, the image data acquisition unit 16 of the gripping device 4 acquires the image data transmitted from the imaging device 10 and received by the gripping device 4 . Further, between the server 3 and the gripping device 4, transmission/reception of image data required for processing by the gripping position determination unit 18 and transmission/reception of processing results by the gripping position determination unit 18 are performed.

Note that FIG. 9 is an example of distributed processing, and distributed processing may be performed by other combinations or other devices. The configuration example shown in FIG. 3 and the configuration example shown in FIG. 9 are not the only examples. Conversely, the system may be configured such that the processing by the learning calculation unit 14 is performed by the gripping device, and the processing by the gripping position determination unit 18 is performed by the server.

It should be noted that the programs described above can be stored and supplied to computers using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., floppy disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical discs), Compact Disc Read Only Memory (CD-ROM), CD-ROM. Includes R, CD-R/W, semiconductor memory (e.g. Mask ROM, Programmable ROM (PROM), Erasable PROM (EPROM), Flash ROM, Random Access Memory (RAM)). The program may also be delivered to the computer on various types of transitory computer readable medium. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable media can deliver the program to the computer via wired channels, such as wires and optical fibers, or wireless channels.

1 gripping device 2 gripping system 3 server 4 gripping device 10 imaging device 11 robot arm 12 control unit 13 training data acquisition unit 14 learning calculation unit 15 model storage unit 16 image data acquisition unit 17 contact position determination unit 18 grip position determination unit

Claims (7)

an image data acquisition unit that acquires image data obtained by imaging an object;
a gripping part that grips an object;
a control unit that controls the operation of the gripping unit;
Using the image data obtained by the image data acquisition unit, in which the flexible object in a folded and stacked state is imaged, whether the part of the flexible object shown in the image data is suitable for gripping. a gripping position determination unit that determines whether or not
has
The control unit controls the gripping unit to deform the end of the upper surface of the soft object on the uppermost stage,
The gripping position determination unit determines whether or not the soft object is a part suitable for gripping, using the image data obtained by imaging the flexible object whose end of the upper surface is deformed under the control of the gripping unit. Gripping device . Using the image data obtained by the image data acquisition unit, in which the flexible object is imaged before the edge of the upper surface is deformed, the gripping unit of the flexible object for deforming the edge of the upper surface It further has a contact position determination unit that determines the contact position with
The gripping device according to claim 1, wherein the control unit controls the gripping unit to displace the soft object at the contact position determined by the contact position determination unit in order to deform the edge of the upper surface. . The gripping device according to claim 1 or 2, wherein the control section controls the gripping section such that the gripping section pushes the soft object to deform the edge of the upper surface. a training data acquisition unit that acquires training data including image data obtained by imaging an object;
a learning computation unit that performs machine learning computation using the training data acquired by the training data acquisition unit;
has
The image data includes first image data and first image data showing a flexible object in a state of being folded and stacked, with the edge of the upper surface of the flexible object being deformed from a regular lamination pattern shape. 2 image data,
The first image data is image data in which the edge of the upper surface is an edge that is combined into one by folding,
The second image data is image data in which the edge of the upper surface is an edge that is not combined into one by folding,
The learning device, wherein the learning calculation unit uses the training data to perform a calculation for learning a part of a flexible object that is suitable for gripping. A program executed by the computer to output a value quantifying the propriety of gripping a part of the object based on image data obtained by imaging the object, wherein the program has been learned. including the model
The weighting coefficients of the neural network that constitutes the trained model are learned using training data including image data obtained by imaging an object,
The image data included in the training data is a flexible object in a folded and stacked state, and shows a flexible object in a state where the edge of the upper surface of the flexible object is deformed from a regular lamination pattern shape. including one image data and a second image data,
The first image data is image data in which the edge of the upper surface is an edge that is combined into one by folding,
The second image data is image data in which the edge of the upper surface is an edge that is not combined into one by folding,
The trained model causes the computer to perform calculations according to the weighting coefficients on image data in which flexible objects in a folded and stacked state are input to the input layer of the neural network, and output from the output layer of the neural network whether or not the part of the soft object shown in the image data is suitable for gripping;
program . an imaging device for imaging surrounding objects;
an image data acquisition unit that acquires image data from the imaging device;
a gripping part that grips an object;
a control unit that controls the operation of the gripping unit;
Using the image data obtained by the image data acquisition unit, in which the flexible object in a folded and stacked state is imaged, whether the part of the flexible object shown in the image data is suitable for gripping. a gripping position determination unit that determines whether or not
has
The control unit controls the gripping unit to deform the end of the upper surface of the flexible object,
The gripping position determining unit determines whether or not the soft object is a part suitable for gripping, using the image data obtained by imaging the flexible object whose end of the upper surface is deformed under the control of the gripping unit. Gripping system . Acquire training data including image data of an object,
Using the acquired training data, perform machine learning calculations for determining a part of a flexible object that is suitable for gripping,
The image data includes first image data and first image data showing a flexible object in a state of being folded and stacked, with the edge of the upper surface of the flexible object being deformed from a regular lamination pattern shape. 2 image data,
The first image data is image data in which the edge of the upper surface is an edge that is combined into one by folding,
The learning method, wherein the second image data is image data in which edges of the upper surface are edges that are not combined into one by folding.

Images