JP6838833B2 - Gripping device, gripping method, and program - Google Patents

Description

The present invention relates to a gripping device, a gripping method, and a program for determining the success or failure of gripping an object.

A gripping device that determines success or failure of gripping an object based on an output from a sensor of an end effector that grips the object and grips the object is known (see, for example, Patent Document 1).

Japanese Unexamined Patent Publication No. 2013-010155

By the way, it is assumed that the object is an object whose shape is likely to change before and after being gripped by the gripping means, such as a cloth product or a paper product. In this case, if the gripping means cannot grip the object, it is determined that the grip has failed. On the other hand, if the gripping means can grip the object, but the object is significantly deformed at this time, it is also considered to be a grip failure. .. However, in the gripping device, it is not assumed that such an object changes its shape before and after gripping by the gripping means. Therefore, although the gripping means can grip the object, if the object is significantly deformed, it may be erroneously determined as a successful grip.

The present invention has been made to solve such a problem, and provides a gripping device, a gripping method, and a program capable of accurately determining a gripping failure when an object can be gripped but the object is significantly deformed. The main purpose is that.

One aspect of the present invention for achieving the above object is a gripping means for gripping an object, an image acquisition means for acquiring image information of the object to be determined, and a determination target acquired by the image acquisition means. The feature amount calculation means for calculating the feature amount of the image information of the object, the gripping means can grip the object, and the deformation amount of the object is less than a predetermined amount and the gripping success is small, and the gripping means is the target. Each of the gripping success / failure patterns including the first gripping failure in which the object can be gripped and the deformation amount of the object is large by a predetermined amount or more, and the second gripping failure in which the gripping means cannot grip the object. A storage means for storing previously acquired time-series data of feature amounts of image information from the start of gripping the object to the completion of gripping by the gripping means, and a storage means stored in the storage means for the object by the gripping means. Each of the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping, and the feature amount of the image information from the start of gripping to the completion of gripping the object to be determined by the gripping means calculated by the feature amount calculating means. The success / failure determination means for calculating the correlation with the time-series data of the above and determining that the gripping success / failure pattern of the object to be determined is the gripping success / failure pattern associated with the time-series data of the feature amount having the highest correlation. The gripping device is characterized by the above.
In this aspect, when the success / failure determination means determines that the gripping success / failure pattern is successful, the gripping means is controlled so that the gripping means executes the next task, and the success / failure determination means controls the gripping means. When it is determined that the gripping success / failure pattern of the first gripping failure is determined, the control means for controlling to stop the operation of the gripping means may be further provided.
In this aspect, in the storage means, the gripping means can move the object, and the deformation amount of the object is less than a predetermined amount and the movement is successful, and the gripping means moves the object. Each of the movement success / failure patterns including the first movement failure in which the object can be deformed by a predetermined amount or more and the deformation amount of the object is large, and the second movement failure in which the gripping means cannot move the object. The time-series data of the feature amount of the image information from the completion of gripping the object by the gripping means acquired in advance to the completion of the movement of the object by the gripping means after the gripping is completed is stored. The success / failure determination means is calculated by each of the time-series data of the feature amount of the image information from the completion of gripping the object to the completion of movement by the gripping means stored in the storage means and the feature amount calculation means. The correlation between the time-series data of the feature amount of the image information from the completion of gripping the object to be determined by the gripping means to the completion of movement is calculated, and the gripping success / failure pattern of the object to be determined has the highest correlation. It may be determined that the movement success / failure pattern is associated with the time-series data of the feature quantity.
In this aspect, when the success / failure determination means determines that the grip success / failure pattern of the first grip failure is determined, the notification means for notifying the content of the first grip failure may be further provided.
One aspect of the present invention for achieving the above object is a gripping means for gripping an object, an image acquisition means for acquiring image information of the object, and image information of the object acquired by the image acquisition means. The feature amount calculating means for calculating the feature amount of the above, and the gripping means can grip the object, and the deformation amount of the object is less than a predetermined amount and the gripping success is small, and the gripping means can grip the object. A success / failure determination means for determining a gripping success / failure pattern including a first gripping failure in which the deformation amount of the object is large by a predetermined amount or more and a second gripping failure in which the gripping means cannot grip the object. The success / failure determination means grips the object, and the gripping means grips the object when the grip is completed, based on the image information of the gripping means when the grip is completed, which is acquired by the image acquisition means. It is determined that the amount of change in the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object calculated by the feature amount calculation means is less than the first predetermined change amount. When the gripping is completed, it is determined that the gripping success / failure pattern is successful, and the gripping means grips the object when the gripping is completed, based on the image information of the gripping means at the time of completing the gripping acquired by the image acquisition means. It is determined that the image information has not been changed, and the amount of change in the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object calculated by the feature amount calculation means is equal to or greater than the first predetermined change amount. When it is determined, it is determined that it is the gripping success / failure pattern of the first gripping failure, and based on the image information of the gripping means at the time of completion of gripping acquired by the image acquisition means, the gripping means is said to be said when the gripping is completed. The gripping device may be characterized in that when it is determined that the object is not gripped, it is determined that it is the gripping success / failure pattern of the second gripping failure.
One aspect of the present invention for achieving the above object is a step of acquiring image information of an object to be determined, a step of calculating a feature amount of the acquired image information of the object to be determined, and grasping. The means can grip the object and the deformation amount of the object is small less than a predetermined amount, and the gripping means succeeds in gripping the object. The gripping means can grip the object and the deformation amount of the object is large by a predetermined amount or more. Images from the start to the completion of gripping the object by the gripping means acquired in advance for each of the gripping success / failure patterns including the gripping failure of 1 and the second gripping failure in which the gripping means cannot grip the object. The step of storing the time-series data of the feature amount of the information, the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the stored gripping means, and the calculated time-series data. The correlation between the time-series data of the feature amount of the image information from the start of gripping the object to be determined by the gripping means to the completion of gripping is calculated, and the gripping success / failure pattern of the object to be determined has the highest correlation. The gripping method may include a step of determining a gripping success / failure pattern associated with the time series data of the quantity.
One aspect of the present invention for achieving the above object is a process of acquiring image information of an object to be determined, a process of calculating a feature amount of the acquired image information of the object to be determined, and grasping. The means can grip the object and the deformation amount of the object is small less than a predetermined amount, and the gripping means succeeds in gripping the object. The gripping means can grip the object and the deformation amount of the object is large by a predetermined amount or more. An image from the start of gripping the object to the completion of gripping by the gripping means acquired in advance for each of the gripping success / failure patterns including the gripping failure of 1 and the second gripping failure in which the gripping means cannot grip the object. The process of storing the time-series data of the feature amount of the information, the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the stored gripping means, and the calculated time-series data. The correlation between the time-series data of the feature amount of the image information from the start of gripping the object to be determined by the gripping means to the completion of gripping is calculated, and the gripping success / failure pattern of the object to be determined has the highest correlation. The program may be characterized in that a computer is made to execute a process of determining a gripping success / failure pattern associated with a quantity of time-series data.

According to the present invention, it is possible to provide a gripping device, a gripping method, and a program that can accurately determine a gripping failure when an object can be gripped but the object is significantly deformed.

It is a block diagram which shows the schematic system structure of the gripping apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the schematic structure of a robot arm. It is a flowchart which shows an example of the processing flow of the gripping method which concerns on Embodiment 1 of this invention.

Embodiment 1
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a schematic system configuration of a gripping device according to a first embodiment of the present invention. The gripping device 1 according to the first embodiment grips an object whose shape is likely to change, such as a cloth product or a paper product, and determines the success or failure of the gripping.

The gripping device 1 first has [1] a learning process of generating learning data of image information of an object and storing it in a storage unit 5, and [2] image information of the object from the start of gripping to the completion of gripping after the learning process. Based on the above, a determination process for determining the success or failure of the gripping operation is executed.

The gripping device 1 includes, for example, a CPU (Central Processing Unit) that performs control processing, arithmetic processing, etc., a ROM (Read Only Memory) or RAM (Random Access Memory) that stores arithmetic programs, control programs, etc. executed by the CPU. Each hardware is configured around a microcomputer consisting of a memory composed of a memory, an interface unit (I / F) for inputting / outputting signals to / from the outside, and the like. The CPU, memory, and interface unit are connected to each other via a data bus or the like.

The gripping device 1 includes an image acquisition unit 2 that acquires an image of an object, a gripping execution unit 4 that executes a gripping operation of the robot arm 3, a robot arm 3 that grips the object, and a storage unit that stores learning data. 5, a gripping position setting unit 6 for setting a gripping position of an object, a success / failure determination unit 7 for determining the success or failure of gripping an object, a feature amount calculation unit 8 for calculating a feature amount of image information, and a user It is provided with a notification unit 9 that notifies the user.

The image acquisition unit 2 is a specific example of the image acquisition means. The image acquisition unit 2 acquires image information of an object by using, for example, a distance image sensor. The image information may include distance information. For example, the image information includes image information obtained by imaging the measurement target from the sensor (measurement point) and distance information from the sensor to the measurement target, and per pixel is "RGBD (color + distance)" or "gray scale". (Brightness value) + distance "is included. The range image sensor is composed of, for example, a stereo camera (three-dimensional camera), an LRF (laser range finder) + camera, a Microsoft Kinect (registered trademark), and the like.

The gripping execution unit 4 is a specific example of the control means. The gripping execution unit 4 executes gripping of an object by controlling the drive of the robot arm 3.

The robot arm 3 is a specific example of the gripping means. The robot arm 3 is, for example, a plurality of links 31, joints (wrist joints, elbow joints, shoulder joints, etc.) 32 that rotatably connect each link 31, and an end provided at the tip thereof to grip an object. It has an effector 33 (FIG. 2).

Each joint 32 detects a rotation sensor 34 such as an encoder that detects rotation information of each joint 32, an actuator 35 such as a servomotor that drives each joint 32, and an operating force of each joint 32. A force sensor 36 is provided. The force sensor 36 is, for example, a torque sensor that detects the torque of each joint portion 32. Each joint portion 32 is provided with a deceleration mechanism. The end effector 33 is composed of, for example, a plurality of finger portions, and the object can be gripped by sandwiching the end effector 33 between the finger portions. The end effector 33 is provided with an actuator 37 for driving the end effector 33 and a force sensor 38 for detecting the operating force of the end effector 33.

The gripping execution unit 4 is a specific example of the control means. The gripping execution unit 4 is, for example, based on the rotation information (rotation angle, etc.) from the rotation sensor 34 of each joint portion 32 and the operating force from the force sensor 36, and the actuator of each joint portion 32 and the end effector 33. By controlling 35 and 37, the robot arm 3 is feedback-controlled. As a result, the robot arm 3 can grasp and move the object.

The feature amount calculation unit 8 is a specific example of the feature amount calculation means. The feature amount calculation unit 8 calculates the feature amount of the image information based on the image information acquired by the image acquisition unit 2. The feature amount calculation unit 8 is suitable for capturing a change in the shape (edge) of an object, for example, it is preferable to calculate a HOG (Histograms of Oriented Gradients) feature amount. As a result, it is possible to better capture the characteristics of the edge of the object whose shape changes, which will be described later, and to perform the gripping success / failure judgment with higher accuracy. The feature amount calculation unit 8 may calculate a feature amount such as a SURF (Speeded Up Robust Features) feature amount or a SIFT (Scale-Invariant Feature Transform) feature amount, or may calculate an arbitrary feature amount. ..

The storage unit 5 is a specific example of the storage means. The storage unit 5 stores in advance learning data such as image information of an object and its feature amount. The storage unit 5 is composed of, for example, the above-mentioned memory.

The learning data are associated with each other as follows, for example.
(1) A plurality of sets of image information of the object before being gripped by the robot arm 3 and a gripping position (grasping posture) of the object optimal for gripping the object are associated with each other.

(2) Time-series data of the feature amount of the image information of the object from the start of the robot arm 3 gripping the object (start of gripping) to the completion of the gripping (grasping completed), and the following gripping A plurality of sets of success / failure patterns (a) to (c) are associated with each other.

The gripping success / failure pattern is as follows: (a) the robot arm 3 can grip the object and the deformation amount of the object is less than a predetermined amount and the grip is small, and (b) the robot arm 3 can grip the object. This includes a first grip failure in which the amount of deformation of the object is larger than a predetermined amount, and (c) a second grip failure in which the robot arm 3 cannot grip the object.

Note that the robot arm 3 can grip the object means, for example, a state in which the finger portion of the end effector 33 of the robot arm 3 can pinch and hold the object. On the other hand, the fact that the robot arm 3 cannot grip the object means, for example, a state in which the finger portion of the end effector 33 of the robot arm 3 is detached from the object and cannot be gripped (held).

The second gripping failure of the gripping success / failure pattern (c) includes (c-1) a gripping failure in which the robot arm 3 cannot touch the object and therefore cannot grip the object and the shape of the object does not change, and (c-1). -2) The robot arm 3 tried to grip the object, but the shape of the object changed and the robot arm 3 failed to grip the object. The amount of deformation of an object is, for example, the amount of dimensional change of the object in the vertical, horizontal, or height direction when the object is a rectangular parallelepiped, and when the object is spherical, the dimension of its diameter. The amount of change. In addition, the predetermined amount is appropriately set in consideration of the properties of the object and the like.

The gripping position setting unit 6 is based on the image information of the object before gripping, which is the learning data stored in the storage unit 5, and the image information of the object acquired by the image acquisition unit 2. Set the gripping position. For example, the gripping position setting unit 6 matches the image information of the object before gripping, which is the learning data stored in the storage unit 5, with the image information of the object acquired by the image acquisition unit 2. As a result of matching, the gripping position setting unit 6 selects the image information of the object acquired by the image acquisition unit 2 and the image information of the object of the learning data of the storage unit 5, which has the highest likelihood. Then, the gripping position setting unit 6 sets the gripping position associated with the image information of the selected object as the gripping position of the object. The gripping execution unit 4 calculates the rotation angle of each joint of the robot arm 3 using reverse kinematics based on the gripping position of the object set by the gripping position setting unit 6, and is based on the calculated rotation angle. Then, the robot arm 3 is controlled to start grasping the object.

The success / failure determination unit 7 is a specific example of the success / failure determination means. The success / failure determination unit 7 was calculated by the feature amount calculation unit 8 based on the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object, which is the learning data stored in the storage unit 5. It is determined which of the gripping success / failure patterns (a) to (c) corresponds to the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping the object, which is the determination data.

[Learning process]
As described above, the gripping device 1 according to the first embodiment first performs a process of generating learning data of image information of an object and storing it in a storage unit 5.

The image acquisition unit 2 acquires image information of the object before being gripped by the robot arm 3. The gripping execution unit 4 controls the robot arm 3 described above to grip the object. At this time, the image acquisition unit 2 acquires time-series data of image information from the start of gripping the object to the completion of gripping. The feature amount calculation unit 8 calculates the feature amount time-series data of the image information based on the time-series data of the image information from the start of gripping to the completion of gripping of the robot arm 3 acquired by the image acquisition unit 2. .. The above process is repeated a plurality of times.

As described above, the storage unit 5 corresponds to a plurality of sets of the image information of the object before gripping acquired by the image acquisition unit 2 and the optimum gripping position of the object for gripping the object. Attach and store as learning data. As described above, the storage unit 5 contains time-series data of the feature amount of the image information acquired by the image acquisition unit 2 from the start of gripping to the completion of gripping, and the gripping success / failure patterns (a) to (c). One of them is associated with a plurality of sets and stored as training data.
For example, an SVM (support vector machine) or a random forest (Random Forest) is used for learning the association.

[Determination process]
Subsequently, the gripping device 1 according to the first embodiment performs the above-mentioned [learning process], and then determines the success or failure of the gripping operation based on the image information of the object from the start of gripping to the completion of gripping. I do.

The gripping execution unit 4 grips the object by controlling the drive of the robot arm 3.
The image acquisition unit 2 acquires time-series data of image information from the start of gripping the object to the completion of gripping, which is the determination data. Then, the feature amount calculation unit 8 calculates the time-series data of the feature amount of the image information based on the time-series data of the image information from the start of gripping to the completion of gripping of the robot arm 3 acquired by the image acquisition unit 2. To do.

The success / failure determination unit 7 is based on the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping the object, which is the learning data stored in the storage unit 5, and the determination data calculated by the feature amount calculation unit 8. A comparison is made with the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping an object.

For example, the success / failure determination unit 7 is time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object, which is the learning data of the storage unit 5, and the determination data calculated by the feature amount calculation unit 8. Correlation values are calculated with the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping. Then, when the success / failure determination unit 7 has the largest feature amount calculated from the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object, which is the learning data of the storage unit 5. The series data is selected, and it is determined that the gripping success / failure patterns (a) to (c) are associated with the time series data of the selected feature amount. The success / failure determination unit 7 calculates, for example, Pearson's product-moment correlation coefficient as the correlation value.

By the way, it is assumed that the object is an object whose shape is likely to change before and after being gripped by the robot arm, such as a cloth product or a paper product. In this case, if the robot arm cannot grip the object, it is determined that the grip has failed. On the other hand, if the robot arm can grip the object, but the object is significantly deformed at this time, it is also considered to be a grip failure. .. However, in the conventional gripping device, it is not assumed that such an object is likely to change its shape before and after gripping by the robot arm. Therefore, the robot arm was able to grip the object, but if the object is significantly deformed at this time, it may be erroneously determined as a successful grip.

On the other hand, in the gripping device 1 according to the present embodiment, as described above, in the [learning process], the storage unit 5 can (a) the robot arm 3 can grip the object and the amount of deformation of the object. Is less than a predetermined amount and the grip is small, (b) the robot arm 3 can grip the object, and the deformation amount of the object is larger than the predetermined amount, the first grip failure, and (c) the robot arm 3 is the target. The time-series data of the feature amount of the image information from the start of gripping the object by the robot arm 3 to the completion of gripping, which is acquired in advance for each of the gripping success / failure patterns including the second gripping failure in which the object cannot be gripped, is stored.

Then, in the [determination process], the success / failure determination unit 7 includes each of the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the robot arm 3 stored in the storage unit 5, and the feature amount. A correlation value is calculated with the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping the object to be determined, which is calculated by the calculation unit 8. Then, the success / failure determination unit 7 determines that the gripping success / failure pattern of the object to be determined is the gripping success / failure pattern associated with the time-series data of the feature amount having the highest correlation.

As a result, the robot arm 3 can grip the object, and the robot arm 3 can grip the object, clearly distinguishing the above-mentioned (a) successful grip and particularly (c) second grip failure in the gripping success / failure pattern. A first gripping failure, in which the amount of deformation is equal to or greater than a predetermined amount, can be included. Therefore, when the object can be gripped but the object is greatly deformed by a predetermined amount or more, it can be accurately determined as a grip failure.

Subsequently, the post-determination processing of the gripping success / failure patterns (a) to (c) will be described.
When the success / failure determination unit 7 determines that the grip is successful (the grip success / failure pattern (a) is successfully gripped), the grip execution unit 4 controls the robot arm 3 so that the robot arm 3 executes the next task. In this case, since the robot arm 3 can grip the object and the amount of deformation of the object is small, the robot arm 3 can execute the next task as it is.

On the other hand, when the success / failure determination unit 7 determines that the first gripping failure of the gripping success / failure pattern (b) is performed, the robot arm 3 is able to grip the object, but the deformation amount of the object is large at a predetermined amount or more. .. In this case, for example, when the robot arm 3 subsequently performs the next task such as moving the object, the amount of deformation of the object becomes larger, and the object being held is dropped. There is a high possibility that it will lead to the operation failure of. Therefore, the gripping execution unit 4 temporarily controls to stop the operation of the robot arm 3.

At this time, for example, the user may modify the object to return to its original shape or bring it closer to the original shape by his / her own hand or by using the robot arm 3 or the like. In this case, the gripping position setting unit 6 matches the learning data again and resets the gripping position of the object. Specifically, the gripping position setting unit 6 has the highest likelihood of learning data of the storage unit 5 with the image information of the object acquired by the image acquisition unit 2, excluding the image information of the object selected last time. Select the image information of the object of. Then, the gripping position setting unit 6 resets the gripping position associated with the image information of the selected object as the gripping position of the object. The gripping execution unit 4 controls the robot arm 3 based on the gripping position of the object reset by the gripping position setting unit 6, and restarts the gripping of the object.
As described above, when the amount of deformation of the object gripped by the robot arm 3 is large, the subsequent operation of the robot arm 3 can be reliably stopped. Further, when the shape of the object is modified or the like, the robot arm 3 can re-grasp the object.
When the success / failure determination unit 7 determines that (b) the robot arm 3 can grip the object and the deformation amount of the object is equal to or greater than a predetermined amount, it determines that the first grip failure pattern is a grip success / failure pattern. May be notified to the user. The success / failure determination unit 7 notifies the user by using, for example, a notification unit 9 such as a speaker or a display device. As a result, the object can be gripped, but the user can be clearly notified that the object has been deformed so much that the grip has failed.

When the success / failure determination unit 7 determines that the second gripping failure of the gripping success / failure pattern (c) is performed, the gripping position setting unit 6 matches the learning data again and resets the gripping position of the object.
Here, in the second gripping failure of the gripping success / failure pattern (c), as described above, (c-1) the robot arm 3 cannot touch the object, so that the object cannot be gripped and the shape of the object does not change. This includes a case where (c-2) the robot arm 3 tries to grip the object but the shape of the object changes and the robot arm 3 cannot grip the object.

In this case, for example, the storage unit 5 cannot contact the time-series data of the feature amount of the image information from the start of gripping the object by the robot arm 3 to the completion of gripping, and (c-1) the robot arm 3 cannot contact the object. A gripping failure in which the object could not be gripped and the shape of the object did not change, and (c-2) a gripping failure in which the robot arm 3 tried to grip the object but the shape of the object changed and could not be gripped. It may be associated and further stored.

The success / failure determination unit 7 includes time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping of the object in the storage unit 5, and the time series data of the feature amount calculated by the feature amount calculation unit 8 from the start of gripping the object to the completion of gripping. The correlation value with the time-series data of the feature amount of the image information is calculated. Then, the success / failure determination unit 7 selects the time-series data of the feature amount having the largest calculated correlation value from the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping, which is the learning data of the storage unit 5. It is selected and determined to be the second grasp failure of (c-1) or (c-2) associated with the time series data of the selected feature amount.

When the success / failure determination unit 7 determines that the second gripping failure of (c-1) has occurred, the shape of the object has not changed. Therefore, the gripping execution unit 4 re-grasps the object as follows.
For example, the gripping position setting unit 6 has the highest probability of the image information of the object acquired by the image acquisition unit 2 except for the image information of the object selected last time, and is the object of the learning data of the storage unit 5. Select the image information of. Then, the gripping position setting unit 6 resets the gripping position associated with the image information of the selected object as the gripping position of the object. The gripping execution unit 4 controls the robot arm 3 based on the gripping position of the object reset by the gripping position setting unit 6, and restarts the gripping of the object.

On the other hand, when the success / failure determination unit 7 determines that the second gripping failure of the gripping success / failure pattern (c-2) has occurred, the shape of the object has changed significantly. Therefore, when the robot arm 3 grips the object whose shape has changed, there is a high possibility that the gripping will fail again. Therefore, the gripping execution unit 4 once controls to stop the operation of the robot arm 3.

After that, for example, the user may make a modification to return or bring the object back to its original shape by his / her own hand or by using the robot arm 3 or the like. In this case, similarly to the gripping success / failure pattern (c-1), the gripping position setting unit 6 matches the learning data again and resets the gripping position of the object. For example, the gripping position setting unit 6 has the highest probability of the image information of the object acquired by the image acquisition unit 2 except for the image information of the object selected last time, and is the object of the learning data of the storage unit 5. Select the image information of. Then, the gripping position setting unit 6 resets the gripping position associated with the image information of the selected object as the gripping position of the object. The gripping execution unit 4 controls the robot arm 3 based on the gripping position of the object reset by the gripping position setting unit 6, and restarts the gripping of the object.
As described above, even if the robot arm 3 cannot grasp the object, the object can be re-grasped.

FIG. 3 is a flowchart showing an example of the processing flow of the gripping method according to the first embodiment.
The gripping position setting unit 6 performs matching and selects the image information of the object of the learning data of the storage unit 5, which has the highest likelihood with the image information of the object acquired by the image acquisition unit 2. Then, the gripping position setting unit 6 sets the gripping position associated with the image information of the selected object as the gripping position of the object (step S101).

The gripping execution unit 4 controls the robot arm 3 based on the gripping position of the object set by the gripping position setting unit 6, and starts gripping the object (step S102).
The image acquisition unit 2 acquires time-series data of the image information of the object from the start of gripping the object to the completion of gripping, which is the determination data (step S103).

The feature amount calculation unit 8 calculates the feature amount of the image information which is the determination data based on the time-series image information of the object acquired by the image acquisition unit 2 (step S104).

The success / failure determination unit 7 is a time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object, which is the learning data of the storage unit 5, and the object object, which is the determination data calculated by the feature amount calculation unit 8. Correlation values are calculated with the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping. Then, when the success / failure determination unit 7 has the largest feature amount calculated from the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object, which is the learning data of the storage unit 5. The series data is selected, and it is determined that the gripping success / failure patterns (a) to (c) are associated with the time series data of the selected feature amount (step S105).

The gripping execution unit 4 controls the operation of the robot arm 3 based on the gripping success / failure patterns (a) to (c) determined by the success / failure determination unit 7 (step S106).

As described above, in the gripping device 1 according to the first embodiment, in the storage unit 5, the robot arm 3 can grip the object, and the deformation amount of the object is less than a predetermined amount and the gripping is successful, and the robot arm 3 is the object. Each of the gripping success / failure patterns including the first gripping failure in which the object can be gripped and the deformation amount of the object is larger than a predetermined amount and the second gripping failure in which the robot arm 3 cannot grip the object is acquired in advance. The time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the robot arm 3 is stored. The success / failure determination unit 7 is calculated by the feature amount calculation unit 8 and each of the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the robot arm 3 stored in the storage unit 5. The correlation value with the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping the object to be determined is calculated. Then, the success / failure determination unit 7 determines that the gripping success / failure pattern of the object to be determined is the gripping success / failure pattern associated with the time-series data of the feature amount having the highest correlation.
Thereby, the gripping success, the first gripping failure, and the second gripping failure can be clearly distinguished and determined. Therefore, when the object can be gripped but the object is greatly deformed by a predetermined amount or more, it can be accurately determined as a grip failure.

Embodiment 2
In the second embodiment of the present invention, the success / failure determination unit 7 moves the object (lifting, lifting, sliding in the horizontal direction, etc.) after the robot arm 3 completes gripping the object (grasping is completed). Judges the success or failure of the movement operation until the completion (movement completion). As a result, even in the moving motion after the gripping is completed, if the object can be moved but the object is greatly deformed by a predetermined amount or more, it can be accurately determined as a movement failure.

The image acquisition unit 2 acquires a plurality of time-series data of the image information of the object from the completion of grasping the object to the completion of movement, which is the learning data. The feature amount calculation unit 8 calculates the time-series data of the feature amount of the image information based on the time-series data of the image information from the completion of grasping to the completion of movement of the robot arm 3 acquired by the image acquisition unit 2. ..

The storage unit 5 learns by associating the calculated time-series data of the image feature amount from the completion of grasping the object to the completion of movement with any of the following movement success / failure patterns (d) to (f). It is stored as data. In the above movement success / failure pattern, (d) the robot arm 3 can move the object, and the deformation amount of the object is less than a predetermined amount and the movement is small, and (e) the robot arm 3 moves the object. This includes a first movement failure in which the object can be deformed by a predetermined amount or more and a large amount of deformation, and (f) a second movement failure in which the robot arm 3 cannot move the object. The second movement failure in (f) above includes, for example, a case where the end effector 33 of the robot arm 3 drops an object being gripped during movement.

The gripping execution unit 4 controls the drive of the robot arm 3 to move the object from the gripped state.
The image acquisition unit 2 acquires time-series data of the image information of the object from the completion of grasping the object to the completion of movement, which is the determination data. The feature amount calculation unit 8 calculates the feature amount time-series data of the image information based on the time-series data of the image information from the completion of gripping to the completion of movement of the robot arm 3 acquired by the image acquisition unit 2.

The success / failure determination unit 7 is based on the time-series data of the feature amount of the image information from the completion of grasping the object to the completion of the movement, which is the learning data stored in the storage unit 5, and the determination data calculated by the feature amount calculation unit 8. A comparison is made with the time-series data of the feature amount of the image information from the completion of grasping to the completion of movement of a certain object. For example, the success / failure determination unit 7 is a time-series data of the feature amount of the image information from the completion of grasping to the completion of movement, which is the learning data of the storage unit 5, and the object which is the determination data calculated by the feature amount calculation unit 8. Correlation values with the time-series data of the feature amount of the image information from the completion of gripping to the completion of movement are calculated. Then, the success / failure determination unit 7 selects the time-series data of the feature amount having the largest calculated correlation value from the time-series data of the feature amount of the image information from the completion of grasping to the completion of movement, which is the learning data of the storage unit 5. It is selected, and it is determined that the movement success / failure patterns (d) to (f) are associated with the time series data of the selected feature amount.

The gripping execution unit 4 controls the robot arm 3 so that the robot arm 3 performs a moving operation when, for example, the success / failure determination unit 7 determines that the gripping is successful (the gripping success / failure pattern (a) is successfully gripped). After that, the success / failure determination unit 7 determines the success / failure of the movement operation from the completion of gripping the object to the completion of the movement.

When the success / failure determination unit 7 determines that the movement success / failure pattern (d) has been successfully moved, the gripping execution unit 4 controls the robot arm 3 so that the robot arm 3 executes the next task. In this case, since the robot arm 3 can move the object and the amount of deformation of the object is small, the robot arm 3 can execute the next task as it is.

On the other hand, when the success / failure determination unit 7 determines that the first and second movement failures of the movement success / failure patterns (e) and (f) are determined, for example, in the grip execution unit 4, the robot arm 3 stops the next task. The robot arm 3 is controlled so as to do so. In this case, since the shape of the object has changed significantly, the gripping execution unit 4 temporarily controls to stop the operation of the robot arm 3.

As described above, after confirming that the robot arm 3 can grip the object and that the deformation of the object is small, the robot arm 3 is subsequently moved, and the shape of the object is changed. Is taken into consideration to determine the success or failure of the movement operation. As a result, not only from the start of gripping the object to the completion of gripping but also from the completion of gripping the object to the completion of movement can be accurately determined as a failure when the object is significantly deformed by a predetermined amount or more, and the robot arm 3 The operation of can be stopped reliably.
In the second embodiment, since the other configurations are substantially the same as those in the first embodiment, the same parts are designated by the same reference numerals and detailed description thereof will be omitted.

Embodiment 3
In the third embodiment of the present invention, the success / failure determination unit 7 changes the time-series data of the feature amount of the image information calculated by the feature amount calculation unit 8 from the start of gripping to the completion of gripping, and when the gripping is completed. The gripping success / failure patterns (a) to (c) may be determined based on the presence / absence of gripping the object by the robot arm 3 of the above.

For example, in the case of successful gripping of the gripping success / failure pattern (a), the amount of deformation of the object is small from the start of gripping the object to the completion of gripping, and the shape of the object is substantially maintained. Therefore, the change in the feature amount of the image information in the time series is small from the start of gripping to the completion of gripping. Further, when the gripping is completed, the end effector 33 of the robot arm 3 grips the object.

In the case of the first grip failure of the gripping success / failure pattern (b), the amount of deformation of the object is large at a predetermined amount or more from the start of gripping to the completion of gripping. Therefore, the change in the feature amount of the image information in the time series is large from the start of gripping to the completion of gripping. Further, when the gripping is completed, the end effector 33 of the robot arm 3 grips the object.

In the case of the second gripping failure of the gripping success / failure pattern (c), the end effector 33 of the robot arm 3 does not grip the object when the gripping is completed.

Focusing on the amount of change in the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping and the presence / absence of gripping the object by the robot arm 3 at the completion of gripping, as follows. The gripping success / failure patterns (a) to (c) are determined.

The success / failure determination unit 7 grips the object by the end effector 33 of the robot arm 3 when the grip is completed, based on the image information of the end effector 33 of the robot arm 3 when the grip is completed acquired by the image acquisition unit 2. When it is determined that the amount of change in the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object calculated by the feature amount calculation unit 8 is less than the first predetermined change amount and is small. , It is determined that the gripping success / failure pattern (a) is successfully gripped. The first predetermined change amount is set in advance in the storage unit 5 or the like based on, for example, the properties of the object.

The success / failure determination unit 7 grips the object by the end effector 33 of the robot arm 3 when the grip is completed, based on the image information of the end effector 33 of the robot arm 3 when the grip is completed acquired by the image acquisition unit 2. When it is determined that the amount of change in the time-series data of the feature amount of the image information from the start of gripping to the completion of gripping of the object calculated by the feature amount calculation unit 8 is larger than the first predetermined change amount. , It is determined that the first gripping failure of the gripping success / failure pattern (b).

The success / failure determination unit 7 does not grip the object by the end effector 33 of the robot arm 3 when the grip is completed, based on the image information of the end effector 33 of the robot arm 3 when the grip is completed acquired by the image acquisition unit 2. When it is determined, it is determined that the second gripping failure of the gripping success / failure pattern (c) is performed.

The storage unit 5 may store in advance the image information of the end effector 33 when the end effector 33 of the robot arm 3 is holding the object and when the end effector 33 is not holding the object. The success / failure determination unit 7 compares these image information stored in the storage unit 5 with the image information of the end effector 33 at the time of completion of gripping acquired by the image acquisition unit 2, so that the robot arm 3 at the time of completion of gripping It is possible to determine whether or not the end effector 33 of the above grips the object.
According to the third embodiment, it is based on the amount of change in the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping, and the presence or absence of gripping the object by the robot arm 3 when the gripping is completed. Therefore, when the object can be gripped but the object is greatly deformed by a predetermined amount or more, it can be accurately determined as a grip failure.
In the third embodiment, since the other configurations are substantially the same as those in the first embodiment, the same parts are designated by the same reference numerals and detailed description thereof will be omitted.

The present invention is not limited to the above-described embodiment, and can be appropriately modified without departing from the spirit.

The present invention can also be realized, for example, by causing the CPU to execute a computer program for the process shown in FIG.

Programs can be stored and supplied to a computer using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, It includes a CD-R / W and a semiconductor memory (for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (random access memory)).

The program may be supplied to the computer by various types of transient computer readable media. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

1 Gripping device, 2 Image acquisition unit, 3 Robot arm, 4 Gripping execution unit, 5 Storage unit, 6 Grip position setting unit, 7 Success / failure judgment unit, 8 Feature amount calculation unit, 9 Notification unit

Claims (7)

A gripping means for gripping an object,
An image acquisition means for acquiring image information of an object to be determined, and
A feature amount calculation means for calculating a feature amount of image information of an object to be determined acquired by the image acquisition means, and a feature amount calculation means.
When the gripping means can grip the object and the deformation amount of the object is less than a predetermined amount and the grip is small, the gripping means can grip the object and the deformation amount of the object is a predetermined amount or more. Gripping is completed from the start of gripping the object by the gripping means acquired in advance for each of the gripping success / failure patterns including the large first gripping failure and the second gripping failure in which the gripping means cannot grip the object. A storage means for storing time-series data of feature quantities of image information up to
Each of the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping by the gripping means stored in the storage means, and the object to be determined by the gripping means calculated by the feature amount calculation means. The correlation between the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping is calculated, and the gripping success / failure pattern of the object to be determined corresponds to the time-series data of the feature amount having the highest correlation. A success / failure determination means for determining the attached gripping success / failure pattern,
A gripping device comprising. The gripping device according to claim 1.
When the success / failure determination means determines that the gripping success / failure pattern is successful, the gripping means is controlled so that the gripping means executes the next task, and the success / failure determination means performs the first gripping. Further provided with a control means for controlling to stop the operation of the gripping means when it is determined that the gripping success / failure pattern is a failure.
A gripping device characterized in that. The gripping device according to claim 1 or 2.
The storage means further
The gripping means can move the object, and the deformation amount of the object is less than a predetermined amount and the movement is small. The gripping means can move the object, and the deformation amount of the object is small. For each of the movement success / failure patterns including the first movement failure that is larger than a predetermined amount and the second movement failure that the gripping means cannot move the object, the object by the gripping means acquired in advance. The time-series data of the feature amount of the image information from the completion of gripping to the completion of movement of the object by the gripping means after the completion of gripping is stored.
The success / failure determination means
Each of the time-series data of the feature amount of the image information from the completion of gripping the object to the completion of movement by the gripping means stored in the storage means and the object to be determined by the gripping means calculated by the feature amount calculation means. The correlation between the time-series data of the feature amount of the image information from the completion of gripping the object to the completion of movement is calculated, and the gripping success / failure pattern of the object to be determined corresponds to the time-series data of the feature amount having the highest correlation. Judge as the attached movement success / failure pattern,
A gripping device characterized in that. The gripping device according to any one of claims 1 to 3.
When the success / failure determination means determines that the grip success / failure pattern of the first grip failure is determined, the notification means for notifying the content of the first grip failure is further provided.
A gripping device characterized in that. A gripping means for gripping an object,
An image acquisition means for acquiring image information of the object and
A feature amount calculation means for calculating a feature amount of image information of an object acquired by the image acquisition means, and a feature amount calculation means.
When the gripping means can grip the object and the deformation amount of the object is less than a predetermined amount and the grip is small, the gripping means can grip the object and the deformation amount of the object is a predetermined amount or more. A success / failure determination means for determining a grip success / failure pattern including a large first grip failure and a second grip failure in which the grip means cannot grip the object.
It is a gripping device equipped with
The success / failure determination means
Based on the image information of the gripping means at the time of completion of gripping acquired by the image acquisition means, it is determined that the gripping means is gripping the object at the time of completion of gripping, and the feature amount calculation means calculates. When it is determined that the amount of change in the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping is less than the first predetermined change amount, it is determined that the gripping success / failure pattern of the gripping success is achieved. ,
Based on the image information of the gripping means at the time of completion of gripping acquired by the image acquisition means, it is determined that the gripping means is gripping the object at the time of completion of gripping, and the feature amount calculation means calculates. When it is determined that the amount of change in the time-series data of the feature amount of the image information from the start of gripping the object to the completion of gripping is equal to or greater than the first predetermined change amount, it is the gripping success / failure pattern of the first gripping failure. Judging that
When it is determined that the gripping means is not gripping the object at the time of the completion of gripping based on the image information of the gripping means at the time of completion of gripping acquired by the image acquisition means, the gripping of the second gripping failure Judge as a success / failure pattern,
A gripping device characterized in that. Steps to acquire image information of the object to be judged, and
The step of calculating the feature amount of the image information of the acquired object to be determined, and
The gripping means can grip the object and the deformation amount of the object is small less than a predetermined amount, and the gripping means succeeds in gripping the object. The gripping means can grip the object and the deformation amount of the object is large when the deformation amount is more than a predetermined amount. From the start to the completion of gripping the object by the gripping means acquired in advance for each of the gripping success / failure patterns including the first gripping failure and the second gripping failure in which the gripping means cannot grip the object. Steps to store time-series data of feature quantities of image information,
Each of the time-series data of the feature amount of the image information from the start of gripping the object by the gripping means to the completion of gripping, and from the start of gripping of the object to be determined by the calculated gripping means to the completion of gripping The correlation with the time-series data of the feature amount of the image information of is calculated, and the gripping success / failure pattern of the object to be determined is determined to be the gripping success / failure pattern associated with the time-series data of the feature amount having the highest correlation. Steps to do and
A gripping method comprising. The process of acquiring the image information of the object to be judged and
The process of calculating the feature amount of the image information of the acquired object to be determined, and
The gripping means can grip the object and the deformation amount of the object is small less than a predetermined amount, and the gripping means succeeds in gripping the object. The gripping means can grip the object and the deformation amount of the object is large when the deformation amount is more than a predetermined amount. From the start to the completion of gripping the object by the gripping means acquired in advance for each of the gripping success / failure patterns including the first gripping failure and the second gripping failure in which the gripping means cannot grip the object. Processing to store time-series data of feature quantities of image information,
Each of the time-series data of the feature amount of the image information from the start of gripping the object by the gripping means to the completion of gripping, and from the start of gripping of the object to be determined by the calculated gripping means to the completion of gripping The correlation with the time-series data of the feature amount of the image information of is calculated, and the gripping success / failure pattern of the object to be determined is determined to be the gripping success / failure pattern associated with the time-series data of the feature amount having the highest correlation. Processing to do and
A program characterized by having a computer execute.

Images