JP2021061014A - Learning device, learning method, learning model, detector, and gripping system - Google Patents

Abstract

To provide a gripping system for performing learning to properly determine a position and an attitude for gripping an object from an image on a computer.SOLUTION: A gripping system 1 includes a camera 18 for acquiring information on an object; a gripper 16 for gripping the object, and at least one controller 12 for inputting the information on the object into a neural network model and estimating information on a position and an attitude of gripping means. The gripper 16 grips the object on the basis of the information on the estimated position and attitude. The neural network model has been learned on the basis of gripping information of the object generated using at least either a VR(Virtual Reality) technology or an AR(Augmented Reality) technology.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning device, a learning method, a learning model, a detection device and a gripping system.

Currently, automation is being carried out using robots for various uses, and research and development is being widely carried out according to each use. Research on gripping an object using an arm attached to a robot has also been widely conducted. In order to grip an object with the gripper portion of the robot, it is necessary to determine the position and orientation of the gripper of the robot from the position of the object detected by using an object detection device, for example, an RGB-D camera. As a method for estimating the position and orientation of the gripper, a method using a neural network or deep learning widely used in other fields has been developed.

However, with conventional methods, it is difficult to obtain a high-dimensional gripping posture, and in particular, to predict information that is difficult to annotate in an image on a computer. There is also a method of recognizing a three-dimensional object using CAD, but a CAD model is required, and the gripping posture must be determined after recognizing the object, which is economically and time-consuming. Further, in the conventional method, there is no example in which the gripper learns a gripping method other than gripping an object from directly above.

Japanese Unexamined Patent Publication No. 2016-32086

Therefore, the present invention provides a learning device that performs learning for appropriately determining a position and a posture for gripping an object from an image on a computer.

A detection means that detects the position where the object exists and the information of the teaching tool that holds the object.
The information of the teaching tool detected by the detection means is converted into the position and orientation information of the teaching tool, and the position information of the object and the position and orientation of the teaching tool holding the object It is a learning model represented by a neural network model having a teacher data generation means and a plurality of layers that generates teacher data which is data associated with information, and is a position where a target object exists via the detection means. Is input, a learning unit that learns a learning model that outputs data on the position and posture of the gripping means capable of gripping the target object using the teacher data, and a learning unit.
A learning device equipped with.

According to the learning device according to the present invention, it is possible to perform learning for appropriately determining a position and a posture for gripping an object from an image on a computer.

The figure which shows the outline of the gripping system which concerns on one Embodiment. The block diagram which shows the function of the computer which concerns on one Embodiment. The figure which shows typically the teaching tool for the teacher data acquisition which concerns on one Embodiment. The figure which shows the layer of the learning model which concerns on one Embodiment. The flowchart which shows the processing of the learning phase which concerns on one Embodiment. The flowchart which shows the processing of the estimation phase which concerns on one Embodiment. The figure which shows the example of the target object which concerns on one Embodiment, and the estimated gripping position and posture.

Hereinafter, embodiments of the present invention will be described in more detail with reference to the drawings. The present embodiment is not limited to the present invention. In each figure, components having the same function are designated by the same reference numerals, and detailed description of the same components will not be repeated.

The learning device according to the present embodiment captures a target object, for example, when the target object (hereinafter referred to as a target object) is grasped by a gripper attached to the robot or a component of the robot. This is an attempt to learn a model that outputs information on the position and orientation of a gripper capable of grasping an object by using an image taken by a camera provided in the robot.

FIG. 1 is an object grasping system by a robot using a learning model learned by the learning device according to the present embodiment. As shown in FIG. 1, the gripping system 1 includes a computer 10, a controller 12, a robot 14, a gripper 16, and a camera 18.

The computer 10 is a computer connected to a controller 12, a gripper 16 and a camera 18 to control the entire gripping system 1, and has, for example, a CPU (Central Processing Unit), various memories, and a user interface. The learning model learned by the learning device is stored in the computer 10. The learning device may be mounted in the computer 10. Further, an accelerator such as a GPU (Graphical Processing Unit) that performs calculations for learning and for applying an actually measured value to a learning model may be mounted.

The controller 12 is a control device that receives a signal from the computer 10 and transmits a signal for controlling the robot 14 to the robot 14. The controller 12 converts the signal received from the computer 10 into a signal for controlling the robot 14, and transmits a signal for causing the robot 14 to operate.

The controller 12 can be omitted when converting signals in the computer 10. Further, the computer 10 and the controller 12 may be mounted on the robot 14. That is, the computer 10, the controller 12, and the robot 14 may be collectively configured as an integrated robot. These configurations are variable based on other factors such as resources and costs.

The robot 14 is a housing that supports a gripper 16 that grips an object. In the description of this embodiment, the robot 14 is mainly described as a support housing for the gripper 16, but the present invention is not limited to this, and the robot 14 may have other functions and may be used for other purposes.

The gripper 16 is a device (grip means) for gripping a target object by receiving information on a grippable position and posture from the computer 10, and is, for example, a gripping device including three movable claws. .. The claw portion has, for example, one or more joints, and can be controlled so that each joint can be operated. The gripper 16 is controlled by a computer 10, a controller 12, or a robot 14. That is, the gripper 16 is controlled by the control unit. When grasping the target object, it is controlled based on the position and posture information received from the computer 10.

The gripper 16 may be composed of not only a claw type but also a slide type or a lever type, and the number of claws is not limited to three and may be two or four or more. The gripper 16 may be physically or electrically connected to the robot 14 via, for example, an arm (not shown).

Further, the gripper 16 does not have to be connected to the computer 10. In this case, the gripper 16 may acquire information on the position and posture (three-dimensional direction) for gripping the target object via the controller 12 and the robot 14.

In the following description, the position of the gripper 16 is assumed to be the position of the gripper 16 with respect to the reference point. The reference point of the gripper 16 is a point for uniquely determining the position of the gripper 16 in the gripping system 1 when the position information is given to the gripper 16. For example, in the case of a gripper having three claws, the center point (or the center of gravity) of the tip of the claw at the initial position of these claws (for example, the position in the most extended state) may be used as the reference point of the gripper 16. Alternatively, the point where these claws are installed may be used as a reference point for the gripper 16. Alternatively, the point where the arm of the robot 14 and the gripper 16 are connected may be used as a reference point of the gripper 16, and is not limited thereto.

Further, the position of the gripper 16 may be adjusted by the operation of the arm of the robot 14, but in the following description, the control of the position and posture of the gripper 16 is the control of the position or posture of the gripper 16. And the concept including control of the movement of the arm of the robot 14.

The camera 18 is a capture device connected to the computer 10 for capturing an image of an object from a predetermined position and direction. The camera 18 is, for example, an RGB-D camera that shoots an object from a vertical direction, and is horizontal to an imaging surface of the camera 18 with a predetermined point as a reference (hereinafter, referred to as a reference point of the camera 18). The RGB value of the object at the position (x, y) and the position z (depth) perpendicular to the imaging surface of the camera 18 at each position (x, y) of the object are captured.

The camera 18 is not limited to the RGB-D camera as long as the device can capture the position (x, y, z) where the object exists in this way, so that the camera 18 can detect the three-dimensional position of another object. It may be a detection device. For example, it may be a device provided with a plurality of cameras so as to capture an object from two or three directions, and capable of capturing a three-dimensional position of the object. Further, the camera may not be a camera for photographing visible light, but may be another detection device capable of measuring an object in three dimensions.

In FIG. 1, the camera 18 is connected only to the computer 10, but the camera 18 is not limited to this. For example, it may be mounted on a computer 10, or may be installed or mounted on a robot 14 or the like. In this case, the camera 18 may be directly connected to the computer 10 or may be connected to the computer 10 via the robot 14 or the like.

The reference point of the camera 18 may be, for example, the position of the camera 18 or the initial position of the gripper 16. Not limited to this, a point at which the position (x, y, z) can be uniquely determined in the measured object may be used as a reference point of the camera 18. In this case, in principle, the camera 18 is in a fixed position within the gripping system 1. That is, it is desirable that the system for photographing the teacher data and the system for photographing the target object are the same system.

Alternatively, the camera 18 may be connected to, for example, the arm of the robot 14 together with the gripper 16 or may be attached to the gripper 16. At the same time, the reference point of the camera 18 can be set to be the same as the reference point of the gripper 16, or a coordinate system in which the position of each point of the object photographed by the camera 18 can be uniquely determined by the gripping system 1 separately. The reference point may be set as follows. In this case, for example, the position where the object exists in the gripping system 1 is calculated based on the position of the camera 18 with respect to the reference point of the camera 18 and the coordinates of each position of the object in the image captured by the camera 18. May be.

The reference point of the gripper 16 and the reference point of the camera 18 may be the same point. In this case, the three-dimensional coordinates in the gripping system 1 can be handled in the same coordinate system. Further, the posture of the gripper 16 or the like may be based on, for example, a posture parallel to the x-axis on the xy plane in these three-dimensional coordinate systems. As described above, the position and posture of the gripper 16 and the like may be any as long as they can be uniquely determined in the gripping system 1.

The system that captures the image that generates the teacher data and the system that grips the target object may be different, but in the system that grips the target object, for example, the distance between the camera 18 and the object and the reference of the camera 18. It is desirable that the relative positional relationship between the point and the reference point of the gripper 16 is equivalent to that of a system for capturing an image that generates teacher data. Even in other cases, it is possible to apply this embodiment by correcting the coordinates in the input image of the target object and the position and attitude information of the gripper 16 to be output.

When the camera 18 is fixed in the gripping system 1, as an example, the camera 18 is 75 cm vertically away from the plane on which the object is placed so that a plane with a width of 70 cm × 50 cm is photographed. It will be installed in the same position. This is just an example and does not exclude other installation methods.

FIG. 2 is an example of a block diagram showing the functions of the computer 10. As shown in FIG. 2, the computer 10 includes, for example, an input unit 100, a teacher data generation unit 102, a learning unit 104, an estimation unit 106, and an output unit 108. Further, it includes a teacher data storage unit 110 for storing teacher data and a learning model storage unit 112 for storing a learning model learned based on the teacher data. The configuration of the CPU and the like is not shown. The solid line in the figure shows the data flow in the learning phase, and the broken line shows the data flow in the grasping information estimation phase.

Data or the like is input to the computer 10 via the input unit 100. The input data is an image of the teaching tool 2 shown in FIG. 3 that grips the object captured by the camera 18 which is the teacher data in the learning phase. In the estimation phase, it is an image of the target object to be grasped. Further, a request for learning and a request for estimating grasping information are input via a user interface (not shown).

When the image of the teaching tool 2 is input to the computer 10 via the input unit 100, the teacher data generation unit 102 converts the input image data and generates the teacher data. The details of the conversion from the input image to the three-dimensional position and the three-dimensional posture provided in the teacher data will be described later.

The learning unit 104 generates a learning model for estimating gripping information using the teacher data stored in the teacher data storage unit 110. The learning unit 104 generates a learning model based on the data stored in the teacher data storage unit 110 based on a request from the user, and stores the generated learning model in the learning model storage unit 112. The learning unit 104 may output the status state during learning and the information that the learning is completed to the output unit 108.

The estimation unit 106 estimates the position and posture information for gripping the target object based on the learning model. For example, when a request for estimating gripping information is received via the input unit 100, the estimation unit 106 grippers the input target object image based on the learning model stored in the learning model storage unit 112. Information on the position (three-dimensional position) and the posture (three-dimensional direction) at which 16 can grip the target object is estimated. The estimated information may be displayed on the output unit 108, for example, or may be transmitted to the robot 14 or the gripper 16 via the output unit 108.

The information estimated by the estimation unit 106 can also be fed back as new teacher data. As shown by the arrow indicated by the alternate long and short dash line, the information estimated by the estimation unit 106 and the information on whether or not the gripper 16 could actually grasp the target object or the user confirms the information and the target object can be grasped. The result of predicting the presence or absence may be output to the teacher data storage unit 110 as teacher data.

In FIG. 2, it has been described that the learning unit 104 and the estimation unit 106 are in the same computer 10, but the present invention is not limited to this. That is, the learning unit 104 and the estimation unit 106 may be provided in separate computers based on the common learning model storage unit 112. Further, the teacher data storage unit 110 and the learning model storage unit 112 may not be provided in the computer 10, and may be provided in, for example, a database server via a network or the like.

In the case of the configuration as shown in FIG. 2, the computer 10 is a learning device that optimizes a learning model for acquiring position and posture information on which the gripper 16 can grip the target object, and is based on the learning model. It is also an estimation device that estimates the position and attitude information at which the gripper 16 can grip the target object.

FIG. 3 is a diagram showing an example of a teaching tool 2 used for creating a learning model according to the present embodiment. The teaching tool 2 includes a teaching tool main body 20 and a marker 22.

The teaching tool main body 20 is a tool operated by a human when creating teacher data, and by grasping the object by the teaching tool main body 20, the position and posture of the teaching tool 2 capable of grasping the object can be determined. It becomes possible to accumulate as teacher data.

The marker 22 is a marker fixed to the teaching tool main body 20. The position of the teaching tool 2 taken by the camera 18 can be obtained by extracting the position of the teaching tool main body 20, but the posture of the teaching tool 2 can be extracted only by acquiring the image of the teaching tool main body 20. Often difficult to do.

Therefore, by fixing the marker 22 to the teaching tool main body 20, the marker 22 is detected from the image, and the position (x, y, z) or posture (R, P, Y) of the marker 22 in the three-dimensional space is estimated. By doing so, it becomes possible to estimate the posture of the teaching tool 2. In FIG. 3, the marker 22 is composed of three different markers, but is not limited to this, and is composed of one or two markers as long as the reading accuracy can be ensured. May be good. In order to improve the accuracy, the direction based on the two vectors in the longitudinal direction and the lateral direction of the teaching tool main body 20 may be detected.

Further, the marker 22 is not limited to be used, and the position or posture of the teaching tool 2 may be extracted by using, for example, a 3-axis sensor, a 6-axis sensor, light, radio waves, sound waves, or the like. Further, in the above, the data that can be grasped is acquired as the teacher data, but the present invention is not limited to this, and the teaching tool 2 may be used to collect data that cannot be grasped. By doing so, it is possible to perform learning using not only positive data but also negative data.

The teaching tool reference point 24 is a point that serves as a reference for the position and posture of the teaching tool 2. The three-dimensional position (x, y, z) of the teaching tool 2 is measured with reference to the position of the teaching tool reference point 24. For example, the position of the teaching tool 2 is determined by the teacher data generation unit 102 finding the position of the teaching tool reference point 24 based on the position of the marker 22. More simply, the position of the center of the two end points of the teaching tool 2 may be the position (x, y) of the teaching tool 2. When the camera 18 is an RGB-D camera, the position z may be obtained from the measurement result.

The position of the teaching tool 2 may be expressed so as to be uniquely determined in the gripping system 1. For example, the position of the teaching tool reference point 24 with respect to the reference point of the gripper 16 described above may be relatively represented, or the position of the teaching tool reference point 24 with respect to the reference point of the camera 18 may be represented relatively. ..

The posture of the teaching tool 2 is detected by reading the marker 22. That is, the posture (R, P, Y) is detected as the roll angle R, the pitch angle P, and the yaw angle Y with reference to the longitudinal direction of the teaching tool main body 20 centering on the teaching tool reference point 24. For example, how much the lateral direction is tilted with respect to the longitudinal direction of the teaching tool main body 20 (R), how much the longitudinal direction is tilted from the horizontal direction (P), and the longitudinal direction and the lateral direction are horizontal. It is detected by how much it is rotating (Y). At this time, the teacher data generation unit 102 tilts the marker 22 in each direction by obtaining a perspective transformation matrix based on, for example, the three-dimensional direction of the teaching tool 2 installed in a predetermined posture on a horizontal plane. Is calculated, the information of the image of the teaching tool 2 taken by the camera 18 is converted into the attitude information.

Next, the operation in each phase will be described. First, a learning model for estimating information on the position and posture at which the gripper 16 can grip the target object will be described.

FIG. 4 is a diagram showing an example of a learning model in the present embodiment. As shown in FIG. 4, the learning model is composed of FCN (Fully Convolutional Network). FCN is a kind of convolutional network (hereinafter referred to as CNN: Convolutional Neural Network), and refers to learning in which operations between all layers are performed by convolutional operations. That is, it is a network in which there is no pooling layer and no layers that are fully connected. By using FCN, it is possible to configure a network whose configuration is not complicated and whose calculation cost is low. The learning model is not limited to FCN, and other networks may be used.

In this learning model, when three images of RGB of 200 × 200 pixels and one image of the depth map are input as input images, the position map (PLM: Predicted Location) predicted to be graspable is obtained. Map) and posture / depth map (PCM: Predicted Configuration Map) are output. The PLM and the PCM are information indicating whether or not each pixel photographed by the camera 18 can be grasped.

More specifically, the PLM is a map showing whether or not there is a possibility that the target object can be grasped when the gripper 16 is present at the two-dimensional position (x, y). For example, each pixel of PLM has a value close to 1 if the gripper 16 is present at the position (x, y) corresponding to the pixel, if the target object can be grasped, and 0.5 if not. It is a map that has a value close to or less than 0.5. This is because, as will be described later, the output PLM and PCM data are in the region (0,1) via the sigmoid function.

The PCM transfers the four-dimensional information of the depth and the attitude that the target object may be grasped by operating the gripper 16 to the depth and the attitude (z, R, P, Y) to the two-dimensional position (x, y). It is a mapping. Specifically, this PCM is generated as follows. Of the data detected by the camera 18, the four-dimensional information that can be actually grasped is clustered in 300 ways, for example, and prepared as teacher data. The possibility that the PCM can actually grasp the four-dimensional information of the estimated depth and orientation of the gripper 16 for each teacher data, that is, for each clustered depth and orientation by referring to this clustered teacher data. It is a mapping of a certain two-dimensional position (x, y).

Since there is almost infinite number of four-dimensional information that the gripper 16 may be able to grip the target object with respect to the target object, the combinations of depth and posture are clustered in the above 300 ways in order to alleviate these. Use the result. For this clustering, the k-means method is used as an example, but other general clustering methods may be used. In the present embodiment, the PCM is generated by using 301 kinds of teacher data including the teacher data when the object cannot be grasped.

Each layer in FIG. 4 will be described. S1 or S2 described above each layer represents a stride at the time of folding into the layer. If it is S1, it indicates that the stride is 1, and if it is S2, it indicates that the stride is 2. Here, as an example, a 3 × 3 convolution kernel is used for the layer represented by S1, and a 4 × 4 convolution kernel is used for the layer represented by S2. The numbers 16, 32, 64, and 302 below each layer indicate the number of channels in each layer.

The leftmost layer is an input layer, and data obtained by photographing a 200 × 200 (pixel) × 4 (channel) object is input. The next layer (hereinafter, excluding the input layer and the output layer, referred to as the first layer, the second layer, ... In order from the left) is the data of 4 channels input to this input layer and 3 × 3. It convolves with a kernel of size and performs an operation to generate data of the same size on 16 channels. That is, the first layer is a 200 × 200 × 16 layer. The padding at the time of folding is 1. Expressed in Chainer (registered trademark) pseudo code,
layer1 = chainer.functions.Convolution2d (4, 16, 3, stride = 1, pad = 1) (input)
It was.

More specifically, 16 channels of first layer data are generated by performing a convolution operation using 16 different 3 × 3 size kernels for each channel of the input image. That is, there are kernels for R (Red), G (Green), B (Blue), and D (Depth) images as one set of kernels, and kernels corresponding to each channel for each channel of the input image. Is convolved and integrated. Then, the result of convolution integration is combined as one image.

The composition is performed, for example, by adding a load to the images of each channel with a predetermined weighting. Weighting factors can also be included in each kernel, in which case the output channels are generated by convolving each channel with the corresponding kernel and then adding the outputs of each channel. By applying 16 such sets of kernels to the input image, the conversion from 4 channels to 16 channels is performed.

Further, in all layers except the final layer (11th layer), batch normalization is performed after the convolution operation, and ReLU (Rectified Linear Unit) is applied as an activation function. Here, the batch normalization refers to a process of normalizing so that the average of each channel is 0 and the variance is 1. ReLU indicates a conversion in which a negative value in the data is set to 0. Expressed in pseudo code,
layer1 = chainer.functions.relu (chainer.functions.BatchNormalization (layer1)).

The second layer is a 100 × 100 × 32 data layer in which the first layer and a 4 × 4 size kernel are convolved and calculated, and the number of channels is 32. Similarly, expressed in pseudo code,
layer2 = chainer.functions.Convolution2d (16, 32, 4, stride = 2, pad = 1) (layer1)
It was. For downsampling, up to the 6th layer of 25x25x64, stride 1 convolution by a 3x3 size kernel (without resizing) and stride 2 convolution by a 4x4 size kernel (downsampling) are performed. It is executed by alternating. Furthermore, as above,
Normalize and activate as layer2 = chainer.functions.relu (chainer.functions.BatchNormalization (layer2)).

After that, the process shifts to the upsampling process. Upsampling is performed by performing the reverse operation of downsampling. For example, the 7th layer
It is expressed as layer7 = chainer.functions.Deconvolution2d (64, 64, 4, stride = 2, pad = 1) (layer6). Then, upsampling is executed by performing the reverse operation of downsampling up to the eleventh layer having a size of 200 × 200 × 16. In the process of upsampling, normalization and activation are also performed, for example.
layer7 = chainer.functions.relu (chainer.functions.BatchNormalization (layer7)).

In the final layer, instead of the above activation, activation is performed using a sigmoid function. Then, by inputting the result of clustering the position and posture information that the target object can grasp as teacher data, the six-dimensional position and posture information that the target object can grasp the result obtained in the final layer is input. The network estimates (x, y, z, R, P, Y) and outputs it to the output layer. For example, 200 × 200 × 1 (x, y) data is output as PLM, and a total of 200 is a combination of 200 × 200 × 300 data that can be grasped as PCM and 200 × 200 × 1 data that cannot be grasped. (Z, R, P, Y) data of × 200 × 301 is output.

In the final layer, the PCM is output, for example, as an image as shown at the bottom of FIG. This image is mapping data indicating a two-dimensional position (x, y) at which the target object can be grasped at the depth and posture indicated by the teacher data for each clustered teacher data. For the mapping data for each teacher data, for example, those colored with different colors may be combined and output as an image showing one PCM. When these images are combined into one image, each position (x, y) may be colored with a color indicating the cluster having the highest output value. On the other hand, since PLM is a one-channel output, for example, the value output from the final layer may be converted into an image by a gray scale or another coloring method and output.

The above learning model is shown as an example, and learning and optimization may be performed by another network. For example, like a general CNN, learning may be performed by a network including a pooling layer and a fully connected layer.

Next, the learning phase for generating the learning model will be described. FIG. 5 is a flowchart showing the learning process in the present embodiment.

First, the object captured by the camera 18 and the image data of the teaching tool 2 are collected (step S10). This image data is acquired by inputting a plurality of positions and postures that can be grasped at each of the plurality of objects using the teaching tool 2 via the camera 18. As image data for conversion to teacher data, a plurality of gripping positions and gripping postures are input for each of the plurality of objects. The acquired teacher data is input to the teacher data generation unit 102 via the input unit 100.

As an example, when the number of objects is 7, each object is arranged at 12 arbitrary positions and arbitrary postures, and 100 grippable positions and attitudes are set for each arrangement. 12 × 100 = 1200 types of image data are collected, and as a whole, 7 × 12 × 100 = 8400 types of image data are collected.

Next, each acquired image data is converted into teacher data (step S11). The teacher data generation unit 102 converts the input image data and generates teacher data. The generated teacher data is stored in the teacher data storage unit 110. Further, for example, when a large amount of data is acquired as described above, the output information on the position and posture in which the target object can be gripped becomes almost infinite. Therefore, the teacher data generation unit 102 uses a clustering method such as k-means to obtain four-dimensional information of the position (z) and the posture (R, P, Y) at which the teaching tool 2 can grip the object, for example. It is classified into 300 types of grippable position and posture information. The results of these clusterings may also be stored in the teacher data storage unit 110.

One set of teacher data can grasp an image of a four-dimensional (for example, R, G, B, D) object and a six-dimensional (for example, x, y, z, R, P, Y) object. It is data including position and posture information and data associated with. That is, for a plurality of objects, data including a plurality of sets of the above-mentioned teacher data is stored in the teacher data storage unit 110.

Next, learning is performed (step S12). For the optimization of learning, for example, Adam (Adaptive Moment Estimation) is used. As learning parameters when Adam is used, α = 0.0004 and batch size may be 80 or the like. The learning optimization method is not limited to Adam, and other optimization methods such as NAG, Adagrad, RMSprop, and Adamelta may be used. In addition, pre-learning may be performed in order to obtain the initial value at the time of learning. Pre-learning may be performed using, for example, the VGG16 model.

Further, in order to suppress overfitting, data may be artificially generated by the method of Label-preserving transformation. This method, for example, creates artificial data with locally modified data labeled as grippable, and assumes that the data is also grippable position and posture information. It is a method.

For example, when there are two teacher data in which the positions and postures of the gripper 16 are close to each other, the position and posture information in the middle of these data is assumed to be the grippable position and posture information, and is new. Generate teacher data. By doing so, for example, the above-mentioned 8400 kinds of data may be enhanced about three times. By reinforcing the sample in this way, for example, it is determined that the sample can be gripped in the information of a certain position and posture, but cannot be gripped when only the roll angle is slightly deviated from the information of the position and posture. It is possible to suppress the possibility of causing such overfitting.

In learning, for example, the learning model is optimized using the following evaluation function. Specifically, supervised learning is performed using the following evaluation function, and for example, the kernel used for convolution is optimized between each layer. In the following, since the output image is the output of the sigmoid function as described above, the pixel values of the output image do not become 0 and 1.

ここで、ａは、倍率、ｎは、学習データの総数、Ｗ／Ｈは、それぞれ学習に用いる画像の幅／高さ、ｔは、ターゲットとなるＰＬＭ、ｙは、出力を示す。倍率ａは、例えば、２００である。 As an evaluation function of PLM

Here, a is the magnification, n is the total number of training data, W / H is the width / height of the image used for learning, t is the target PLM, and y is the output. The magnification a is, for example, 200.

ここで、Ｃは、クラス数、Ｓ_ｋは、ｔ_ｋ ^{（ｉ，ｊ）}＝１となるピクセルの総数、ｕは、ターゲットとなるＰＣＭ、ｙは、出力を示す。 As an evaluation function of PCM

Here, C is the number of classes, _{S k is} the total number of ^{t k (i, j) =} 1 and becomes a pixel, u is PCM to be targeted, y represents the output.

The training is executed by optimizing the learning model by backpropagation using the _{evaluation function L = L PLM} + λL _PCM represented by the evaluation function of PCM and the evaluation function of PLM. Here, λ is, for example, 200. By performing such learning, for example, each kernel that performs convolution is optimized. The evaluation function may be any function that can evaluate PLM and PCM, and is not limited to the above. Further, in the learning phase, the teacher data may be divided into two groups and cross-validated. The learned learning model is stored in the learning model storage unit 112.

By performing the above learning, the learning model shown in FIG. 4 is generated. Next, the estimation phase of the grippable position and posture information when the image of the target object is taken will be described. FIG. 6 is a flowchart showing the estimation process in the present embodiment.

First, the estimation unit 106 acquires an image of the target object taken by the camera 18 via the input unit 100 (step S20). In this image, when the camera 18 is an RGB-D camera, an image of each color component of R, G, and B shown on a plane and an image showing the respective depths in the captured image are acquired. To.

Next, the estimation unit 106 inputs the image acquired in step S20 into the learning model stored in the learning model storage unit 112 as an input image, and the position of the gripper 16 capable of gripping the target object and the position of the gripper 16 The PLM and PCM indicating the posture are acquired (step S21). The image of the target object acquired by the camera 18 is input to the leftmost input layer shown in FIG. The learning model including the convolutional network into which the image is input outputs the PCM and PLM for the target object.

The outputs of the PCM and the PLM may be output as an image via the output unit 108 as shown in FIG. The PLM image outputs the position of the gripper 16 at which the gripper 16 is likely to be able to grip the target object as a set of points. The PCM collates the gripper depth (distance in the vertical direction from the camera 18) and the four-dimensional data (z, R, P, Y) of the posture at each point with the result of clustering in 300 ways, and outputs the image. .. For example, as described above, each class is output as a different color on the image.

Next, the estimation unit 106 selects information having a high score from the depth and posture data of the gripper 16 output by the learning model, and outputs the information via the output unit 108 (step S22). As the score, for example, the output PLM and PCM maps themselves are referred to.

Next, by operating the gripper 16 that has received the gripper position and posture information that can be gripped from the computer 10 via the output unit 108, the robot 14 can grip the target object using the gripper 16 (step). S23).

FIG. 7A is an example of a target object. As shown in FIG. 7A, the target object is, for example, a bottle. 7 (b) and 7 (c) are diagrams showing the gripping position and posture of the target object estimated by the learning model described above. In these figures, the learning model is optimized by using seven kinds of objects for teacher data, and the target object that is not used as the object for teacher data is applied to the learning model.

FIG. 7 (b) shows the position and posture of gripping the target object from the upper surface, and FIG. 7 (c) is a cross-sectional view taken along the line AA'of FIG. 7 (b). In these FIGS. 7 (b) and 7 (c), the solid line indicates the position and posture of the gripper 16 based on the information on the grippable position and posture having the highest score, followed by the broken line. The one with the highest score is shown in the order of the alternate long and short dash line.

It can be read from these figures that the gripper 16 is capable of gripping the target object when the position and posture are based on the information of the grippable position and posture having the highest score. That is, the gripper 16 is located at the horizontal position shown in FIG. 7 (b) and the vertical position shown in FIG. 7 (c), and the posture of the gripper 16 is set so as to sandwich the target object in each figure. You can read that. Similarly, it can be read that the target object can be grasped for the broken line and the alternate long and short dash line.

In the examples shown in FIGS. 7 (b) and 7 (c), first, the point (x, y) having the highest PLM output value is extracted as the score, and the point (x, y) is used. Information on three grippable positions and postures is shown in descending order of PCM score. The extraction of the grippable position and posture is not limited to this, and may be extracted by evaluating the output values of PLM and PCM by a predetermined evaluation function. For example, the position and posture in which the product of the output values of PLM and PCM is the highest may be simply output, or the position and posture in which the weighted average value of PLM and PCM by a predetermined weighting is the highest is output. It may be.

As described above, according to the present embodiment, by using a learning model optimized based on multidimensional, for example, 6-dimensional teacher data, it is adapted to a high degree of freedom, in other words, a multidimensional degree of freedom. It is possible to estimate information on the grippable position and posture. In the above-described embodiment, it is assumed to be six-dimensional, but it is higher-dimensional by using teacher data based on other parameters such as the degree of bending of the gripper 16 at the nail joint and the distance between the nails. It is possible to estimate the information indicating the grippable state with respect to the degree of freedom of. By increasing the degree of freedom in this way, it is possible to adopt the learning method according to the present embodiment even when gripping means having various shapes are used.

For example, when learning the movement of the joint of the gripper 16 and outputting it as grippable information, the teaching tool may be deformed according to the shape of the gripper 16. For example, it is possible to have one or more joints in the claw portion of the teaching tool, and to capture the movement of the joints with the camera 18 and use it as teacher data. As another example, when it is desired to limit the distance between the claws of the gripper 16, the distance between the claws of the teaching tool may be made equal to the distance between the claws of the gripper 16. Further, when the degree of freedom of the posture of the gripper 16 is 2, for example, another degree of freedom such as the distance between these claws and the length of the claws is added to learn as a degree of freedom of 6 dimensions or more. , The grippable data may be output. As described above, the learning device in the present embodiment makes it possible to perform learning with a degree of freedom of 6 dimensions or more and estimate data.

As a method for creating teacher data, a teaching tool 2 having a shape different from that of the gripper 16 may be used as in the above-described embodiment, or as another example, a teaching tool having the same shape as the gripper 16 may be used. .. In this case, it is possible to eliminate or reduce the shape error between the teaching tool 2 and the gripper 16, and avoid the problem that the teaching tool 2 can grip the teacher data but the actual gripper 16 cannot grip the data. Is possible. When a teaching tool having the same shape as the gripper 16 is used, the teacher data may be collected by a human being actually operating the robot 14 using the operation system of the robot 14.

Furthermore, using a robot that can be deformed by a human being applying a physical force from the outside, such as a collaborative robot, the gripper 16 as a teaching tool is moved to a position and posture at which an object can actually be grasped. The state of the position and the posture may be used as the teacher data. In this case, the position and posture of the gripper 16 which is a teaching tool may be acquired based on what was taken by the camera 18, as in the above-described embodiment. Further, when the information on the position and posture of the gripper 16 can be acquired via the robot 14, not only the camera 18 can take a picture of the gripper 16, but also the camera 18 can take a picture of an object and hold the gripper position and the position. The posture information may be acquired via the robot 14 and may be used as teacher data by associating the image information of the object with the grippable position and posture information.

Furthermore, when acquiring teacher data, instead of grasping the actual object, the information of the object is captured by the camera 18, and the object is virtually virtualized using VR (Virtual Reality) technology or AR (Augmented Reality) technology. Alternatively, it may be defined in an augmented manner so that a human can operate the teaching tool 2 or the gripper 16 to acquire information on the position and posture at which the virtual object or the augmented object can be grasped. By doing so, it is possible to reduce the cost of the idiomatic construction for teaching to acquire the teacher data, and it is easy for a plurality of people to teach the situation of the same object. It is possible to suppress the bias of teaching due to.

Further, in the present embodiment, by outputting the position and posture information estimated by the learning model as an image, the position and posture information that can be automatically grasped for the target object photographed by the camera 18 is estimated. Later, the user can select the position and posture information that can be easily grasped from the output candidates.

Further, in the present embodiment, since the learning model is based on FCN, it is possible to perform calculations at high speed, and it is possible to reduce the time cost especially in the learning phase. This is also effective when, for example, whether or not the estimated position and posture information can be actually grasped is used as new teacher data.

The gripping system 1 according to the present embodiment may be used for picking a large amount or a wide variety of objects such as picking of goods in a distribution warehouse or picking in a factory. Further, it may be used for remote control when it is difficult for the user to approach the object or when the user does not want to approach the object, for example, picking an object in a clean room. Further, when the robot 14 works using a tool, it may be applied to grip the tool.

In the description of the above-described embodiment, words indicating directions such as vertical, horizontal, and vertical are used, but these may be defined in any direction as long as they can be relatively determined in the gripping system 1. .. For example, the camera 18 may be photographed in the gripping system 1 in a horizontal direction with respect to gravity. In this case, as an example, the x-axis is parallel to the imaging surface of the camera 18 and with respect to the direction of gravity. It may be defined as a horizontal axis, the y-axis may be defined as an axis parallel to the imaging surface and perpendicular to the x-axis, and the z-axis may be defined as an axis perpendicular to the imaging surface of the camera 18. Furthermore, the axes do not have to be orthogonal in a strict sense as long as they are linearly independent axes that can uniquely determine the three-dimensional position in the gripping system 1.

The coordinate system is not limited to the Cartesian coordinate system according to the situation to be implemented, and other coordinate systems such as a cylindrical coordinate system can also be used. The same applies to the posture, as long as it can be uniquely determined within the gripping system 1, it is not limited to the above-mentioned R, P, Y, and is represented by using, for example, Euler angles, declinations, and direction cosine. It may be what is done. In this way, any device can be used as long as the position and orientation of the teaching tool 2 and the gripper 16 can be uniquely determined by the three-dimensional position and the three-dimensional posture, and a tool that is easy to use in system design is selected. It is possible.

In all the above descriptions, at least a part of the gripping system 1 may be composed of hardware, or may be composed of software, and may be implemented by a CPU or the like by information processing of the software. When composed of software, even if the gripping system 1 and a program that realizes at least a part of the functions are stored in a storage medium such as a flexible disk or a CD-ROM and read by a computer for execution. Good. The storage medium is not limited to a removable one such as a magnetic disk or an optical disk, and may be a fixed storage medium such as a hard disk device or a memory. That is, information processing by software may be concretely implemented using hardware resources. Further, the processing by software may be implemented in a circuit such as FPGA (Field-Programmable Gate Array) and executed by hardware. The generation of the learning model and the processing after inputting to the learning model may be performed using, for example, an accelerator such as a GPU.

Further, the learning model according to the present embodiment can be used as a program module which is a part of artificial intelligence software. That is, the CPU of the computer 10 calculates the image data taken by the camera 18 input to the input layer of the convolutional network based on the model stored in the learning model storage unit 112, and calculates the image data of the convolutional network. It works to output the result from the output layer.

Based on all the above descriptions, those skilled in the art may be able to conceive of additions, effects or various modifications of the present invention, but aspects of the present invention are not limited to the individual embodiments described above. Absent. Various additions, changes and partial deletions are possible without departing from the conceptual idea and purpose of the present invention derived from the contents defined in the claims and their equivalents.

1: Gripping system 10: Computer 102: Teacher data generation unit 104: Learning unit 106: Estimating unit 14: Robot 16: Gripper 18: Camera 2: Teaching tool

Claims (11)

A detection device that acquires object information,
A gripping means for gripping the object and
It comprises at least one processor that inputs information about the object into a neural network model and estimates information about the position and orientation of the gripping means.
The gripping means grips the object based on the estimated information about the position and posture.
The neural network model is learned based on the gripping information of an object generated by using at least one of VR (Virtual Reality) technology and AR (Augmented Reality) technology.
Gripping system. A controller that controls the gripping means based on the estimated information about the position and the posture is further provided.
The gripping system according to claim 1. The detection device is installed in the gripping means,
The gripping system according to claim 1 or 2. The detection device is a camera capable of acquiring distance information.
The gripping system according to any one of claims 1 to 3. The detection device is one or more cameras.
The gripping system according to any one of claims 1 to 3. The estimated information about the posture includes information capable of expressing rotation angles around a plurality of axes.
The gripping system according to any one of claims 1 to 5. The output of each layer of the neural network model contains information other than the position, orientation and area of the object.
The gripping system according to any one of claims 1 to 6. A model generator that generates a neural network model that outputs information about the position and orientation of the gripping means when information about an object is input.
The neural network model is learned based on the gripping information of the object generated by using at least one of the VR technique and the AR technique.
Model generator. By at least one processor
It is a model generation method that generates a neural network model that outputs information about the position and posture of the gripping means when information about an object is input.
The neural network model is learned based on the gripping information of the object generated by using at least one of the VR technique and the AR technique.
Model generation method. The information regarding the posture output by the neural network includes information capable of expressing rotation angles around a plurality of axes.
The model generation method according to claim 9. The output of each layer of the neural network model contains information other than the position, orientation and area of the object.
The model generation method according to claim 9 or 10.

Images