JP2021070122A - Learning data generation method - Google Patents

Abstract

To readily generate learning data that is used when a machine learns a gripping operation.SOLUTION: A learning data generation method includes a step of generating an image of works loaded in bulk with an image generation software to use the image as learning data for machine learning.SELECTED DRAWING: Figure 1

Description

The present invention relates to a learning data generation method.

In order to grip a plurality of objects (workpieces) stacked separately by a robot arm or the like, a plurality of work models are stacked in a virtual work space that is a virtual work space that models the three-dimensional shape of the work, and the pieces are separated. A technique for performing a simulation for verifying a stacking picking operation is known.

JP-A-2018-144152

However, it takes a long time to model a large number of real works stacked separately in three dimensions and to simulate the picking operation.

The present invention is an example of the above problem, and is a learning data generation method for creating learning data for machine learning a gripping motion by a robot arm or the like by using image generation software, and using this learning data. It is an object of the present invention to provide a machine learning method and a gripping device using this machine learning method.

In the learning data generation method according to one aspect of the present invention, images of works stacked separately by image generation software are generated and used as learning data for machine learning.

According to one aspect of the present invention, learning data used for machine learning of gripping motion can be easily generated.

FIG. 1 is a diagram showing an example of an object grasping system equipped with the image processing device according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of the object gripping system according to the first embodiment. FIG. 3 is a flowchart showing an example of the learning process. FIG. 4 is a diagram showing an example of three-dimensional data of an object. FIG. 5 is a diagram showing an example of a captured image of a virtual space in which a plurality of objects are arranged. FIG. 6 is a diagram showing an example of processing related to control of the robot arm. FIG. 7 is a diagram showing another example of processing related to control of the robot arm. FIG. 8 is a diagram showing an example of a detection model according to the first embodiment. FIG. 9 is a diagram showing an example of a feature map output by the feature detection layer (u1) according to the first embodiment. FIG. 10 is a diagram showing an example of an estimation result of the position and posture of the object according to the first embodiment. FIG. 11 is a diagram showing another example of the estimation result of the gripping position of the object according to the first embodiment. FIG. 12 is a diagram showing an example of loosely stacked images taken by the stereo camera according to the first embodiment. FIG. 13 is a diagram showing an example of the relationship between the loosely stacked images and the matching map according to the first embodiment. FIG. 14 is a flowchart showing an example of the estimation process according to the first embodiment. FIG. 15 is a diagram showing an example of the estimation process according to the first embodiment. FIG. 16 is a diagram showing an example of a loosely stacked image including a tray according to a modified example. FIG. 17 is a diagram showing an example of a position deviation estimation model according to a modified example. FIG. 18 is a diagram showing another example of the position deviation estimation model according to the modified example.

Hereinafter, the image processing apparatus and the image processing method according to the embodiment will be described with reference to the drawings. The present invention is not limited to this embodiment. In addition, the relationship between the dimensions of each element in the drawing, the ratio of each element, and the like may differ from reality. Even between drawings, there may be parts where the relationship and ratio of dimensions are different from each other. Further, in principle, the contents described in one embodiment or modification are similarly applied to other embodiments or modifications.

(First Embodiment)
The image processing apparatus in the first embodiment is used, for example, in the object grasping system 1. FIG. 1 is a diagram showing an example of an object grasping system equipped with the image processing device according to the first embodiment. The object grasping system 1 shown in FIG. 1 includes an image processing device 10 (not shown), a camera 20, and a robot arm 30. The camera 20 is provided at a position where, for example, both the robot arm 30 and the loosely stacked workpieces 41, 42, etc., which are objects to be gripped by the robot arm 30, can be photographed. The camera 20 captures, for example, images of the robot arm 30 and the works 41 and 42 and outputs them to the image processing device 10. The robot arm 30 and the workpieces 41, 42 and the like stacked separately may be photographed by different cameras. As the camera 20 in the first embodiment, as shown in FIG. 1, a camera capable of capturing a plurality of images, such as a known stereo camera, is used. The image processing device 10 estimates the positions and orientations of the works 41, 42, etc. using the image output from the camera 20. The image processing device 10 outputs a signal for controlling the operation of the robot arm 30 based on the estimated positions and postures of the works 41, 42, and the like. The robot arm 30 performs an operation of gripping the works 41, 42, and the like based on the signal output from the image processing device 10. Although a plurality of different types of works 41, 42 and the like are disclosed in FIG. 1, the type of work may be one type. In the first embodiment, a case where there is only one type of work will be described. Further, the works 41, 42 and the like are arranged so that their positions and postures are irregular. As shown in FIG. 1, for example, a plurality of workpieces may be arranged so as to overlap each other in a top view. The works 41 and 42 are examples of objects.

FIG. 2 is a block diagram showing an example of the configuration of the object gripping system according to the first embodiment. As shown in FIG. 2, the image processing device 10 is communicably connected to the camera 20 and the robot arm 30 through the network NW. Further, as shown in FIG. 2, the image processing device 10 includes a communication I / F (interface) 11, an input I / F 12, a display 13, a storage circuit 14, and a processing circuit 15.

The communication I / F 11 controls data input / output communication with an external device through the network NW. For example, the communication I / F 11 is realized by a network card, a network adapter, a NIC (Network Interface Controller), etc., receives image data output from the camera 20, and transmits a signal to be output to the robot arm 30.

The input I / F 12 is connected to the processing circuit 15, converts the input operation received from the administrator (not shown) of the image processing device 10 into an electric signal, and outputs the input operation to the processing circuit 15. For example, the input I / F12 is a switch button, a mouse, a keyboard, a touch panel, or the like.

The display 13 is connected to the processing circuit 15 and displays various information and various image data output from the processing circuit 15. For example, the display 13 is realized by a liquid crystal monitor, a CRT (Cathode Ray Tube) monitor, a touch panel, or the like.

The storage circuit 14 is realized by, for example, a storage device such as a memory. The storage circuit 14 stores various programs executed by the processing circuit 15. Further, the storage circuit 14 temporarily stores various data used when various programs are executed by the processing circuit 15. The storage circuit 14 has a machine (deep) learning model 141. Further, the machine (deep) learning model 141 includes a neural network structure 141a and learning parameters 141b. The neural network structure 141a is an application of a known network such as the convolutional neural network b1 of FIG. 8, and is a network structure shown in FIG. 15 described later. The learning parameter 141b is, for example, the weight of the convolutional filter of the convolutional neural network, and is a parameter that is learned and optimized for estimating the position and orientation of the object. The neural network structure 141a may be provided in the estimation unit 152. The machine (deep) learning model 141 in the present invention will be described by taking a trained model as an example, but the present invention is not limited to this. In the following, the machine (deep) learning model 141 may be simply referred to as a “learning model 141”.

The learning model 141 is used in a process of estimating the position and orientation of the work from the image output from the camera 20. The learning model 141 is generated, for example, by learning the positions and postures of a plurality of works and images of the plurality of works as teacher data. In the first embodiment, the learning model 141 is generated by, for example, the processing circuit 15, but the learning model 141 is not limited to this, and may be generated by an external computer. In the following, an embodiment in which the learning model 141 is generated and updated by a learning device (not shown) will be described.

In the first embodiment, the large number of images used to generate the learning model 141 may be generated by, for example, arranging a plurality of works in the virtual space and capturing the images in the virtual space. FIG. 3 is a flowchart showing an example of the learning process. As shown in FIG. 3, the learning device acquires three-dimensional data of the object (step S101). The three-dimensional data can be acquired by a method such as a known 3D scan. FIG. 4 is a diagram showing an example of three-dimensional data of an object. By acquiring the three-dimensional data, the posture of the work can be arbitrarily changed and arranged in the virtual space.

Next, the learning device sets various conditions for arranging the object in the virtual space (step S102). The object can be placed in the virtual space by using, for example, known image generation software or the like. Conditions such as the number, position, and posture of the objects to be arranged can be set so that the image generation software randomly generates the objects, but the present invention is not limited to this, and the administrator of the image processing device 10 arbitrarily sets the conditions. You may. Next, the learning device arranges the object in the virtual space according to the set conditions (step S103). Next, the learning device acquires an image, a position, and a posture of the arranged objects by capturing, for example, a virtual space in which a plurality of objects are arranged (step S104). In the first embodiment, the position and orientation of the object are indicated by, for example, three-dimensional coordinates (x, y, z), and the attitude of the object is a quaternion which is a quaternion representing the attitude or rotational state of the object. It is indicated by (qx, qy, qz, qw). FIG. 5 is a diagram showing an example of a captured image of a virtual space in which a plurality of objects are arranged. As shown in FIG. 5, a plurality of objects W1a and W1b are arranged at random positions and postures in the virtual space, respectively. Further, in the following, an image of randomly arranged objects may be referred to as a “separately stacked image”. Next, the learning device stores the acquired image and the position and orientation of the arranged object in the storage circuit 14 (step S105). Further, the learning device repeats steps S102 to S105 a predetermined number of times (step S106). Here, the combination of the image acquired in the above step and the position and posture in which the object is arranged, which is stored in the storage circuit 14, may be referred to as “teacher data”. By repeating the processes from step S102 to step S105 a predetermined number of times, a sufficient number of teacher data is generated to repeat the learning process.

Then, the learning device generates or updates the learning parameter 141b used as weighting in the neural network structure 141a by performing the learning process a predetermined number of times using the generated teacher data (step S107). By arranging the object from which the three-dimensional data has been acquired in this way on the virtual space, it is easy to generate teacher data including the image of the object and the combination of the position and the posture used in the learning process. can do.

Returning to FIG. 2, the processing circuit 15 is realized by a processor such as a CPU (Central Processing Unit). The processing circuit 15 controls the entire image processing device 10. The processing circuit 15 reads various programs stored in the storage circuit 14 and executes the read programs to execute various processes. For example, the processing circuit 15 includes an image acquisition unit 151, an estimation unit 152, and a robot control unit 153.

The image acquisition unit 151 acquires the separately stacked images through the communication I / F11, for example, and outputs the images to the estimation unit 152. The image acquisition unit 151 is an example of an acquisition unit.

The estimation unit 152 estimates the position and orientation of the object using the output loose-stacked image. The estimation unit 152 performs estimation processing on an image of an object using, for example, the learning model 141, and outputs the estimation result to the robot control unit 153. The estimation unit 152 may further estimate the position and orientation of the tray or the like on which the object is placed, for example. The configuration for estimating the position and orientation of the tray will be described later.

The robot control unit 153 generates a signal for controlling the robot arm 30 based on the estimated position and orientation of the object, and outputs the signal to the robot arm 30 through the communication I / F 11. The robot control unit 153 acquires, for example, information regarding the current position and posture of the robot arm 30. Then, the robot control unit 153 generates a trajectory that the robot arm 30 moves when gripping the object according to the current position and posture of the robot arm 30 and the estimated position and posture of the object. The robot control unit 153 may correct the trajectory in which the robot arm 30 moves based on the position and posture of the tray or the like.

FIG. 6 is a diagram showing an example of processing related to control of the robot arm. As shown in FIG. 6, the estimation unit 152 estimates the position and orientation of the target object from the images stacked separately. Similarly, the estimation unit 152 may estimate the position and orientation of the tray or the like on which the object is placed from the images stacked separately. The robot control unit 153 calculates the coordinates and posture of the hand position of the robot arm 30 based on the estimated model of the object and the tray, and generates the trajectory of the robot arm 30.

After the robot arm 30 grips the object, the robot control unit 153 may further output a signal for controlling the operation of the robot arm 30 for aligning the gripped object. FIG. 7 is a diagram showing another example of processing related to control of the robot arm. As shown in FIG. 7, the image acquisition unit 151 acquires an image of an object gripped by the robot arm 30 taken by the camera 20. The estimation unit 152 estimates the position and posture of the target object gripped by the robot arm 30, and outputs the position and orientation to the robot control unit 153. Further, the image acquisition unit 151 may further acquire an image of a tray or the like of the alignment destination, which is a movement destination of the grasped object, taken by the camera 20. At that time, the image acquisition unit 151 further acquires an image (arranged image) of the object already aligned on the tray or the like of the alignment destination. The estimation unit 152 estimates the position and orientation of the tray and the like to be aligned, and the position and orientation of the already aligned objects from the aligned image or the aligned image. Then, the robot control unit 153 determines the estimated position and orientation of the object held by the robot arm 30, the position and orientation of the tray or the like to be aligned, and the position and orientation of the already aligned object. Based on this, the coordinates and posture of the hand position of the robot arm 30 are calculated, and the trajectory of the robot arm 30 when aligning the objects is generated.

Next, the estimation process in the estimation unit 152 will be described. The estimation unit 152 extracts the feature amount of the object by using, for example, a model to which an object detection model having known downsampling, upsampling, and skip connection is applied. FIG. 8 is a diagram showing an example of a detection model according to the first embodiment. In the object detection model shown in FIG. 8, for example, the d1 layer divides the loosely stacked image P1 (320 × 320 pixels) into 40 × 40 grids in length and width by downsampling via a convolutional neural network b1, and has a plurality of features for each grid. Calculate the amount (for example, 256 types). Further, in the d2 layer, which is a layer lower than the d1 layer, the grid divided by the d1 layer is divided coarser than the d1 layer (for example, 20 × 20 grids), and the feature amount of each grid is calculated. Similarly, the d3 layer and the d4 layer, which are lower layers than the d1 layer and the d2 layer, divide the grid divided by the d2 layer more coarsely, respectively. The d4 layer calculates the feature amount in a finer division by upsampling, and at the same time, integrates with the feature amount of the d3 layer by the skip connection s3 to generate the u3 layer. The skip connection may be a simple addition or a connection of features, or a transformation such as a convolutional neural network may be added to the features of the d3 layer. Similarly, the feature amount calculated by upsampling the u3 layer and the feature amount of the d2 layer are integrated by the skip connection s2 to generate the u2 layer. Further, the u1 layer is generated in the same manner. As a result, in the u1 layer, the feature amount of each grid divided into 40 × 40 grids is calculated as in the d1 layer.

FIG. 9 is a diagram showing an example of a feature map output by the feature extraction layer (u1) according to the first embodiment. The horizontal direction of the feature map shown in FIG. 9 indicates each grid in the horizontal direction of the loosely stacked image P1 divided into 40 × 40 grids, and the vertical direction indicates each grid in the vertical direction. Further, the depth direction of the feature map shown in FIG. 9 indicates an element of the feature amount in each grid.

FIG. 10 is a diagram showing an example of an estimation result of the position and posture of the object according to the first embodiment. As shown in FIG. 10, the estimation unit includes two-dimensional coordinates (Δx, Δy) indicating the position of the object, quaternions (qx, qy, qz, qw) indicating the posture of the object, and a classification score (C0). , C1, ..., Cn) is output. In the first embodiment, as an estimation result, among the coordinates indicating the position of the object, the value of the depth indicating the distance from the camera 20 to the object is not calculated. The configuration for calculating the depth value will be described later. The depth referred to here means the distance from the z-coordinate of the camera to the z-coordinate of the object in the z-axis direction parallel to the optical axis of the camera. The classification score is a value output for each grid, and is the probability that the center point of the object is included in the grid. For example, when there are n types of objects, the score of n + 1 class classifications is output by adding the "probability that the center point of the object is not included". For example, when there is only one type of target work, the scores of two classifications are output. Also, when there are a plurality of objects in the same grid, the probability of the objects stacked on top is output.

In FIG. 10, the point C indicates the center of the grid Gx, and the point ΔC which is the coordinates (Δx, Δy) indicates, for example, the center point of the detected object. That is, in the example shown in FIG. 10, the center of the object is offset from the center point C of the grid Gx by Δx in the x-axis direction and Δy in the y-axis direction.

Instead of FIG. 10, arbitrary points a, b, and c other than the center of the object are set as shown in FIG. 11, and arbitrary points a, b, and c from the center point C of the grid Gx are set. The coordinates (Δx1, Δy1, Δz1, Δx2, Δy2, Δz2, x3, Δy3, Δz3) may be output. Any point may be set at any position of the object, and may be one point or a plurality of points.

If the grid division is coarser than the size of the object, a plurality of objects may be included in one grid, and the features of the objects may be mixed and erroneously detected. In the embodiment of the above, only the feature map which is the output of the feature extraction layer (u1) in which the finally generated fine (40 × 40 grid) feature amount is calculated is used.

Further, in the first embodiment, the distance from the camera 20 to the object is specified by taking two types of images on the left and right, for example, using a stereo camera. FIG. 12 is a diagram showing an example of loosely stacked images taken by the stereo camera according to the first embodiment. As shown in FIG. 12, the image acquisition unit 151 acquires two types of loosely stacked images, the left image P1L and the right image P1R. Further, the estimation unit 152 performs estimation processing using the learning model 141 on both the left image P1L and the right image P1R. When performing the estimation process, a part or all of the learning parameters 141b used for the left image P1L may be shared as weighting for the right image P1R. It should be noted that, instead of using a stereo camera, one camera may be used, and the positions of the cameras may be shifted to capture images corresponding to two types of left and right images at two locations.

Therefore, the estimation unit 152 in the first embodiment suppresses erroneous recognition of the object by using a matching map in which the feature amount of the left image P1L and the feature amount of the right image P1R are combined. In the first embodiment, the matching map shows the strength of the correlation between the right image P1R and the left image P1L for each feature amount. That is, by using the matching map, it is possible to match the left image P1L and the right image P1R by paying attention to the feature amount in each image.

FIG. 13 is a diagram showing an example of the relationship between the loosely stacked images and the matching map according to the first embodiment. As shown in FIG. 13, in the matching map ML that is based on the left image P1L and corresponds to the right image P1R, the feature amount of the grid including the center point of the object W1L of the left image P1L and the right image. The grid MLa having the largest correlation with the feature amount contained in P1R is highlighted and displayed. Similarly, in the matching map MR that is based on the right image P1R and corresponds to the left image P1L, the feature amount of the grid including the center point of the object W1R of the right image P1R and the left image P1L are included. The grid MRa, which has the largest correlation with the feature amount, is highlighted and displayed. Further, the grid MLa having the largest correlation in the matching map ML corresponds to the grid on which the object W1L in the left image P1L is located, and the grid MRa having the largest correlation in the matching map MR is the position of the object W1R in the right image P1R. Corresponds to the grid to be. Thereby, it can be specified that the grid on which the object W1L is located in the left image P1L and the grid on which the object W1R is located in the right image P1R match. That is, in FIG. 12, the matching grids are the grid G1L of the left image P1L and the grid G1R of the right image P1R. As a result, the parallax with respect to the object W1 can be specified based on the X coordinate of the object W1L in the left image P1L and the X coordinate of the object W1R in the right image P1R. Can be identified.

FIG. 14 is a flowchart showing an example of the estimation process according to the first embodiment. Further, FIG. 15 is a diagram showing an example of the estimation process according to the first embodiment. Hereinafter, description will be made with reference to FIGS. 12 to 15. First, the image acquisition unit 151 acquires the left and right images of the object as shown in the left image P1L and the right image P1R shown in FIG. 12 (step S201). Next, the estimation unit 152 calculates the feature amount for each grid in the horizontal direction of each of the left and right images. Here, as described above, when each image is divided into 40 × 40 grids and 256 features are calculated for each grid, the first left side of the equation (1) is calculated in the horizontal direction of each image. A 40-by-40 matrix as shown in the terms and the second term is obtained.

Next, the estimation unit 152 executes the process m shown in FIG. First, the estimation unit 152 calculates, for example, the matrix product of the feature amount of the same column extracted from the right image P1R transposed to the feature amount of the specific column extracted from the left image P1L by the equation (1). In the formula (1), in the first term on the left side, the feature amounts l11 to l1n in the first grid in the horizontal direction of the specific column of the left image P1L are arranged in the row direction, respectively. On the other hand, in the second term on the left side of the equation (1), the feature amounts r11 to r1n of the first grid in the horizontal direction of the right image P1R specific column are arranged in the column direction, respectively. That is, the matrix of the second term on the left side is a transposed matrix in which the feature amounts r11 to r1m of the grid are arranged in the row direction in the horizontal direction of the specific column of the right image P1R. Further, the right side of the equation (1) is a calculation of the matrix product of the first term on the left side and the matrix of the second term on the left side. The first column on the right side of the equation (1) represents the correlation between the feature amount of the first grid extracted from the right image P1R and the feature amount of each horizontal grid of the specific column extracted from the left image P1L, and one row. The eye represents the correlation between the feature amount of the first grid extracted from the left image P1L and the feature amount of each horizontal grid of a specific column extracted from the right image P1R. That is, the right side of the equation (1) shows a correlation map between the feature amount of each grid of the left image P1L and the feature amount of each grid of the right image P1R. In the equation (1), the subscript "m" indicates the position of the grid in the horizontal direction of each image, and the subscript "n" indicates the number of the feature amount in each grid. That is, m is 1 to 40 and n is 1 to 256.

Next, the estimation unit 152 calculates the matching map ML of the right image P1R with respect to the left image P1L as shown in the matrix (1) by using the calculated correlation map. The matching map ML of the right image P1R with respect to the left image P1L is calculated, for example, by applying the Softmax function to the row direction of the correlation map. This normalizes the value of the correlation in the horizontal direction. That is, all the values in the row direction are converted so that the sum is 1.

Next, the estimation unit 152 convolves the calculated matching map ML with the feature amount extracted from the right image P1R by, for example, the equation (2). The first term on the left side of the equation (2) is a transposed version of the matrix (1), and the second term on the left side is the matrix of the first term on the left side of the equation (1). In the present invention, the feature amount for correlating and the feature amount for convolution in the matching map are the same, but a new correlation is obtained from the extracted feature amount by a convolutional neural network or the like. The feature amount for convolution and the feature amount for convolution may be generated separately.

Next, the estimation unit 152 connects the feature amount obtained by the equation (2) to the feature amount extracted from the left image P1L, and generates a new feature amount by, for example, a convolutional neural network. By integrating the features of the left and right images in this way, the estimation accuracy of the position and orientation is improved. The process m in FIG. 15 may be repeated a plurality of times.

Next, the estimation unit 152 estimates the position, orientation, and classification from the features obtained here, for example, by a convolutional neural network. At the same time, the estimation unit 152 calculates the matching map MR of the left image P1L with respect to the right image P1R as shown in the matrix (2) using the calculated correlation map (step S202). The matching map MR of the left image P1L with respect to the right image P1R is also calculated by applying the Softmax function to the row direction of the correlation map, for example, in the same manner as the matching map ML of the right image P1R with respect to the left image P1L.

Next, the estimation unit 152 convolves the feature amount of the left image P1L into the calculated matching map by, for example, the equation (3). The first term on the left side of the equation (3) is the matrix (2), and the second term on the left side is the one before the transposition of the second term on the left side of the equation (1).

Next, the estimation unit 152 selects and compares the preset threshold value and the grid with the largest estimation result of the target (object) classification estimated from the left image P1L (step S203). If the threshold is not exceeded, it ends as if there is no target. If the threshold value is exceeded, the grid with the largest value is selected from the matching map ML with the right image P1R for that grid (step S204).

Next, in the selected grid, the estimation result of the target classification of the right image P1R is compared with the preset threshold value (step S208). If the threshold value is exceeded, the grid with the largest value is selected from the matching map ML with the left image P1L for that grid (step S209). If the threshold value is not exceeded, the classification score of the grid selected from the estimation result of the left image P1L is set to 0, and the process returns to step S203 (step S207).

Next, it is compared whether the grid of the matching map ML selected in step S209 and the grid selected from the estimation result of the left image P1L in step S204 are equal (step S210). If the grids are different, the classification score of the grid selected from the estimation result of the left image P1L in step S204 is set to 0, and the process returns to the grid selection in step S203 (step S207). Finally, the parallax is calculated from the detection result of the position information (for example, the value of x in the horizontal direction in FIG. 1) of the grid selected in the left image P1L and the right image P1R (step S211).

Next, the depth of the target is calculated based on the parallax calculated from step S211 (step S212). When calculating the depth for a plurality of targets, after step S211, the classification score of the grid selected from the estimation results of the left image P1L and the right image P1R is set to 0, and then the process returns to step S203. The process up to step S212 may be repeated.

As described above, the image processing apparatus 10 in the first embodiment includes an acquisition unit and an estimation unit. The acquisition unit acquires a first image and a second image obtained by photographing the workpieces stacked separately. The estimation unit generates a matching map of the feature amount of the first image and the feature amount of the second image, and the position and orientation of each target work for each of the first image and the second image. And the classification score is estimated, and the depth from the stereo camera to the work is calculated by estimating the work position based on the matching result and the position estimation result using the attention map. As a result, erroneous detection in object recognition can be suppressed.

(Modification example)
Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the spirit of the present invention. For example, in the first embodiment, the case where the object (work) is one type has been described, but the present invention is not limited to this, and the image processing device 10 is configured to detect a plurality of types of the work. May be good. Further, the image processing device 10 may not only detect the object, but may further detect the position and orientation of the tray or the like on which the object is placed. FIG. 16 is a diagram showing an example of a loosely stacked image including a tray according to a modified example. In the example shown in FIG. 16, the image processing device 10 can set a trajectory so that the robot arm 30 does not collide with the tray by specifying the position and posture of the tray on which the object is placed. The tray to be detected is an example of an obstacle. The image processing device 10 may be configured to detect other obstacles other than the tray.

Further, the example in which the image processing device 10 divides the loosely stacked images into a grid of 40 × 40 has been described, but the present invention is not limited to this, and the object is detected by dividing the images into a finer or coarser grid. The estimation process may be performed on a pixel-by-pixel basis. As a result, the image processing device 10 can calculate the distance between the camera and the object more accurately. FIG. 17 is a diagram showing an example of a position deviation estimation model according to a modified example. As shown in FIG. 17, the image processing apparatus 10 may cut out a portion of the left image P1L and the right image P1R that is smaller in size than the grid around the estimated position and combine them. Then, the estimation process may be performed in the same manner as the estimation process in the first embodiment, and the positional deviation may be estimated based on the processing result.

Further, when the estimation process is performed in fine or coarse grid units or pixel units, the estimation process may be performed individually for the left image P1L and the right image P1R as in the first embodiment. .. FIG. 18 is a diagram showing another example of the position deviation estimation model according to the modified example. In the example shown in FIG. 18, the image processing device 10 separately performs estimation processing on the left image P1L and the right image P1R. In this case as well, the image processing apparatus 10 may share the weighting for the left image P1L with the weighting for the right image P1R when performing each estimation processing, as in the first embodiment.

Further, the estimation process described above is not applied to the images of the workpieces 41 and 42 stacked separately, but to the robot arm 30, the workpieces 41 and 42 held by the robot arm 30, or the workpieces aligned at the alignment destination. You may go to 41, 42.

Moreover, the present invention is not limited by the above-described embodiment. The present invention also includes a configuration in which the above-mentioned components are appropriately combined. Further, further effects and modifications can be easily derived by those skilled in the art. Therefore, the broader aspect of the present invention is not limited to the above-described embodiment, and various modifications can be made.

1 Object grasping system 10 Image processing device 20 Camera 30 Robot arm 41, 42 Work

Claims (3)

A learning data generation method that generates images of workpieces stacked separately by image generation software and uses them as learning data for machine learning. Obtain a 3D model of the workpiece to be gripped and
Determine the position and orientation of the 3D model and
A plurality of the three-dimensional models are arranged in the virtual space based on the determined position and orientation, and the three-dimensional models are arranged in the virtual space.
The learning data generation method according to claim 1, wherein an image of the loosely stacked workpieces is generated by acquiring an image of the virtual space in which the three-dimensional model is arranged. The learning data generation method according to claim 1 or 2, further performing machine learning processing using teacher data including the coordinates and orientations of the loosely stacked workpieces and images of the loosely stacked workpieces.

Images