JP6895563B2 - Robot system, model generation method, and model generation program - Google Patents

Description

The present invention relates to an apparatus, a robot system, a model generation method, and a model generation program that perform processing for taking out an object.

Conventionally, in order to detect the take-out position of an object (hereinafter, also referred to as “work”), teaching using a distance image of the work measured by a three-dimensional measuring unit including a distance image sensor has been performed. As a method for teaching using a distance image, for example, a method by CAD (Computer-Aided Design) matching and a method of searching based on setting parameters are generally used. Here, the distance image means an image obtained by measuring the surface of a measurement object (work), and each pixel on the captured image has depth information from the three-dimensional measurement unit. .. That is, it can be said that each pixel on the distance image has the three-dimensional coordinate information in the three-dimensional coordinate system provided by the three-dimensional measuring unit.

In the CAD matching method, first, CAD data corresponding to the shape of the work is created. Next, the position of taking out the work that can be grasped by the robot's hand is taught to the CAD data. In addition, a location matching the CAD data is searched from the distance image obtained by measuring the work. Then, the position corresponding to the take-out position of the work taught to the CAD data is selected at the matched portion. Based on this selection result, the work can be taken out by controlling the gripping by the hand of the robot.
Patent Document 1, for example, discloses a technique for taking out bulk-loaded works by teaching a work taking-out position using such CAD data.

In addition, when performing a search based on the setting parameters, based on the experience of the instructor, select a search algorithm for searching the extraction position of the work that can be grasped by the robot's hand, and select the parameters of the selected search algorithm. By adjusting the parameters while checking whether the work can be found with respect to the actual distance image while changing the work, it is possible to take out the work.

JP-A-2017-124450

As described above, the method by CAD data matching is widely used. However, in this method, it is necessary to teach the CAD model a plurality of take-out positions of the work that can be grasped by the hand, and it takes time to teach.
In addition, a CAD model is required for matching, but a CAD model cannot be created for an amorphous work in the first place. Further, when the object to be taken out is not a finished product but a work being machined, a CAD model of the work being machined must be additionally created for taking out.

On the other hand, when searching from a distance image based on parameters without using CAD data, it is necessary to select a search algorithm based on the experience of the instructor and considering the relationship between the work and the hand as described above. There is.
As described above, in the conventional method, it is not easy to select the take-out position of the work in the robot system.

Therefore, an object of the present invention is to provide a robot system and a work taking-out method for selecting a work taking-out position by a simpler method.

(1) The apparatus of the present invention (for example, the image processing apparatus 10a described later) is a position in an image including the plurality of objects for taking out at least one of a plurality of objects (for example, the work 50 described later) by hand. A reception unit that accepts teaching (for example, an operation reception unit 13 described later) and
A device including a learning unit (for example, a learning unit 14 described later) that learns a learning model for extracting at least one of the plurality of objects based on the teaching.

(2) The device according to (1) above may further include a display unit (for example, a display unit 12 described later) for displaying the image.
(3) In the device according to (2) above, the display unit may draw the displayed position.
(4) In the apparatus according to any one of (1) to (3) above, the teaching of the position may be such that the position is designated by either a point or a region.

(5) In the apparatus according to any one of (1) to (4) above, information on the taught position and a point cloud in the vicinity thereof is stored as search information, and the image is used for the search. A take-out position selection unit (for example, a selection processing unit 11 described later) for selecting a new take-out position by the hand may be provided by performing a search based on information.
(6) In the apparatus according to any one of (1) to (5) above, the learning unit determines at least one of the evaluation value based on the teaching or the success or failure of taking out the object based on the teaching. Based on this, the learning model may be trained.

(7) In the apparatus according to any one of (1) to (5) above, the learning unit uses the information of the point group at the taught position and its vicinity as input data, and uses the point group as the input data. By performing machine learning using at least one of the evaluation value according to the teaching of the information and the evaluation value according to the success or failure of extraction as a label, the evaluation value for the information of the point group input as input data is output. The learning model may be trained.

(8) The apparatus according to (7) above cuts out an image of a predetermined region from the image, and inputs the information of the point cloud of the cut out image into the learning model as input data to output the points. A take-out position selection unit (for example, a selection processing unit 11 described later) that selects a new take-out position by the hand based on the evaluation value of the group information may be further provided.
(9) In the apparatus according to any one of (1) to (8) above, the learning model may be a neural network.

(10) The robot system of the present invention (for example, the robot system 1a described later) is the device according to any one of (1) to (8) above, and a measuring device (for example, a measuring device for generating images of the plurality of objects). A robot system including a three-dimensional measuring machine 40) described later and a robot having the hand (for example, the robot 30 described later).

(11) The robot system of the present invention (for example, the robot system 1a described later) is a measuring machine (for example, a three-dimensional measuring machine 40 described later) that generates images of a plurality of objects, and at least one of the plurality of objects. A robot having a hand for taking out (for example, a robot 30 described later) and a reception unit (for example, an operation receiving unit 13 described later) for receiving an instruction of a taking-out position for taking out by the hand based on the image. A robot system in which the robot takes out at least one of the plurality of objects by the hand based on the information of the point group corresponding to the taught take-out position.

(12) The robot system according to (11) above may further include a display unit for displaying the image.
(13) In the robot system according to the above (12), the display unit may draw the taught position on the displayed image.
(14) In the robot system according to any one of (11) to (13) above, the teaching of the position may be such that the position is designated by either a point or a region.
(15) In the robot system according to any one of (11) to (14) above, information on the taught position and a point cloud in the vicinity thereof is stored as search information, and the search is performed on the image. A take-out position selection unit (for example, a selection processing unit 11 described later) for selecting a new take-out position by the hand may be further provided by performing a search based on the information.

(16) The work taking-out method of the present invention is a robot having a measuring machine (for example, a three-dimensional measuring machine 40 described later) that generates images of a plurality of objects and a hand for taking out at least one of the plurality of objects. (For example, the robot 30 described later) and an object taking-out method performed by a robot system (for example, the robot system 1a described later), wherein the taking-out position for taking-out by the hand is based on the image. The robot includes a reception step for receiving teaching and a step for learning a learning model based on the teaching of the take-out position, and the robot uses the hand to perform at least one of the plurality of objects based on the learning model. How to take out an object.

(17) In the object taking-out method according to the above (16), the learning step is based on at least one of an evaluation value based on the teaching or the success or failure of taking out the object based on the teaching. The learning model may be trained.
(18) In the object extraction method according to (16) above, the learning step uses the information of the taught extraction position and the point group in the vicinity thereof as input data, and with respect to the information of the point group used as the input data. The learning that outputs the evaluation value of the information of the point group input as input data by performing machine learning using at least one of the evaluation value according to the teaching and the evaluation value according to the success or failure of extraction as a label. You may try to train the model.
(19) In the object extraction method according to (18) above, an image of a predetermined region is cut out from the image, and the information of the point cloud of the cut out image is input to the learning model as input data to be output. The robot further includes a take-out position selection step of selecting a new take-out position based on the evaluation value of the point cloud information, and the robot uses the hand to perform the plurality of objects based on the taught take-out position. At least one may be taken out and each object may be taken out by the hand at the new take-out position selected by the take-out position selection step.

(20) Another robot system of the present invention (for example, the robot system 1b described later) is a measuring machine (for example, a three-dimensional measuring machine 40 described later) that generates an image of a plurality of objects (for example, a work 50 described later). An evaluation value map showing a robot having a hand for taking out at least one of the plurality of objects and an area in which an object that can be taken out by the hand exists, which is acquired by inputting the image into the learning model. The robot includes a position selection unit (for example, a take-out position selection unit 153 described later) that selects at least one take-out position of the plurality of objects based on the above. Take out at least one of the plurality of objects by hand.
(21) The robot system according to (20) above (for example, the robot system 1b described later) has a display unit (for example, a display unit 12 described later) for displaying the image and the image displayed on the display unit. A reception unit (for example, an operation reception unit 13 described later) that receives teaching of at least one teaching position based on the above, and a label map showing at least one teaching position based on the teaching position received by the reception unit are generated. An annotation processing unit (for example, an annotation processing unit 152 described later) that associates a label map with the image and stores it as a data set in a teacher data storage unit (for example, a teacher data storage unit 151 described later), and the teacher data storage unit. Further includes a learning processing unit (for example, a learning processing unit 141 described later) that performs machine learning by using the data set stored in the data set as an input and outputs the learning model.

(22) Another work taking-out method of the present invention includes a measuring machine (for example, a three-dimensional measuring machine 40 described later) that generates images of a plurality of objects (for example, the work 50 described later) and at least the plurality of objects. This is an object extraction method performed by a robot having a hand for extracting one (for example, the robot 30 described later) and a robot system (for example, the robot system 1b described later), and inputting the image into a learning model. The robot includes a position selection step of selecting at least one extraction position of the plurality of objects based on an evaluation value map indicating an area in which an object that can be extracted by the hand exists. Takes out at least one of the plurality of objects by the hand based on the selected take-out position.
(23) The object extraction method according to (22) above includes a display step for displaying the image, a reception step for receiving teaching of at least one teaching position based on the image displayed in the display step, and the above. An annotation processing step that generates a label map indicating at least one teaching position based on the teaching position received in the reception step, associates the label map with the image, and stores the data set in the teacher data storage unit, and the above. The data set stored in the teacher data storage unit may be used as an input to perform machine learning, and a learning processing step for outputting the learning model may be further provided.

According to the present invention, it is possible to select the take-out position of the work by a simpler method.

It is a schematic diagram which shows the whole structure of each embodiment of this invention. It is a block diagram which shows the functional block of the image processing apparatus in 1st Embodiment of this invention. It is a schematic diagram which shows the work piled in bulk to be taken out in each embodiment of this invention. It is a figure which shows the distance image generated by measuring the work piled in bulk in each embodiment of this invention. It is a figure which shows the distance image which drew the extraction position taught in each embodiment of this invention. It is a figure which shows the 3D point cloud group information around a drawing position recorded in 1st Embodiment of this invention. It is a figure which shows the cutout made in 1st Embodiment of this invention. It is a figure which shows the scanning at the time of cutting performed in 1st Embodiment of this invention. It is a flowchart (1/2) which shows the operation in 1st Embodiment of this invention. It is a flowchart (2/2) which shows the operation in 1st Embodiment of this invention. It is a block diagram which shows the functional block of the image processing apparatus in the 2nd Embodiment of this invention. It is a flowchart which shows the operation in 2nd Embodiment of this invention.

Next, an embodiment of the present invention will be described in detail with reference to the drawings.
Hereinafter, two embodiments, a first embodiment and a second embodiment, will be described. Here, each embodiment is common in a configuration in which a work take-out position is selected.
However, in the process of selecting the extraction position, in the first embodiment, preprocessing such as image matching processing and image cropping processing is performed, whereas in the second embodiment, such preprocessing is omitted. It's different.
In the following, first, the first embodiment will be described in detail, and then the parts of the second embodiment that are different from the first embodiment will be described.

<Structure of the entire embodiment>
The configuration of the robot system 1a according to the first embodiment will be described with reference to FIG. The robot system 1a includes an image processing device 10a, a robot control device 20, a robot 30, a three-dimensional measuring machine 40, a plurality of workpieces 50, and a container 60.
The image processing device 10a is communicably connected to the robot control device 20 and the three-dimensional measuring device 40. Further, the robot control device 20 is communicably connected to the robot 30 in addition to the image processing device 10a.

First, the outline of the robot system 1a will be described.
In the robot system 1a, a plurality of randomly placed workpieces 50 including a state of being piled up in bulk are measured by a three-dimensional measuring machine 40 to generate a distance image.

Then, using the image processing device 10a, the teaching for selecting the extraction position of the work 50 from the distance image is given. This teaching is performed by the user directly teaching the take-out position of the work 50 to the distance image. Specifically, the image processing device 10a displays the distance image on the image display unit by, for example, gray scale or RGB gradation so that the difference in height of each pixel constituting the distance image can be grasped.

When the user who refers to this selects an appropriate place as a take-out position candidate by operating a mouse or the like and designates it as the take-out position of the work 50, the image processing device 10a teaches the displayed distance image by the user. The extracted position (hereinafter referred to as "teaching position") is drawn. Further, the image processing device 10a acquires and stores the data of the distance image around the extraction position taught by the user (hereinafter referred to as "teaching position neighborhood image data"). An appropriate location as a take-out position candidate is, for example, a location with a high height of a three-dimensional point.

After that, when the extraction position of the work 50 is selected, the image processing device 10a searches for an image region matching with the stored image data near the teaching position using a distance image. Then, the image processing device 10a selects the matched image region as the new teaching position neighborhood image data, and selects, for example, the center position of the image data as the new extraction position. In this way, the image processing device 10a can select a new extraction position based on the image data in the vicinity of the teaching position stored based on the instruction of the user, so that the work 50 does not need to receive a new instruction from the user. It is possible to select the take-out position.
With such a configuration, in the robot system 1a, the take-out position of the work 50 can be selected by a simpler method as compared with the conventional case.
Further, the robot control device 20 generates a control signal for taking out the work 50 selected by the image processing device 10a. Then, the robot 30 executes the extraction of the work 50 based on the control signal generated by the robot control device 20. With such a configuration, the robot system 1a can actually take out the work 50 based on the selected take-out position.

Further, the image processing device 10a performs machine learning based on the teaching result, and constructs a learning model for selecting the extraction position of the work 50. Then, the image processing device 10a selects a new extraction position from the distance image based on the constructed learning model.
With such a configuration, the robot system 1a can select the extraction position with higher accuracy.
The above is the outline of the robot system 1a. Next, each device included in the robot system 1a will be described.

The image processing device 10a is a device for performing teaching and machine learning using a distance image. The details of the image processing device 10a will be described later with reference to the functional block diagram of FIG.

The robot control device 20 is a device for controlling the operation of the robot 30. The robot control device 20 generates a control signal for controlling the operation of the robot 30 based on information such as a take-out position of the work 50 selected by the image processing device 10a. Then, the robot control device 20 outputs the generated control signal to the robot 30.

The image processing device 10a and the robot control device 20 associate a machine coordinate system for controlling the robot 30 with a camera coordinate system indicating a take-out position of the work 50 by calibration performed in advance. And.

The robot 30 is a robot that operates under the control of the robot control device 20. The robot 30 includes a base portion for rotating about an axis in the vertical direction, an arm for moving and rotating, and a hand attached to the arm for gripping the work 50.
The robot 30 drives the arm and the hand in response to the control signal output by the robot control device 20, moves the hand to the teaching position, grasps the work 50 piled up in bulk, and takes it out of the container 60.
The transfer destination of the removed work 50 is not shown. Further, since the specific configuration of the robot 30 and the hand is well known to those skilled in the art, detailed description thereof will be omitted.

The three-dimensional measuring machine 40 generates a distance image by measuring the work 50 in the container 60. The three-dimensional measuring machine 40 can be realized by, for example, a camera provided with a distance image sensor as described above, or a three-dimensional measuring machine using a stereo camera. However, if it is possible to generate a distance image of a three-dimensional point group in the process of measurement, it is not limited to a camera equipped with a distance image sensor or a three-dimensional measuring device using a stereo camera. The three-dimensional measuring machine 40 may be realized by the dimensional measuring machine.
The distance image generated by the three-dimensional measuring device 40 is output to the image processing device 10a.

The work 50 is randomly placed in the container 60 including the state of being piled up in bulk. The work 50 may be any as long as it can be gripped by a hand attached to the arm of the robot 30, and its shape and the like are not particularly limited.

<Functional block of image processing device 10a>
Next, each functional block included in the image processing apparatus 10a will be described with reference to FIG. In FIG. 2, the components other than the image processing device 10a in the robot system 1a are collectively illustrated as the environment 100.

The image processing device 10a includes a selection processing unit 11, a display unit 12, an operation reception unit 13, and a learning unit 14.

The selection processing unit 11 selects the take-out position of the work 50 by performing various image processing as described above as an outline of the robot system 1a. The image processing performed by the selection processing unit 11 is roughly divided into two, "matching processing" and "cutting processing".
In order to perform these processes, the selection processing unit 11 includes a selection data storage unit 111, an annotation processing unit 112, a matching unit 113, and a cutting unit 114.
The selection data storage unit 111 is a part that stores various data used by each functional block in the selection processing unit 11.
Annotation processing unit 112, a part that performs processing for receiving instruction of the extraction position of the work 50 from the user.
The matching unit 113 is a portion that performs matching processing. Further, the cutout portion 114 is a portion for performing the cutout process.
The function of each functional block included in the selection processing unit 11 and the contents of the matching process and the cutout process will be described later.

The display unit 12 is a portion that displays an image output by the selection processing unit 11. The image output by the selection processing unit 11 is, for example, a distance image generated by measuring the work 50 with the three-dimensional measuring machine 40, or an image in which the teaching position taught by the user is drawn on the distance image. The display unit 12 is realized by, for example, a liquid crystal display or an organic EL display.

The operation receiving unit 13 is a part that receives an operation from the user. The operation reception unit 13 accepts, for example, an operation of designating the take-out position for teaching the take-out position of the work 50 from a user who has referred to the distance image displayed on the display unit 12. The operation reception unit 13 is realized by, for example, a mouse, a keyboard, or the like.

For example, the user may use a mouse or a keyboard to move the cursor displayed on the display unit 12 to the extraction position (on the image showing the image) to specify the cursor. Further, by integrally realizing the display unit 12 and the operation reception unit 13 with the touch panel, for example, the extraction position (on the image showing the image) may be tapped to specify the display unit 12.

The learning unit 14 is a part that performs processing related to machine learning. The learning unit 14 includes a learning processing unit 141, a learned model storage unit 142, and an estimation processing unit 143.
The learning processing unit 141 is a part that executes machine learning, and for example, performs deep learning using a convolutional neural network (convolutional neural network).
The trained model storage unit 142 is a part that stores the parameters of the learning model that the learning processing unit 141 is learning in machine learning and the parameters of the trained model.
The estimation processing unit 143 performs estimation using the trained model stored in the trained model storage unit 142 in order to select the extraction position by the selection processing unit 11.
The detailed contents of machine learning performed by each of these parts will be described later in the item <Machine learning>.

The functional blocks included in the image processing apparatus 10a have been described above.
In order to realize these functional blocks, the image processing device 10a includes an arithmetic processing unit such as a CPU (Central Processing Unit). Further, the image processing device 10a is temporarily used as an auxiliary storage device such as an HDD (Hard Disk Drive) that stores various control programs such as application software and an OS (Operating System), or when an arithmetic processing unit executes a program. It also has a main storage device such as a RAM (Random Access Memory) for storing the required data.

Then, in the image processing device 10a, the arithmetic processing unit reads the application software and the OS from the auxiliary storage device, and while deploying the read application software and the OS to the main storage device, the arithmetic processing based on these application software and the OS is performed. Do it. In addition, various hardware included in each device is controlled based on the calculation result. Thereby, the functional block of the first embodiment is realized. That is, the first embodiment can be realized by the cooperation of hardware and software.

As a specific example, the image processing device 10a can be realized by incorporating application software for realizing the first embodiment into a general personal computer or server device.
However, since the image processing device 10a has a large amount of calculation due to machine learning performed by the learning unit 14, for example, a GPU (Graphics Processing Units) is mounted on a computer and is called GPGPU (General-Purpose computing on Graphics Processing Units). Depending on the technology, if the GPU is used for arithmetic processing associated with machine learning, high-speed processing may be possible. Further, if the computer is equipped with an FPGA (Field-Programmable Gate Array) and the FPGA is used for arithmetic processing associated with machine learning, high-speed processing may be possible.

Furthermore, in order to perform higher-speed processing, a computer cluster is constructed using a plurality of computers equipped with such GPUs and FPGAs, and parallel processing is performed by a plurality of computers included in the computer cluster. You may do so.

<Matching process>
Next, the contents of the processing performed by the selection processing unit 11 will be described in detail. As described above, the selection processing unit 11 performs two processes, "matching process" and "cutout process". First, the matching process will be described with reference to FIGS. 3 to 6.

FIG. 3 shows a bird's-eye view of the work 50 stacked in the container 60. As shown in FIG. 3, the works 50 are randomly piled up in bulk, and it is difficult to grasp and take out the work 50 by the hand of the robot 30 without being taught the take-out position of the work 50. ing. In the figure, for convenience of illustration, only some of the works 50 are designated by reference numerals.

The three-dimensional measuring machine 40 generates a distance image by measuring the bulk workpieces 50 stacked in the container 60. The three-dimensional measuring device 40 transmits the generated distance image to the selection processing unit 11 of the image processing device 10a. The distance image received by the selection processing unit 11 is stored in the selection data storage unit 111.
The annotation processing unit 112 acquires a distance image from the selection data storage unit 111 in order to receive the instruction of the extraction position of the work 50 from the user. Then, the annotation processing unit 152 outputs the acquired distance image to the display unit 12. The display unit 12 displays the input distance image as shown in FIG. In the figure, for convenience of illustration, only some of the works 50 are designated by reference numerals.

When three-dimensional measurement is performed, it is possible to generate a distance image in which each pixel position has height information on a two-dimensional pixel. The annotation processing unit 112 displays the distance image by expressing the height information of each pixel in RGB color tones or by expressing it in grayscale density. The user who teaches can recognize the depth information by referring to the distance image of such an expression method.

In FIG. 4, the height information of each pixel is represented by hatching due to the limitation of describing the patent drawing, but the present invention is not limited to this. For example, the height information of each pixel may be expressed by color gradation, gradation, or the like. In FIG. 4, those that are close to the 3D measuring instrument 40 (that is, those that are high from the ground) are white, and those that are far from the 3D measuring instrument 40 (that is, those that are far from the ground) are made white. The one with a low height) is expressed as black.

For example, since the work 50a exists at a position higher than the work 50b from the ground, the work 50a is hatched so as to be whiter than the work 50b. Further, the surface corresponding to the work 50b is partially hidden by the surface corresponding to the work 50a. By referring to this, the user can know that the work 50a is stacked on the work 50b.
Further, for example, the height of the surface corresponding to the work 50c differs depending on the location. By referring to this, the user can know that the work 50c is in a state of being stacked in an inclined manner.

The user who has referred to the distance image displayed on the display unit 12 in this way uses the operation reception unit 13 realized by the mouse or the touch panel as described above to specify the take-out position of the work 50 to work. The take-out position of 50 is taught. The operation receiving unit 13 notifies the annotation processing unit 112 of the teaching position, which is the take-out position of the taught work 50. The annotation processing unit 112 draws on the teaching position on the distance image so that the user can recognize the notified teaching position. The drawing is performed by a method that is easy for the user to grasp, such as changing the color of the pixel at the teaching position on the distance image. FIG. 5 shows a display example when drawing is performed.

The user teaches the take-out position for each of the works 50 in the take-out position. Then, as shown as the teaching position 71 in the drawing, drawing is performed so that the teaching position taught by the user can be grasped. In the figure, reference numerals are given only to some teaching positions 71 for convenience of illustration.

The annotation processing unit 112 acquires three-dimensional point information of the teaching position and image data near the teaching position, which is three-dimensional point group information within a predetermined range in the vicinity of the teaching position centered on the teaching position. Then, the annotation processing unit 112 associates the acquired three-dimensional point information of the teaching position with the image data in the vicinity of the teaching position and stores it in the selection data storage unit 111 as matching point group information.

An example of the matching point cloud information is shown in FIG. 6 as the matching point cloud information 80. In the example of FIG. 6, the teaching points are drawn for explanation, but the drawing indicating the teaching position is not included in the image data near the teaching position included in the matching point cloud group information 80. Further, in the following description, the description of "80", which is the code of the matching point cloud information, is omitted.

The size of the range of the three-dimensional point cloud acquired as the matching point cloud information is set in advance based on the size of the work 50 and the like. It should be noted that the three-dimensional point cloud information in a large range to some extent is stored as the matching point cloud information, and when the stored matching point cloud information is used in the matching described later, it is necessary to adjust the settings. The one trimmed to the size may be used as the matching point cloud information.

Note that the annotation processing unit 112 may add incidental information to the matching point cloud information. For example, information indicating the characteristics of the matching point cloud information may be added. The characteristic information is, for example, information such as the average height of three-dimensional points of a plurality of pixels included in the matching point cloud information, the height of three-dimensional points of pixels corresponding to the teaching position, and the like. is there.

In the first embodiment, since the plurality of works 50 are piled up in bulk, a plurality of teaching positions are taught by the user as described above by using one distance image measured by the three-dimensional measuring machine 40. By doing so, it becomes possible to teach the take-out position for each of the plurality of postures that the work 50 can take.

The teaching in the CAD model described as the background technique assumes that the three-dimensional points can be uniformly acquired with respect to the shape. However, when an image is actually taken, three-dimensional points cannot be uniformly acquired with respect to the shape due to optical conditions and the like. As described above, since there is a difference in the ease of obtaining three-dimensional points between the hypothetical image and the actual image pickup, this deviation may affect the degree of matching.

On the other hand, in the first embodiment, the distance image by the three-dimensional points acquired under the actual optical conditions can be acquired as an input, and the three-dimensional point cloud information around the drawing position can be saved. As described above, it is possible to prevent a problem that the teaching position on the CAD model and the teaching position of the work 50 corresponding to the teaching position on the CAD model cannot be obtained due to optical conditions or the like.

Next, matching using the matching point cloud information performed by the matching unit 113 when the work 50 is taken out will be described.
When the work 50 is taken out, the matching unit 113 acquires a distance image of the work 50s piled up in bulk in the container 60 from the three-dimensional measuring machine 40. Then, the matching unit 113 refers to the acquired distance image, for example, based on the teaching position neighborhood image data included in the respective matching point cloud information stored in the selection data storage unit 111, for example, ICP (Iterative Closest). Point) Search is performed using a matching method for a three-dimensional point cloud such as matching. Then, the matching unit 113 selects, for example, the center position of the image region having a high evaluation of matching in the distance image as the extraction position of the work 50 to be extracted. A plurality of image regions having a matching evaluation higher than the threshold value may be selected, and the image region having the highest height from the ground among the plurality of image regions may be used as new teaching position neighborhood image data.

The matching unit 113 transmits the take-out position of the selected work 50 to the robot control device 20. Then, the robot control device 20 attempts to take out the work 50 by controlling the robot 30 based on the received take-out position of the work 50.

Here, the matching point cloud information is created based on the instruction of the user as described above, but not all the matching point cloud information is appropriate. For example, when a part with a high evaluation of matching with a certain matching point cloud information is set as the extraction position, the extraction is successful, but when a part with a high evaluation of matching with other matching point cloud information is set as the extraction position. May fail to retrieve.
In this way, success or failure may differ depending on the matching point cloud information. Therefore, the matching unit 113 assigns an evaluation value to each matching point cloud information by evaluating each matching point cloud information. It is good to do it. Then, it is desirable that the matching unit 113 uses the matching point cloud information having a high evaluation value.
Further, since the matching point cloud information to which the evaluation value is given is used as teacher data in machine learning described later, the matching unit 113 stores the matching point cloud information to which the evaluation value is given in the selection data storage unit 111. Store. The matching point cloud information with a low evaluation value is also necessary data as teacher data (failure data) for machine learning. Therefore, the matching unit 113 stores not only the matching point cloud information having a high evaluation value but also the matching point cloud information having a low evaluation value in the selection data storage unit 111 as teacher data.

The matching unit 113 can give an evaluation value according to the success or failure of taking out the work 50. For example, when the matching point cloud information and the portion where the evaluation of matching is high is set as the extraction position, the matching unit 113 has a higher evaluation value when the work 50 is successfully extracted than when the extraction fails. Is given. For example, the matching unit 113 assigns a first predetermined value or more (for example, 60 points or more) when the work 50 is successfully taken out, and gives a second predetermined value or less (for example, 50 points) when the work 50 is taken out unsuccessfully. Below). Further, for example, when the matching unit 113 succeeds in taking out the work 50, the evaluation value may be further changed according to the time required for taking out the work 50. For example, the matching unit 113 may set the evaluation value to be higher as the time required for taking out the work 50 is shorter. Further, for example, when the matching unit 113 fails to take out the work 50, the evaluation value may be different depending on the degree of failure. For example, if the matching unit 113 can grip the work 50 but falls during the removal, the evaluation value may be higher than that in the case where the work 50 cannot be gripped.

The matching unit 113 performs matching based on each matching point cloud information with respect to the distance image measured this time. In addition, the matching unit 113 attempts to take out the work 50 at each matching location and assigns the evaluation value described above. This process is repeated by newly acquiring a distance image of the work 50s piled up in bulk in the container 60 from the three-dimensional measuring machine 40.

In this way, the matching unit 113 evaluates each matching point cloud information by repeating matching using the matching point cloud information, taking out the work 50, and assigning an evaluation value according to the taking out result. It can be carried out. Then, the matching unit 113 can improve the probability of succeeding in taking out the work 50 by selecting the matching point cloud information having a high evaluation value.

Further, the matching unit 113 acquires the three-dimensional point cloud information in the range of the same size as the matching point cloud information for the portion that has succeeded in matching in the distance image, and newly obtains the acquired three-dimensional point cloud information. It is preferable to use the matching point cloud information.
In this way, the matching unit 113 can take out the work 50 with respect to the new matching point cloud information acquired from the matching portion in addition to the matching point cloud information used for matching. That is, the matching unit 113 can automatically increase the matching point cloud information. This makes it possible to collect matching point cloud information with a higher evaluation value.

As described above, in the first embodiment, the matching unit 113 repeats the matching using the matching point cloud information and the extraction of the work 50, and the addition of the evaluation value according to the extraction result, thereby extracting the work 50. It becomes possible to select the position. It is also possible to automatically increase the matching point cloud information.
Therefore, in the first embodiment, the work 50 can be taken out based on the selected take-out position without receiving a new instruction from the user.

The first embodiment is based on the teaching for the distance image acquired by actually measuring the work 50 with the three-dimensional measuring machine 40, unlike the teaching for the model such as CAD which has been conventionally performed. When the distance image acquired by actually measuring the work 50 in this way is used, matching is often not established due to reasons such as disturbance being included. Therefore, there is a case where the threshold value regarding the success or failure of the matching is set loosely so that the matching is established.
However, if the means of loosening the threshold value is taken in this way, there arises a problem that matching is established even in a place that is not suitable for being actually detected as the take-out position. Therefore, in order to alleviate this problem, we try to match even the places that were not taught by drawing, and if the matching is established at the places that were not taught, it is regarded as inappropriate as the detection position and finally. The process of excluding from the detection position may be further added to the process of the first embodiment described above.
The matching process by the matching unit 113 has been described above. By the above-mentioned matching process, it is possible to select the take-out position of the work 50.
By further combining this configuration with machine learning by the learning unit 14 and cutting processing by the cutting unit 114, it is possible to improve the selection accuracy of the take-out position of the work 50. Hereinafter, machine learning by the learning unit 14 will be described. Next, the cutting process by the cutting portion 114 will be described.

<Machine learning>
As described above, the matching unit 113 repeats matching using the matching point cloud information, taking out the work 50, and giving an evaluation value according to the taking-out result, so that the matching point to which the evaluation value is given is given. Create point cloud information. Then, as described above, the matching unit 113 stores the matching point cloud information to which the evaluation value is given in the selection data storage unit 111.
The learning processing unit 141 of the learning unit 14 performs supervised machine learning using the matching point cloud information to which the evaluation value is given stored in the selection data storage unit 111 as teacher data. Further, the learning processing unit 141 constructs a learning model for improving the selection accuracy of the extraction position of the work 50 by this supervised learning. Then, the learning processing unit 141 stores the constructed learning model in the trained model storage unit 142.
Then, in the clipping process described later, the estimation processing unit 143 matches the distance image measured by the three-dimensional measuring machine 40 by using the learning model stored in the learned model storage unit 142. It is possible to select the take-out position of the work 50 without having to do so. Further, the learning processing unit 141 can further update the once constructed learning model stored in the learned model storage unit 142 by further performing machine learning based on the success or failure of the extraction in the clipping process. ..

Next, the construction of the learning model will be specifically described. The learning processing unit 141 uses the teaching position vicinity image data included in the matching point cloud information stored in the selection data storage unit 111 as input data, and the evaluation value given to the matching point cloud information as a label. Learn with a teacher. As a method of supervised learning, the learning processing unit 141 performs deep learning using a convolutional neural network (CNN), which is a neural network suitable for learning targeting image data. Do. Therefore, a convolution neural network having three or more layers and including at least one image convolution operation is prepared. However, this does not mean that the machine learning applied in the first embodiment is limited to the convolution neural network. A deep learning model other than the convolution neural network, machine learning using a linear model, or the like may be applied to the first embodiment.

Here, the convolution neural network has a structure including a convolution layer, a pooling layer, a fully connected layer, and an output layer. However, this is just a structural example for explanation, and for example, the pooling layer may be omitted. Further, for example, as described above, when learning is performed using an image as a label, a deconvolution layer may be further provided.

In the convolution layer, a filter of a predetermined parameter is applied to the input three-dimensional point cloud information in order to perform feature extraction such as edge extraction. The predetermined parameters in this filter correspond to the weights of a general neural network, and are learned by repeating the learning.

In the pooling layer, for example, the image output from the convolution layer is divided into small windows, and the characteristics of each window (for example, the maximum value in each window) are output.
By combining these convolution layers and pooling layers, features can be extracted from the three-dimensional point cloud information.

In the fully connected layer, the features taken out through the convolution layer and the pooling layer are combined into one node, and the value converted by the activation function is output. Here, the activation function is a function that sets all output values less than 0 to 0, and is used to send only the portion above a certain threshold value to the output layer as meaningful information.

The output layer outputs an evaluation value for extraction based on the matching point cloud information as input data based on the output from the fully connected layer. Then, the error between the output of the output layer and the label is calculated. Here, as described above, the label is an evaluation value given to the matching point cloud information as input data.

At the start of learning, each parameter included in the convolution neural network is not properly weighted, so this error is likely to be a large value. Therefore, the learning processing unit 141 corrects the weighting value so as to reduce the calculated error. Specifically, in order to reduce the error, the weighting value of each perceptron included in the convolution neural network is changed by repeating a process called forward propagation or back propagation.

In this way, the learning processing unit 141 learns the characteristics of the teacher data and recursively acquires a learning model for outputting an evaluation value from the matching point cloud information used as input data. The learning processing unit 141 stores the constructed learning model as a trained model in the trained model storage unit 142.

The above-mentioned learning may be performed by online learning, or may be supervised learning by batch learning or mini-batch learning.
Online learning is a learning method in which supervised learning is performed immediately each time teacher data is created. In addition, batch learning is a learning method in which a plurality of teacher data corresponding to the repetition are collected while the teacher data is repeatedly created, and supervised learning is performed using all the collected teacher data. Is. Further, mini-batch learning is a learning method in which supervised learning is performed every time teacher data is accumulated to some extent, which is intermediate between online learning and batch learning. When performing batch learning or mini-batch learning, the collected teacher data may be stored in the selection data storage unit 111 until the learning is started.
Further, when the teacher data is newly acquired, the accuracy of the estimation by the trained model may be improved by performing the training with the parameters of the trained model as the initial values. In addition, when the teacher data is newly acquired, a separate learning model may be newly constructed regardless of the trained model.

<Cutout processing>
Next, the cutting process performed by the cutting unit 114 will be described.
By using the learning model constructed by the learning processing unit 141, the cutting unit 114 selects the extraction position of the work 50 without matching with the distance image measured by the three-dimensional measuring machine 40. Hereinafter, this process of selecting the extraction position without performing matching will be referred to as “cutout process” and will be described.

When performing the clipping process, when a new distance image is acquired during the extraction operation, the clipping unit 114 has the same size as the input to the trained model over the entire acquired distance image (that is, the same size as the matching point cloud information). ) Distance image is cropped. Then, the cutout portion 114 acquires the three-dimensional point cloud information of the cut out portion (hereinafter referred to as “take-out position candidate”).

The extraction process (cutout process) of the extraction position candidate by the cutout unit 114 will be described with reference to FIGS. 7 and 8. FIG. 7 shows a region to be cut out as a take-out position candidate 90. As described above, the size of the extraction position candidate 90 is the same as the input to the trained model (that is, the same size as the matching point cloud information).

The cutout unit 114 cuts out the entire image by scanning the target to be the extraction position candidate 90 while shifting the target position by several pixels from the upper left of the image, for example, as shown in FIG. Although scanning is performed linearly in FIG. 8, rotation may be added to the scanning. Further, the scanning start position and the scanning direction may be arbitrarily set. At the time of cropping, the cropping unit 114 also acquires the position of the extraction position candidate 90 on the distance image (for example, the center position of the cropped image data), associates it with the extraction position candidate, and stores it in the selection data storage unit 111. To do.
The estimation processing unit 143 acquires each of the three-dimensional point cloud information of all the extraction position candidates extracted in this way from the selection data storage unit 111. Then, the estimation processing unit 143 inputs each of the acquired three-dimensional point cloud information of all the fetched position candidates into the trained model instead of the above-mentioned matching point cloud information, and evaluates each fetched position candidate. Is obtained as an output. The estimation processing unit 143 notifies the clipping unit 114 of this output.

The cutout unit 114 selects a position stored in association with the cutout image having a high evaluation value output from the trained model as a take-out position. The cutout unit 114 transmits the selected take-out position to the robot control device 20. Then, the robot control device 20 attempts to take out the work 50 by controlling the robot 30 based on the received take-out position.

Here, even when the work 50 is tried to be taken out by the three-dimensional point cloud information of the take-out position candidate, the take-out may be successful or unsuccessful as in the case where the work 50 is taken out by the matching point cloud information. obtain. Therefore, the cutout unit 114 assigns an evaluation value to the extraction position candidate according to the success or failure of extraction of the work 50, as in the case of the matching point cloud information at the time of the matching process. Further, the cutout unit 114 stores the extraction position candidate to which the evaluation value is given in the selection data storage unit 111. The extraction position candidate to which this evaluation value is given can be used as new teacher data for the learning processing unit 141 to update the trained model.

Specifically, the learning processing unit 141 uses the three-dimensional point cloud data of the extraction position candidate as input data, and uses the evaluation value given to the extraction position candidate as the input data as a label, and performs the above-mentioned online learning or mini-batch learning. Is preferable. As a result, the trained model can be updated to a more accurate trained model in real time while executing the fetching operation of the work 50. It is not limited to online learning or mini-batch learning. The learning model may be updated by batch learning.

<Operation of the first embodiment>
Next, the operation of the first embodiment will be described with reference to the flowcharts of FIGS. 9A and 9B. Note that FIG. 9A is a flowchart of an operation corresponding to the above-mentioned matching process, and FIG. 9B is a flowchart of an operation corresponding to the above-mentioned clipping process.

In step S11, the annotation processing unit 112 acquires a distance image generated by measuring the works 50 piled up in bulk from the three-dimensional measuring machine 40.
In step S12, the annotation processing unit 112 causes the display unit 12 to display the distance image.
In step S13, the annotation processing unit 112 draws the teaching position on the distance image based on the teaching of the extraction position of the work 50 from the user received by the operation receiving unit 13.

In step S14, the annotation processing unit 112 sets the size of the matching point cloud information. The setting is performed according to a setting value given in advance or an operation of the user.
In step S15, the annotation processing unit 112 generates matching point cloud information based on the settings made in step S14. Further, the annotation processing unit 112 stores the generated matching point cloud information in the selection data storage unit 111.
In step S16, the matching unit 113 assigns an evaluation value to the matching point cloud information by performing matching and taking out the work 50 using the matching point cloud information stored in the selection data storage unit 111. .. The matching unit 113 stores the matching point cloud information to which the evaluation value is given in the selection data storage unit 111.

In step S17, the matching unit 113 determines whether or not additional matching point cloud information is required. When a predetermined number or more of matching point cloud information whose evaluation value is equal to or more than a predetermined value is stored, it is determined as No in step S17, and the process proceeds to step S18. On the other hand, when the matching point cloud information whose evaluation value is equal to or more than a predetermined value is not stored in a predetermined number or more, it is determined as Yes in step S17, the process returns to step S11, and the process is repeated again.

In step S18, the learning processing unit 141 uses the matching point cloud information stored in the learning unit 14 as input data, and performs learning using the evaluation value given to the matching point cloud information as a label. As a result, the trained model is created and stored in the trained model storage unit 142.

Next, the operation during the cutting process will be described with reference to FIG. 9B.
In step S19, the cutout portion 114 acquires a distance image generated by measuring the bulk-stacked workpieces 50 from the three-dimensional measuring machine 40.

In step S20, the cutout unit 114 cuts out a distance image having the same size as the input to the trained model (that is, the same size as the matching point cloud information) over the entire acquired distance image as a extraction position candidate. Then, the cutout unit 114 acquires the three-dimensional point cloud information of the extraction position candidate. Further, the cutout unit 114 stores the acquired three-dimensional point cloud information of the extraction position candidate in the selection data storage unit 111. The estimation processing unit 143 inputs each of the three-dimensional point cloud information of all the extraction position candidates stored in the selection data storage unit 111 to the trained model, and inputs the evaluation value for each extraction position candidate. Get as output. The estimation processing unit 143 notifies the clipping unit 114 of this output.

In step S21, the cutout unit 114 selects a position stored in association with the cutout image having a high evaluation value output from the trained model as a take-out position. The cutout unit 114 transmits the selected take-out position to the robot control device 20. Then, the robot control device 20 attempts to take out the work 50 by controlling the robot 30 based on the received take-out position. As described above, as a result of trying to take out the work 50, the taking out may be successful or unsuccessful. Therefore, the cutout portion 114 assigns an evaluation value to the take-out position candidate according to the success or failure of take-out of the work 50.

In step S22, the learning processing unit 141 determines whether or not to update the trained model by performing learning using the extraction position candidate to which the evaluation value is given as the teacher data in step S21. In the case of performing mini-batch learning, when a predetermined number of teacher data is recorded or when a predetermined time has elapsed from the previous learning, it is determined as Yes in step S23, and the process proceeds to step S24. On the other hand, when less than a predetermined number of teacher data are recorded or when a predetermined time has not elapsed since the previous learning, it is determined as No in step S23, and the process proceeds to step S24. If it is online learning, it is determined to be Yes in step S23, and the process proceeds to step S24.

In step S24, the learning processing unit 141 performs the above-mentioned learning using the three-dimensional point cloud data of the extraction position candidate as input data and the evaluation value given to the extraction position candidate as the input data as a label. As a result, the trained model stored in the trained model storage unit 142 is updated.

In step S24, the cutout portion 114 determines whether or not to continue taking out. If there is a take-out position candidate for the evaluation value and the take-out position candidate is not taken out, it is considered that there is a work 50 that has not been taken out yet. Therefore, it is determined as Yes in step S24, and the process is performed. The process proceeds to step S20. On the other hand, when all the extraction position candidates having high evaluation values are extracted, it is considered that all the works 50 have been extracted, so that it is determined as No in step S24 and the process ends.

According to the matching process of the first embodiment described above, after receiving the instruction of the extraction position from the user in step S13, the stored matching point cloud information ( Based on the 3D point information of the teaching position and the image data near the teaching position centered on the teaching position), the extraction position is performed by performing a search using matching with the distance image measured by the 3D measuring machine 40. Can be selected. Further, according to the matching process of the first embodiment, the user only teaches the extraction position and does not require knowledge based on experience such as selection of a search algorithm. Further, according to the matching process of the first embodiment, since the CAD data is not used, the trouble of preparing the CAD data can be saved.
That is, according to the matching process of the first embodiment, the take-out position of the work 50 can be selected by a simpler method as compared with the conventional method.

Further, according to the clipping process of the first embodiment, a learning model is further constructed based on the matching point cloud information, and based on the constructed learning model, the image data automatically clipped from the distance image is used. The take-out position can be selected, and the take-out position can be selected more efficiently and with higher accuracy.
Further, according to the clipping process of the first embodiment, new teacher data can be acquired as the extraction is continued, so that the constructed learning model can be updated in real time.

<Second embodiment>
Next, the second embodiment will be described in detail. The basic configuration of the second embodiment is common to that of the first embodiment. For example, the overall configuration of the robot system 1b according to the second embodiment is the same as the overall configuration of the robot system 1a according to the first embodiment shown in FIG. By replacing the image processing device 10b, which is the image processing device of the embodiment, with the image processing device 10b, the configuration of the second embodiment is obtained.
In the following, in order to avoid duplicate explanations, the description of the points common to both embodiments will be omitted, and the differences between the two embodiments will be described in detail.

<Outline of differences between the first embodiment and the second embodiment>
In the first embodiment, in performing the learning, the image processing device 10a performs preprocessing of matching the distance image acquired from the three-dimensional measuring device 40 and creating the matching point cloud information. Then, the image processing device 10a uses the matching point group information as input data of the learning model and uses the evaluation value given to the matching point group information as a label to construct a learning model by machine learning. The extraction position was selected using the constructed learning model.

Alternatively, in the first embodiment, the image processing device 10a has the same size as the input to the trained model over the entire range of the distance image acquired from the three-dimensional measuring device 40 (that is, the same size as the matching point cloud information). The distance image of was cut out as a candidate for the extraction position. Then, the image processing device 10a uses the three-dimensional point cloud information of the cut-out extraction position candidate as input data of the learning model, and uses the evaluation value given to the three-dimensional point cloud information as a label, thereby performing machine learning. The learning model was constructed and the extraction position was selected using the constructed learning model.

On the other hand, in the second embodiment, the preprocessing such as matching and clipping is omitted, and the entire distance image acquired from the three-dimensional measuring machine 40 is used as the input data of the learning model, so that the learning model by machine learning can be obtained. Build and select the extraction position using the built learning model.

Although it depends on the environment to which the second embodiment is applied, by omitting the preprocessing in this way, in the second embodiment, it is possible to efficiently perform the arithmetic processing and facilitate the implementation. It becomes. Further, in the second embodiment, by using the entire distance image as the input data of the learning model in this way, it is possible to consider the influence between pixels far apart in the image.

<Functional block of image processing device 10b>
Next, in order to omit the preprocessing and use the entire distance image as the input data of the learning model, each functional block included in the image processing device 10b will be described with reference to FIG.
In FIG. 10, similarly to FIG. 2, the components other than the image processing device 10b in the robot system 1b are collectively shown as the environment 100.

The image processing device 10b is different in that the selection processing unit 15 is provided in place of the selection processing unit 11 included in the image processing device 10a. That is, the image processing device 10b includes a selection processing unit 15, a display unit 12, an operation reception unit 13, and a learning unit 14.
Further, the selection processing unit 15 includes a teacher data storage unit 151, an annotation processing unit 152, and an extraction position selection unit 153.

The teacher data storage unit 151 is a unit that stores teacher data for performing machine learning. The distance image input from the three-dimensional measuring device 40 is stored in the teacher data storage unit 151 as input data in the teacher data. Further, the label generated by the annotation processing unit 152 is stored in the teacher data storage unit 151 as input data in the teacher data.
The input data and the label are associated with each other when the label is stored by the annotation processing unit 152.

The annotation processing unit 152 is a part that generates a label included in the teacher data. The annotation processing unit 152 acquires a distance image from the teacher data storage unit 151 in order to generate a label. Further, the annotation processing unit 152 displays the acquired distance image on the display unit 12. A display example of the distance image is as described above with reference to FIG.

With reference to the displayed distance image, the user teaches the extraction position in the same manner as in the first embodiment. Specifically, the operation reception unit 13 realized by the mouse or the touch panel is used to specify the take-out position to teach the take-out position.

The operation reception unit 13 notifies the annotation processing unit 152 of the teaching position, which is the teaching position. The annotation processing unit 152 draws on the teaching position on the distance image so that the user can recognize the notified teaching position. The drawing is performed by a method that is easy for the user to grasp, such as changing the color of the pixel at the teaching position on the distance image. A display example when drawing is performed is as described above with reference to FIG.

Here, in the first embodiment, the user teaches the take-out position by a point. On the other hand, in the second embodiment, the user teaches the take-out position as a predetermined area. For example, in the second embodiment, the user teaches the take-out position by coloring the take-out area.

Based on the instruction by the user, the annotation processing unit 152 has an attribute (for example, the extraction position of the work 50) indicating whether or not it is the extraction position (that is, the teaching position) of the work 50 for each pixel included in the entire distance image. Is "1". An image to which "0") is assigned to a position other than the extraction position of the work 50 is generated. In the following, this image will be referred to as a "label map".

In the second embodiment, this label map is used as a label. Further, when generating the label map, the annotation processing unit 152 does not assign an attribute indicating whether or not it is the extraction position of the work 50 for each pixel, but further sets it to 1 / s (s is an arbitrary natural number). The resolution may be increased and an attribute indicating whether or not the work 50 is taken out may be assigned to each 1 / s pixel. Further, since the 3D point cloud information is not required as the label, the 3D point cloud information included in the distance image may be deleted in the label map. Therefore, the label map is based on the information on the two-dimensional coordinates of each pixel (or each pixel set to 1 / s) in the distance image and the extraction position of the work 50 for each pixel (or each pixel set to 1 / s). It is an image that includes information on attributes indicating whether or not it exists.

The annotation processing unit 152 uses the generated label map as a label, uses the distance image used for teaching from the user as input data, and generates teacher data by associating the two. Then, the annotation processing unit 152 stores the generated teacher data in the teacher data storage unit 151. The annotation processing unit 152 does not store the input data and the label in the teacher data storage unit 151 in association with each other, but separately stores the input data in the teacher data storage unit 151 and links the two to obtain the teacher data. You may do so.
When performing machine learning, the learning processing unit 141 of the second embodiment performs machine learning using the teacher data stored in the teacher data storage unit 151. Specifically, in the second embodiment, a learning model is constructed in which an image similar to a label map is output when a distance image is input by machine learning by the learning processing unit 141. That is, a learning model is constructed in which an image segmented from the retrievable area is output.

This machine learning can be realized by, for example, inputting an image and performing some estimation (for example, classification) for all pixel pixels in the image. Examples of such a method include Semantic Segmentation. Semantic Segmentation is a technology aimed at application to automatic driving of automobiles. For example, image data of a photograph can be input and color-coded for each area such as a car or a pedestrian.
In the second embodiment, as described above, machine learning is performed by using the input data as a distance image and the label as an annotation for the distance image (for example, an image in which a human has painted a position where the work 50 can be taken out). , Evaluation value estimation for all pixels can be performed at once. Therefore, the pre-processing such as the matching process and the clipping process in the first embodiment can be omitted.

Specifically, the machine learning of the second embodiment can be realized by using, for example, the convolution encoder-decoder disclosed in the following references.

<References>
Vijay Badrinarayanan, Alex Kendall, Roberto Cipolla, "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation", [online], August 10, 2016, [Search September 10, 2017], Internet <URL : Https://arxiv.org/pdf/1511.00561.pdf>

The learning processing unit 141 builds a trained model by machine learning. Then, the learning processing unit 141 stores the constructed trained model in the trained model storage unit 142.

The extraction position selection unit 153 is a portion that selects the extraction position based on the evaluation value map output by the estimation processing unit 143. The selection of the extraction position by the extraction position selection unit 153 and the details of the evaluation value map will be described below.
When the take-out position selection unit 153 acquires a distance image from the dimension measuring device 40 in order to select the take-out position, the take-out position selection unit 153 outputs the acquired distance image to the estimation processing unit 143. The estimation processing unit 143 inputs the input distance image as input data to the trained model stored in the trained model storage unit 142. In response to this input, the trained model outputs an image in which the retrievable area is segmented. This output image is called an "evaluation value map".
The evaluation value map has the same data structure as the map for labels, and has information on the two-dimensional coordinates of each pixel (or each pixel as 1 / s) and a work for each pixel (or each pixel as 1 / s). It is an image including the information of the attribute indicating whether or not it is the extraction position of 50.
The estimation processing unit 143 takes out the evaluation value map and outputs it to the position selection unit 153.

The take-out position selection unit 153 selects the take-out position of the work 50 based on the evaluation value map. Specifically, each section segmented as the take-out position of the work 50 is a candidate for the take-out position. Then, based on the coordinate information indicating the region of the candidate of the extraction position, the region corresponding to the candidate of the extraction position is specified on the distance image as the input data. Then, the extraction position is selected by performing known point cloud processing or image processing on the specified area.

The take-out position selection unit 153 outputs the selected take-out position to the robot control device 20. After that, the robot control device 20 generates a control signal based on the take-out position as in the first embodiment. Then, the robot 30 takes out the work 50 by hand based on the control signal.

With the configuration described above, in the second embodiment, preprocessing such as matching and clipping is omitted, and the entire distance image acquired from the three-dimensional measuring machine 40 is used as the input data of the learning model, so that the learning model by machine learning is used. Can be constructed and the extraction position can be selected using the constructed learning model.

<Operation of the second embodiment>
Next, the operation of the second embodiment will be described with reference to the flowchart of FIG.
In step S31, the teacher data storage unit 151 stores the distance image generated by the three-dimensional measuring machine 40 measuring the works 50 piled up in bulk.
In step S32, the annotation processing unit 152 causes the display unit 12 to display the distance image stored in the teacher data storage unit 151.
In step S33, the annotation processing unit 152 draws the teaching position on the distance image based on the teaching of the taking-out position from the user received by the operation receiving unit 13.

In step S34, the annotation processing unit 152 generates a label map based on the instruction from the user. The annotation processing unit 152 stores the label map and the distance image as teacher data in the teacher data storage unit 151.

In step S35, the learning processing unit 141 performs learning using the distance image as input data in the teacher data stored in the teacher data storage unit 151 and the label map corresponding to the distance image as a label. As a result, the trained model is created and stored in the trained model storage unit 142.

In step S36, the take-out position selection unit 153 acquires a distance image generated by measuring the works 50 piled up in bulk from the three-dimensional measuring machine 40. Then, the extraction position selection unit 153 outputs the acquired distance image to the estimation processing unit 143.

In step S37, the estimation processing unit 143 inputs the input distance image to the trained model stored in the trained model storage unit 142. Then, the estimation processing unit 143 acquires the evaluation value map as the output of the trained model. The estimation processing unit 143 takes out the acquired evaluation value map and outputs it to the position selection unit 153.

In step S38, the extraction position selection unit 153 selects the extraction position based on the evaluation value map. The take-out position selection unit 153 transmits the selected take-out position to the robot control device 20.

In step S39, the robot control device 20 takes out the work 50 by controlling the robot 30 based on the received take-out position. When a plurality of work 50s are taken out, a plurality of take-out positions may be selected in step S38, and take-out may be executed for each of the plurality of take-out positions in step S39. However, since the state of bulk loading of the work 50 changes by taking out the work 50, the process may be repeated by returning to step S36 each time the work 50 is taken out.

With the configuration described above, in the second embodiment, preprocessing such as matching and clipping is omitted, and the entire distance image acquired from the three-dimensional measuring machine 40 is used as the input data of the learning model, so that the learning model by machine learning is used. Can be constructed and the extraction position can be selected using the constructed learning model.

<Collaboration of hardware and software>
Each of the devices included in the above robot system can be realized by hardware, software, or a combination thereof. In addition, a machine learning method performed in collaboration with each device included in the above robot system can also be realized by hardware, software, or a combination thereof. Here, what is realized by software means that it is realized by a computer reading and executing a program.

Programs can be stored and supplied to a computer using various types of non-transitory computer readable medium. Non-transient computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory), CD- R, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable medium. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

<Modification of the embodiment>
Further, although each of the above-described embodiments is a preferred embodiment of the present invention, the scope of the present invention is not limited to the above-described embodiment, and a combination of the respective embodiments and a gist of the present invention can be described. It is possible to carry out in a form with various changes within a range that does not deviate.

<Modification example 1>
In each of the above-described embodiments, the image processing device 10a or the image processing device 10b and the robot control device 20 are separate devices, but these devices may be realized as one.

<Modification 2>
Further, in each of the above-described embodiments, the robot control device 20 and the image processing device 10a or the image processing device 10b are shown in the vicinity thereof, but these are shown via a network such as a LAN (Local Area Network) or the Internet. It may be located far away.

Further, one image processing device 10a or an image processing device 10b may be connected to a plurality of robot control devices 20. Then, one image processing device 10a or an image processing device 10b may perform learning based on the teacher data acquired from each of the plurality of robot control devices 20.

<Modification example 3>
Further, in each of the above-described embodiments, the three-dimensional measuring machine 40 has been described as being fixedly installed at a predetermined position, but it does not necessarily have to be fixed at a predetermined position. For example, if the position of the three-dimensional measuring machine 40 in the machine coordinate system for controlling the robot 30 is known, the installation position may be changed during the implementation such as attaching to the arm of the robot 30.

<Modification example 4>
Further, in each of the above-described embodiments, it is assumed that the image processing device 10a or the image processing device 10b is realized by one device, but each function of the image processing device 10a or the image processing device 10b can be appropriately performed by a plurality of devices. It may be a distributed processing system that distributes to. For example, a distributed processing system may be used in which the functions of the selection processing unit 11 of the image processing device 10a or the selection processing unit 15 of the image processing device 10b and the functions of the learning unit 14 are appropriately distributed to a plurality of devices. In this case, it may be a distributed processing system in which each functional block included in the learning unit 14 is appropriately distributed to a plurality of devices. Further, each function of the image processing device 10a or the image processing device 10b may be realized by using the virtual server function or the like on the cloud.

<Modification 5>
Further, in the first embodiment described above, the matching process is performed, and the trained model is constructed based on the success or failure of the extraction result actually tried at the matched position. In addition, the learning model constructed by the matching process in this way was used, and then the clipping process was performed.
However, when performing the clipping process, it is not always necessary to use the learning model constructed by the matching process.

In this case, the learning processing unit 141 constructs a learning model assuming that the extraction at the teaching point taught by the user is successful. That is, the learning unit 14 creates the teacher data using the image data in the vicinity of the teaching position as the input data and the evaluation value indicating that the extraction is successful (considered) as the label. Then, the learning processing unit 141 constructs a learning model by learning with the teacher data.

The selection processing unit 11 can perform the clipping process using the learning model constructed in this way. As a result, the selection processing unit 11 can construct the learning model and select the extraction position without performing the matching process or the actual extraction.
In this case, the learning processing unit 141 may acquire the non-teaching position neighborhood image data which is the data near the position (non-teaching position) different from the teaching position from the user. Then, the learning processing unit 141 may create teacher data indicating a failure example by using the non-teaching position vicinity image data as input data and using the evaluation value indicating that the retrieval has failed (considered) as a label. Good. Then, the learning processing unit 141 may further perform learning using the teacher data indicating this failure example. In this case, the non-teaching position may be selected by the user, or may be randomly selected from a position other than the teaching position.

Here, the following additional notes will be further disclosed with respect to the above embodiments.
(Appendix 1)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
A display unit that displays the distance image generated by the three-dimensional measuring device, and
On the displayed distance image, a reception unit that receives an instruction of a take-out position for taking out by the hand is provided.
The robot is a robot system that takes out at least one of the plurality of workpieces by the hand based on the taught take-out position.
(Appendix 2)
The robot system according to Appendix 1, wherein the display unit draws the taught take-out position on the displayed distance image.
(Appendix 3)
The information of the taught extraction position and the three-dimensional point cloud in the vicinity thereof is stored as search information, and the distance image is searched by the search information to select a new extraction position. Equipped with a position selection unit
The robot system according to Appendix 1 or Appendix 2, wherein the robot takes out each work by the hand at the new take-out position selected by the take-out position selection unit.
(Appendix 4)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
A display unit that displays the distance image generated by the three-dimensional measuring device and the extraction position taught on the distance image.
On the displayed distance image, a reception unit that receives an instruction of a take-out position for taking out by the hand, and a reception unit.
The information of the three-dimensional point group in the teaching position and its vicinity is used as input data, and at least the evaluation value according to the instruction for the information of the three-dimensional point group used as the input data or the evaluation value according to the success or failure of the extraction. A learning unit that builds a learning model that outputs evaluation values for information on 3D point groups input as input data by performing machine learning using either one as a label.
With
The robot is a robot system that takes out at least one of the plurality of workpieces by the hand based on the taught take-out position.
(Appendix 5)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
A display unit that displays the distance image generated by the three-dimensional measuring device and the extraction position taught on the distance image.
On the displayed distance image, a reception unit that receives an instruction of a take-out position for taking out by the hand, and a reception unit.
The information of the three-dimensional point group in the teaching position and its vicinity is used as input data, and at least the evaluation value according to the instruction for the information of the three-dimensional point group used as the input data or the evaluation value according to the success or failure of the extraction. A learning unit that builds a learning model that outputs evaluation values for information on 3D point groups input as input data by performing machine learning using either one as a label.
Evaluation of the information of the three-dimensional point cloud output by cutting out the distance image of a predetermined region from the distance image and inputting the information of the three-dimensional point cloud of the cut out distance image into the learning model as input data. A take-out position selection unit that selects a new take-out position based on the value,
With
The robot takes out at least one of the plurality of works by the hand based on the taught take-out position, and takes out each work by the hand at the new take-out position selected by the take-out position selection unit. system.
(Appendix 6)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
An evaluation value map generated by estimating a take-out position for taking out at least one of the plurality of works by the hand based on the distance image generated by the three-dimensional measuring machine, and at least one. An estimation unit that outputs an evaluation value map consisting of two evaluation values,
A regioselective unit that selects at least one extraction position of the plurality of workpieces to be extracted by the hand based on the evaluation value map output by the estimation unit.
With
The robot is a robot system that takes out at least one of the plurality of workpieces by the hand based on the take-out position selected by the position selection unit.
(Appendix 7)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
A teacher data storage unit that stores the distance image generated by the three-dimensional measuring machine as teacher data for machine learning, and a teacher data storage unit.
A display unit that displays the distance image stored in the teacher data storage unit, and
A reception unit that receives teaching at least one teaching position based on the distance image displayed on the display unit, and
A label map showing at least one teaching position is generated based on the teaching position received by the reception unit, and the teacher data is associated with the label map and the distance image stored in the teacher data storage unit as a data set. An annotation processing unit that saves in the storage unit and
A learning processing unit that uses the data set stored in the teacher data storage unit as input, performs machine learning, and outputs a trained model.
A trained model storage unit that stores the trained model output from the training processing unit, and a trained model storage unit.
The trained model stored in the trained model storage unit and at least one of the new plurality of works based on a distance image of the new plurality of works newly generated by the three-dimensional measuring machine. An estimation unit that outputs an evaluation value map that is an evaluation value map generated by estimating the extraction position when taking out by hand and consists of at least one evaluation value, and an estimation unit.
A regioselective unit that selects at least one extraction position of the new plurality of workpieces to be extracted by the hand based on the evaluation value map output by the estimation unit.
With
The robot is a robot system that takes out at least one of the new plurality of workpieces by the hand based on the take-out position selected by the position selection unit.
(Appendix 8)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
This is a work removal method performed by a robot system equipped with
A display step for displaying the distance image generated by the three-dimensional measuring device, and
On the displayed distance image, a reception step for receiving an instruction of a take-out position for taking out by the hand, and a reception step.
With
A work taking-out method in which the robot takes out at least one of the plurality of works by the hand based on the taught taking-out position.
(Appendix 9)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
This is a work removal method performed by a robot system equipped with
A display step for displaying the distance image generated by the three-dimensional measuring device and the extraction position taught on the distance image, and
On the displayed distance image, a reception step for receiving an instruction of a take-out position for taking out by the hand, and a reception step.
The information of the three-dimensional point group in the teaching position and its vicinity is used as input data, and at least the evaluation value according to the instruction for the information of the three-dimensional point group used as the input data or the evaluation value according to the success or failure of the extraction. A learning step to build a learning model that outputs an evaluation value for information on a 3D point group input as input data by performing machine learning using either one as a label.
With
A work taking-out method in which the robot takes out at least one of the plurality of works by the hand based on the taught taking-out position.
(Appendix 10)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
This is a work removal method performed by a robot system equipped with
A display step for displaying the distance image generated by the three-dimensional measuring device and the extraction position taught on the distance image, and
On the displayed distance image, a reception step for receiving an instruction of a take-out position for taking out by the hand, and a reception step.
The information of the three-dimensional point group in the teaching position and its vicinity is used as input data, and at least the evaluation value according to the instruction for the information of the three-dimensional point group used as the input data or the evaluation value according to the success or failure of the extraction. A learning step to build a learning model that outputs an evaluation value for information on a 3D point group input as input data by performing machine learning using either one as a label.
Evaluation of the information of the three-dimensional point cloud output by cutting out the distance image of a predetermined region from the distance image and inputting the information of the three-dimensional point cloud of the cut out distance image into the learning model as input data. A take-out position selection step that selects a new take-out position based on the value,
With
The robot takes out at least one of the plurality of works by the hand based on the taught take-out position, and takes out each work by the hand at the new take-out position selected by the take-out position selection step. How to take out.
(Appendix 11)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
This is a work removal method performed by a robot system equipped with
An evaluation value map generated by estimating a take-out position for taking out at least one of the plurality of works by the hand based on the distance image generated by the three-dimensional measuring machine, and at least one. An estimation step that outputs an evaluation value map consisting of two evaluation values, and
A position selection step of selecting at least one extraction position of the plurality of workpieces to be extracted by the hand based on the evaluation value map output in the estimation step, and a position selection step.
With
A work taking-out method in which at least one of the plurality of works is taken out by the hand based on the taking-out position selected by the robot in the position selection step.
(Appendix 12)
A 3D measuring machine that generates distance images of multiple workpieces,
A robot having a hand for taking out at least one of the plurality of workpieces, and
This is a work removal method performed by a robot system equipped with
A teacher data storage step of storing the distance image generated by the three-dimensional measuring machine as teacher data for machine learning in the teacher data storage unit, and
A display step for displaying the distance image stored in the teacher data storage unit, and
A reception step that accepts teaching of at least one teaching position based on the distance image displayed in the display step, and
A label map showing at least one teaching position is generated based on the teaching position received in the reception step, and the teacher is associated with the label map and the distance image stored in the teacher data storage unit as a data set. Annotation processing step to save in the data storage part,
A learning process step in which the data set stored in the teacher data storage unit is input, machine learning is performed, and a trained model is output.
A trained model storage step for storing the trained model output in the training process step in the trained model storage unit, and a trained model storage step.
The trained model stored in the trained model storage unit and at least one of the new plurality of works based on a distance image of the new plurality of works newly generated by the three-dimensional measuring machine. An estimation step that outputs an evaluation value map that is an evaluation value map generated by estimating the extraction position when taking out by hand and consists of at least one evaluation value, and an estimation step.
A position selection step of selecting at least one extraction position of the new plurality of workpieces to be extracted by the hand based on the evaluation value map output in the estimation step.
With
A work taking-out method for taking out at least one of the new plurality of works by the hand based on the taking-out position selected by the robot in the position selection step.

1a, 1b Robot system 10a, 10b Image processing device 11 Selection processing unit 111 Selection data storage unit 112 Annotation processing unit 113 Matching unit 114 Clipping unit 12 Display unit 13 Operation reception unit 14 Learning unit 141 Learning processing unit 142 Learned model storage Unit 143 Estimation processing unit 15 Selection processing unit 151 Teacher data storage unit 152 Annotation processing unit 153 Extraction position selection unit 20 Robot control device 30 Robot 40 3D measuring machine 50 Work 60 Container

Claims (19)

And the robot,
An image processing device that acquires a distance image including a plurality of objects generated by a measuring machine and analyzes the distance image, and an image processing device.
A control device for controlling the operation of taking out an object by the robot is provided.
The image processing device is
And obtaining for extracting at least one of the robots of said plurality of objects, the teaching position in the range image,
Obtained from the range image, applied to an object point group information that definitive at least the teaching position, the teaching, and, at least evaluation value according to any one of the extraction of the object in accordance with the teachings Results To do and
By performing machine learning using the information of the point cloud of the object as input data and using the evaluation value as a label, it is possible to learn a learning model that outputs an evaluation value for the input point cloud information. Consists of
The control device controls the operation of taking out an object by the robot based on the evaluation value output by the learning model of the image processing device.
Robot system. The teaching of the position indicates the position as either a point or a region.
The robot system according to claim 1. When the extraction fails, the image processing device gives a lower evaluation value than when the extraction is successful.
The robot system according to claim 1 or 2 . The evaluation value output by the learning model is information indicating a recoverable region in the input point cloud information.
The robot system according to any one of claims 1 to 3 . The learning model is a neural network,
The robot system according to any one of claims 1 to 4 . The result of taking out the object based on the teaching is obtained by attempting to take out the object at the taught position by the robot.
The robot system according to any one of claims 1 to 5 . The measuring instrument is either a distance image sensor or a stereo camera.
The robot system according to any one of claims 1 to 6. The size of the range of the point cloud input to the learning model is preset based on the size of the object to be extracted.
The robot system according to any one of claims 1 to 7 . The image processing device gives the evaluation value according to the time required for taking out.
The robot system according to any one of claims 1 to 8 . The learning model outputs an image in which the retrievable area is segmented.
The robot system according to any one of claims 1 to 9 . The image processing device selects an object extraction position based on the evaluation value output by the learning model.
The control device controls the taking-out operation of the object by the robot based on the selected taking-out position.
The robot system according to any one of claims 1 to 10 . The image processing device and the control device are composed of one device.
The robot system according to any one of claims 1 to 11 . The measuring instrument is attached to the robot,
The robot system according to any one of claims 1 to 12 . Based on the result of taking out the object based on the teaching, the evaluation value is given to the information of the point cloud of the object.
The robot system according to any one of claims 1 to 13 . The learning model is learned by the image processing device in the robot system according to any one of claims 1 to 14.
Model generation method . A model generation method executed by at least one processor.
A step of acquiring a position instruction in a distance image including the plurality of objects generated by a measuring instrument for extracting at least one of the plurality of objects by a robot.
An evaluation value corresponding to at least one of the teaching and the result of taking out the object based on the teaching is given to the information of the point cloud of the object at least at the taught position obtained from the distance image. Steps to do and
A pre-Symbol object point group of the input information data, comprising by performing machine learning the evaluation value as a label, and the step of learning a learning model for outputting an evaluation value for information group of points is inputted, the ,
The operation of taking out an object by the robot is controlled based on the evaluation value output by the learning model.
Model generation method. The step of assigning the evaluation value is to assign the evaluation value to the information of the point cloud of the object based on the result of taking out the object based on the teaching.
The model generation method according to claim 16 . A step of acquiring a position instruction in a distance image including the plurality of objects generated by a measuring instrument for extracting at least one of the plurality of objects by a robot.
Obtained from the range image, the object point group information that definitive at least the teaching position, the teaching, and the evaluation value corresponding to at least one of the results of extraction of the object based on the teachings Steps to give and
A step to the previous SL object point group of the input information data, by performing machine learning the evaluation value a label, for learning the learning model for outputting an evaluation value for information point group is input,
The A model generation program to be executed by the at least one processor,
The operation of taking out an object by the robot is controlled based on the evaluation value output by the learning model.
Model generator. The step of assigning the evaluation value is to assign the evaluation value to the information of the point cloud of the object based on the result of taking out the object based on the teaching.
The model generation program according to claim 18 .

Images