JP6671694B1 - Machine learning device, machine learning system, data processing system, and machine learning method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To realize automation of gripping / transporting work by a robot hand with a simple configuration. SOLUTION: One or more three-dimensional coordinate data of a robot hand when a robot hand on a simulator 2 that grips any one or a plurality of works W undergoes a gripping operation and succeeds in gripping; And a data set storage unit 32 for acquiring a learning data set from the simulator 2 and storing a plurality of learning data sets including the two-dimensional imaged image data of the work W captured by the two-dimensional image pickup device ID on the simulator 2 from a predetermined angle of view. A learning unit 33 that learns a learning model that infers the three-dimensional coordinates of the robot hand in the real world from a two-dimensional captured image in which one or a plurality of works W are captured by the two-dimensional imaging device ID from the same angle of view as the predetermined angle of view. And a learned model storage unit 34 that stores the learning model learned by the learning unit 33. [Selection diagram] Fig. 1

Description

  The present invention is obtained by a machine learning device, a machine learning system, and a machine learning method for learning an operation of taking out one or a plurality of randomly arranged works, and the machine learning device, the machine learning system, and the machine learning method. The present invention relates to a data processing system using a learned model.

  2. Description of the Related Art Conventionally, attempts have been made to use a robot hand (manipulator) to hold and transport a work of a predetermined shape that is randomly arranged in a box-shaped tray (including so-called “bulk stacking”). Then, as in Patent Document 1, for example, an attempt is made to use a machine learning device to automatically execute such an operation without human hands (for example, an operation of inputting control information). The machine learning device described in Patent Literature 1 forms a three-dimensional map of a workpiece using a three-dimensional measuring device including a plurality of cameras and the like. Machine learning is performed using command data for the robot.

JP 2017-030135 A

  As described above, the machine learning device described in Patent Literature 1 has an essential requirement to acquire a three-dimensional map of a workpiece using a three-dimensional measuring device including a plurality of cameras and the like during machine learning. However, a three-dimensional measuring device is a complicated device having a large number of components compared to a general two-dimensional imaging device (camera), such as requiring a plurality of cameras, and realizing the machine learning device is costly. There is a problem in.

  In view of the above, it is an object of the present invention to realize automation of a gripping / transporting operation of a work by a robot hand with a simple configuration, and to reduce costs required for the realization.

  In order to achieve the above object, the machine learning device 3 according to the first aspect of the present invention includes, as shown in FIGS. 1 and 5, one or a plurality of workpieces arranged in a predetermined area VWA of the simulator 2. One or a plurality of three-dimensional coordinate data of the robot hand R when the robot hand R on the simulator 2 gripping any one of the W through the gripping operation succeeds, and the robot hand R successfully grips Two-dimensional captured image data obtained by capturing the one or a plurality of works W arranged in the predetermined area from a predetermined angle of view using a two-dimensional imaging device ID on the simulator before the gripping operation when performing the holding operation. A data set storage unit 32 for acquiring a plurality of sets of learning data sets from the simulator and storing a plurality of sets of learning data sets; Three-dimensional coordinates of the robot hand R in the real world from a two-dimensional captured image obtained by capturing one or a plurality of works W arranged in a predetermined area RWA in the field from the same angle of view as the predetermined angle of view using a two-dimensional imaging device ID. And a learned model storage unit 34 that stores the learning model learned by the learning unit 33.

  With this configuration, by applying the learned model obtained by the machine learning device to data processing in bulk picking, it is possible to automate bulk picking work without requiring a complicated device such as a three-dimensional measuring device. Can be realized. In addition, since a simulator is used for this machine learning, collection of a learning data set for machine learning can be realized in a short time and stably.

  A machine learning system 3 according to a second embodiment of the present invention includes a simulator 2 and a machine learning device 3, as shown in FIGS. 1 and 5, for example: the simulator 2 is arranged in a predetermined area VWA. One or a plurality of three-dimensional coordinate data of the robot hand R when the robot hand R on the simulator holding one of the one or a plurality of works W succeeds in holding through a holding operation; A two-dimensional captured image obtained by capturing one or a plurality of works W arranged in the predetermined region from a predetermined angle of view using a two-dimensional imaging device ID on the simulator 2 before the gripping operation when R successfully grips. A function of generating a learning data set including data and a plurality of learning data sets from the simulator 2 A plurality of sets of the learning data sets, whereby one or a plurality of workpieces W arranged in a predetermined area RWA in the real world are input to the predetermined set by the two-dimensional imaging device ID. A learning unit 33 for learning a learning model for inferring the three-dimensional coordinates of the robot hand R in the real world from a two-dimensional captured image captured from the same angle of view as the angle of view; And a learned model storage unit 34 for storing.

  With this configuration, by applying the learned model obtained by this machine learning system to data processing in bulk picking, automation of bulk picking work can be automated without the need for a complicated device such as a three-dimensional measuring device. Can be realized. In addition, since a simulator is used for this machine learning, collection of a learning data set for machine learning can be realized in a short time and stably.

  The machine learning device according to the third aspect of the present invention is configured such that, as shown in FIGS. 1 and 5, for example, a robot hand R that grips one or a plurality of workpieces arranged in a predetermined area RWA is gripped. Three-dimensional coordinate data of the robot hand R when the gripping is successful through the operation, and one or a plurality of the plurality of the robot hands R arranged in the predetermined area RWA before the gripping operation when the robot hand R successfully grips. A data set storage unit 32 that stores a plurality of learning data sets each including two-dimensional captured image data obtained by capturing the work W from a predetermined angle of view using a two-dimensional imaging device ID; and inputs a plurality of the learning data sets. A learning unit 33 for learning a learning model for inferring the three-dimensional coordinates from the two-dimensional captured image; and the learning model learned by the learning unit 33. Including; a learned model storage unit 34 for storing.

  With this configuration, by applying the learned model obtained by the machine learning device to data processing in bulk picking, it is possible to automate bulk picking work without requiring a complicated device such as a three-dimensional measuring device. Can be realized.

  The data processing system 100 according to the fourth aspect of the present invention captures one or a plurality of works W arranged in a predetermined area RWA from a predetermined angle of view using a two-dimensional imaging device ID, for example, as shown in FIG. An acquisition unit 110 for acquiring a two-dimensional captured image; and inputting the two-dimensional captured image acquired by the acquisition unit to a learned model generated by the machine learning device 3 of the first to third aspects. An inference unit 140 for inferring three-dimensional coordinates of the robot hand.

  With this configuration, the three-dimensional coordinates of the robot hand capable of holding the workpiece can be inferred from only the image data captured by the relatively inexpensive two-dimensional imaging device having a simple structure. Bulk picking can be realized with a simple configuration and at low cost.

  The data processing system 100B according to the fifth aspect of the present invention, when the inference unit 140 infers a plurality of three-dimensional coordinates as shown in FIG. Further includes a specifying unit 150 for specifying the three-dimensional coordinates of.

  With this configuration, one three-dimensional coordinate can be selected from the plurality of three-dimensional coordinates inferred by the inference unit 140, and a more accurate gripping operation can be performed.

  A machine learning method according to a sixth aspect of the present invention uses, for example, a computer as shown in FIG. 4: a robot hand R holding one of a plurality of works W arranged in a predetermined area VWA. 3D coordinate data of the robot hand R when the robot hand R succeeds in grasping via the grasping operation, and 1 or 3 arranged in the predetermined area VWA before the grasping operation when the robot hand R succeeds in grasping. Storing a plurality of sets of learning data sets each including two-dimensional captured image data obtained by capturing a plurality of works W from a predetermined angle of view using a two-dimensional imaging device ID; by inputting a plurality of sets of the learning data sets, Learning a learning model for inferring the three-dimensional coordinates from the two-dimensional captured image; step S24; storing the learned learning model; Including; 6 and.

  With this configuration, by applying the learned model obtained by this machine learning method to data processing in bulk picking, it is possible to automate the bulk picking work without requiring a complicated device such as a three-dimensional measuring device. Can be realized.

  The machine learning device 3A according to the seventh aspect of the present invention grasps one of one or a plurality of works W arranged in a predetermined area VWA of the simulator 2A, for example, as shown in FIGS. One or a plurality of three-dimensional coordinate data and angle data of the robot hand R when the robot hand R on the simulator 2A succeeds in grasping through the grasping operation, and the grasp when the robot hand R succeeds in grasping. A learning data set including two-dimensional captured image data obtained by capturing one or a plurality of workpieces W arranged in the predetermined region VWA from a predetermined angle of view using a two-dimensional imaging device ID on the simulator before the operation. A data set storage unit 32 that acquires from the simulator 2A and stores a plurality of sets; Three-dimensional coordinates of the robot hand R in the real world from a two-dimensional captured image obtained by capturing one or a plurality of workpieces W arranged in a predetermined region RWA in the world from the same angle of view as the predetermined angle of view using a two-dimensional imaging device ID. And a learning unit 33 that learns a learning model that infers angles and angles; and a learned model storage unit 34 that stores the learning model learned by the learning unit 33.

  With this configuration, by applying the learned model obtained by the machine learning device to data processing in bulk picking, it is possible to automate bulk picking work without requiring a complicated device such as a three-dimensional measuring device. Can be realized. In addition, since a simulator is used for this machine learning, collection of a learning data set for machine learning can be realized in a short time and stably. Further, since the learned model outputs not only the three-dimensional coordinate data but also the angle data, it is possible to always accurately grasp the workpiece regardless of the shape of the robot hand.

  A machine learning system 1A according to an eighth aspect of the present invention includes a simulator 2A and a machine learning device 3A, for example, as shown in FIGS. 7 and 5: the simulator 2A is arranged in a predetermined area VWA. One or a plurality of three-dimensional coordinate data and angle data of the robot hand R when the robot hand R on the simulator 2A gripping any one or a plurality of workpieces W succeeds in gripping through a gripping operation. Before the gripping operation when the robot hand R successfully grips, one or a plurality of works W arranged in the predetermined area VWA are imaged from a predetermined angle of view by a two-dimensional imaging device ID on the simulator 2A. And a function for generating a learning data set including the obtained two-dimensional captured image data. A data set storage unit 32 for acquiring and storing a plurality of sets of the learning data sets from a plurality of learning data sets; and inputting a plurality of sets of the learning data sets to one or more workpieces arranged in a predetermined area RWA in the real world. A learning unit 33 that learns a learning model that infers three-dimensional coordinates and angles of the robot hand R in the real world from a two-dimensional captured image obtained by capturing W from the same angle of view as the predetermined angle of view using the two-dimensional imaging device ID; A learned model storage unit 34 for storing the learning model learned by the learning unit 33.

  With this configuration, by applying the learned model obtained by this machine learning system to data processing in bulk picking, automation of bulk picking work can be automated without the need for a complicated device such as a three-dimensional measuring device. Can be realized. In addition, since a simulator is used for this machine learning, collection of a learning data set for machine learning can be realized in a short time and stably. Further, since the learned model outputs not only the three-dimensional coordinate data but also the angle data, it is possible to always accurately grasp the workpiece regardless of the shape of the robot hand.

  The machine learning device according to the ninth aspect of the present invention includes, as shown in FIGS. 7 and 5, for example, a robot hand R that grasps one or a plurality of works W arranged in a predetermined area RWA. The three-dimensional coordinate data and the angle data of the robot hand R when the grip is successful after the grip operation, and the robot hand R are arranged in the predetermined area RWA before the grip operation when the robot hand R successfully grips. A data set storage unit 31 that stores a plurality of learning data sets each including two-dimensional captured image data obtained by capturing one or a plurality of works W from a predetermined angle of view using a two-dimensional imaging device ID; A learning unit 33 for learning a learning model for inferring the three-dimensional coordinates and angles from the two-dimensional captured image by inputting a plurality of sets; Including; a learned model storage unit 34 for storing the learning has been said learning model.

  With this configuration, by applying the learned model obtained by the machine learning device to data processing in bulk picking, it is possible to automate bulk picking work without requiring a complicated device such as a three-dimensional measuring device. Can be realized. Further, since the learned model outputs not only the three-dimensional coordinate data but also the angle data, it is possible to always accurately grasp the workpiece regardless of the shape of the robot hand.

  The data processing system according to the tenth aspect of the present invention captures one or a plurality of works W arranged in a predetermined area RWA from a predetermined angle of view by a two-dimensional imaging device as shown in FIGS. 7 and 5, for example. An acquisition unit 110 for acquiring the obtained two-dimensional captured image data; and inputting the two-dimensional captured image data acquired by the acquisition unit to a learned model generated by the machine learning device according to the seventh to ninth aspects. And an inference unit 140 for inferring three-dimensional coordinates and angles of the robot hand.

  With this configuration, the three-dimensional coordinates and angle of the robot hand capable of holding the workpiece can be inferred only from image data captured by a relatively inexpensive two-dimensional imaging device having a simple structure. In addition, bulk picking can be realized with high precision and without requiring human labor.

  The data processing system 100B according to the eleventh aspect of the present invention is configured such that, when the inference unit 140 infers a plurality of three-dimensional coordinates and angles, for example, as illustrated in FIG. And a specifying unit 150 for specifying one predetermined three-dimensional coordinate and angle.

  With this configuration, one three-dimensional coordinate can be selected from the plurality of three-dimensional coordinates inferred by the inference unit 140, and a more accurate gripping operation can be performed.

  The machine learning method according to the twelfth aspect of the present invention uses, for example, a computer as shown in FIG. 9: a robot hand R holding one of a plurality of works W arranged in a predetermined area VWA. The three-dimensional coordinate data and the angle of the robot hand R when gripping succeeds through the gripping operation, and the robot hand R is arranged in the predetermined area VWA before the gripping operation when the robot hand R successfully grips. Storing a plurality of learning data sets each including two-dimensional captured image data obtained by capturing one or a plurality of works W from a predetermined angle of view using a two-dimensional imaging device ID; and inputting a plurality of the learning data sets. Learning a learning model for inferring the three-dimensional coordinates and the angles from the two-dimensional captured image; and learning the learning model. Including; a step S56 of storing the LE.

  With this configuration, the learning (existing) model obtained by this machine learning method is applied to data processing in bulk picking, thereby eliminating the need for a complicated device such as a three-dimensional measuring device and performing bulk picking work. Automation can be realized. Further, since the learning (completed) model outputs not only the three-dimensional coordinate data but also the angle data, it is possible to always hold the workpiece with high accuracy regardless of the shape of the robot hand.

  According to the present invention, since a learned model obtained by machine learning has only two-dimensional captured image data as a state variable associated with (input to) its input layer, a complicated device such as a three-dimensional measuring device is used. , And automation of bulk picking work can be realized with a simple configuration. Thereby, the cost when applying the present invention can be suppressed. Further, if a simulator is used for machine learning according to the present invention, collection of a learning data set for machine learning can be realized in a short period of time and stably. Furthermore, if the data output from the trained model generated through machine learning includes angle data in addition to the three-dimensional coordinate data, the workpiece can be always grasped with high accuracy regardless of the shape of the robot hand. A possible trained model can be provided. Furthermore, at the time of machine learning, if a data set is created by associating a plurality of three-dimensional coordinate data (and angle data) with one two-dimensional captured image data, the three-dimensional coordinates at which the success of the gripping operation is most expected can be obtained. This makes it possible to select, and realize data processing with high accuracy and high degree of freedom.

FIG. 1 is a schematic diagram illustrating a machine learning system according to a first embodiment of the present invention. FIG. 2 is a flowchart showing a simulated calculation process of the simulator according to the first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a neural network model for supervised learning performed in the machine learning device according to the first embodiment of the present invention. FIG. 4 is a flowchart illustrating the machine learning method according to the first embodiment of the present invention. FIG. 5 is a schematic diagram showing an actual work area to which the data processing system according to the first embodiment of the present invention is applied. FIG. 6 is a flowchart illustrating the bulk picking process in the actual work area according to the first embodiment of the present invention. FIG. 7 is a schematic diagram showing a machine learning system according to the second embodiment of the present invention. FIG. 8 is a flowchart showing a simulated calculation process of the simulator according to the second embodiment of the present invention. FIG. 9 is a flowchart illustrating a machine learning method according to the second embodiment of the present invention. FIG. 10 is a schematic diagram illustrating a machine learning system according to a third embodiment of the present invention. FIG. 11 is a schematic diagram illustrating an example of two-dimensional captured image data generated by the simulator according to the third embodiment of the present invention. FIG. 12 is a flowchart showing a simulated calculation process of the simulator according to the third embodiment of the present invention. FIG. 13 is a schematic diagram showing an actual work area to which the data processing system according to the third embodiment of the present invention is applied. FIG. 14 is a flowchart showing a bulk picking process in an actual work area according to the third embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following, the range necessary for the description for achieving the object of the present invention is schematically shown, and the range necessary for the description of the relevant part of the present invention will be mainly described. It shall be based on a known technique.

  First, a so-called bulk picking as a learning target according to the embodiment of the present invention will be briefly described. For example, as illustrated in FIG. 5, the separated picking includes a plurality of works W randomly stacked in a box-shaped tray TR (separated stacking), and this information is obtained from the two-dimensional imaging device ID. Is carried and carried one by one (or a plurality in some cases) by the robot hand R based on. At the time of this bulk picking, the position of which work W is gripped by the robot hand R is a very important factor for a successful picking operation. Therefore, in order to identify the position, various methods such as using a three-dimensional measuring device described in Patent Document 1 have been studied.

  The present inventors have studied various methods for performing machine learning without newly requiring complicated and expensive equipment in realizing the bulk picking without requiring any manpower, and as a result, the structure is simple and inexpensive. The present invention has conceived of machine learning using two-dimensional captured image data that can be acquired by a simple two-dimensional imaging device, and has led to the present invention.

  By the way, for example, the machine learning device described in Patent Literature 1 actually performs a work arrangement and a robot operation to generate a learning data set for machine learning (hereinafter, also simply referred to as a “data set”). Collecting. However, in this method, a series of preparations is required each time one data set is obtained, and the operation time of the robot arm R is also required, so that a relatively large amount of time is required. Since a plurality of data sets need to be prepared, as a result, the time required for preparation for executing sufficient machine learning tends to be long. In the present invention, as in the case of the existing machine learning device, a learning model capable of inferring with high accuracy is generally obtained in proportion to the amount of the learning data set. In order to obtain a model, it is preferable that the number of training data sets be as large as possible. Therefore, in the first embodiment of the present invention, as a means for solving this problem, a configuration using a simulator for machine learning is further adopted. However, even if a plurality of data sets are obtained by actually arranging the work and operating the robot hand without using the simulator used in the embodiment of the present invention described below, the main feature of the present invention is as follows. It can be easily understood by those skilled in the art that it is possible to solve the problem (realizing the automation of the gripping / transporting operation of the work by the robot hand with a simple configuration). Therefore, the use of a simulator is optional in the present invention.

<Machine learning system>
FIG. 1 is a schematic diagram illustrating a machine learning system according to a first embodiment of the present invention. The machine learning system 1 includes a simulator 2 and a machine learning device 3. The simulator 2 and the machine learning device 3 may be built in separate computers or the like, or may be built in different computers or the like.

<Simulator>
The simulator 2 is a device for obtaining various types of information when performing bulk picking under predetermined conditions specified in advance by a user or the like. The simulator 2 includes an information processing unit 21, a condition storage unit 22, a two-dimensional captured image data acquisition unit 23, a three-dimensional coordinate data acquisition unit 24, a data set generation unit 25, and a transmission unit 26.

  The information processing unit 21 generates a three-dimensional virtual work area VWA based on various conditions stored in a condition storage unit 22, which will be described later, and generates a work W and a work W formed in the virtual work area VWA. By operating a three-dimensional graphic model of the robot hand R or the like inside, a simulation calculation of picking in bulk is executed. The setting condition of the virtual work area VWA is that the virtual work area VWA is a work environment in which a learned model generated by the machine learning device 3 described later is actually applied, for example, an actual work as shown in FIG. The area is set to be the same as the area RWA. It should be noted that a well-known method implemented in the technical field of a simulator can be used for a specific calculation method itself, and thus detailed description is omitted here. FIG. 1 shows a virtual work area VWA virtualized in the information processing unit 21 for the purpose of facilitating understanding. Such a three-dimensional model of the virtual work area VWA can be made visible to the user via display means (not shown) as necessary. Note that the specific method of the simulation is not limited to the above method, and it is also possible to adopt another method known in the technical field of the simulator.

  The condition storage unit 22 is a storage area for storing various conditions for constructing the virtual work area VWA, for example, information such as the structure of the robot hand R, the shape of the work W, and the angle of view of the two-dimensional imaging device ID. is there. Various conditions stored in the condition storage unit 22 are arbitrarily set by a user (operator) via input means (not shown) or the like.

  The two-dimensional captured image data acquisition unit 23 acquires two-dimensional captured image data captured by the three-dimensional model of the two-dimensional imaging device ID installed in the virtual work area VWA. Here, the arrangement and the angle of view of the three-dimensional model of the two-dimensional imaging device ID are determined based on the conditions stored in the condition storage unit 22. For example, as shown in FIG. Above the region VWA, the entire inside of the tray TR on which the plurality of works W are stacked in a bulk is supported at an angle of view that allows the tray TR to be visually recognized without leakage. In the present invention, the data acquired by (the three-dimensional model of) the two-dimensional imaging device ID is not three-dimensional data or the like using a three-dimensional measuring device as described in Patent Document 1, but has a simple structure. Is one of the characteristic configurations in that it is two-dimensional captured image data that can be acquired by a relatively inexpensive camera or the like. By using only one two-dimensional imaging device ID composed of a relatively inexpensive camera or the like, bulk picking can be realized with a simple configuration, and such a two-dimensional imaging device ID can be used. If there is, it is possible to divert these because it is often provided in the existing work area, and as a result, in most cases, when trying to introduce the technology of the present invention into the existing work area, Does not involve a separate capital investment.

  The three-dimensional coordinate data acquisition unit 24 determines that the three-dimensional model of the robot hand R in the virtual work area VWA has successfully grasped one (or a plurality of) works W in the virtual work area VWA. 3D coordinate data representing the gripping position of the robot hand R of FIG. Here, the three-dimensional coordinate data indicates (x, y, z) coordinates when the front-back direction of the robot hand R is x, the left-right direction is y, and the up-down direction is z, as shown in FIG. Things. It should be noted that any reference coordinate system may be used, and a base coordinate system, a world coordinate system, a camera coordinate system, or the like may be employed.

  The data set generation unit 25 outputs the two-dimensional captured image data and the three-dimensional coordinate data acquired by the two-dimensional captured image data acquisition unit 23 and the three-dimensional coordinate data acquisition unit 24 as a common simulation calculation result. This is for associating each other as one data set. The data set generation unit 25 may perform not only generation of a data set but also temporary storage of the generated data set.

  The transmission unit 26 is for transmitting the data set generated by the data set generation unit 25 to a data set acquisition unit 31 of the machine learning device 3 described below. The specific method of this transmission can be appropriately adjusted according to the connection state between the simulator 2 and the machine learning device 3. The transmission timing can also be set as appropriate, such as at any time or when the number of generated data sets reaches a predetermined number.

  Next, with reference to FIG. 2, a series of simulated calculation steps performed by the simulator 2 will be described below. FIG. 2 is a flowchart showing a simulated calculation process of the simulator 2 according to the first embodiment of the present invention. As shown in FIG. 2, the simulator 2 according to the first embodiment of the present invention executes the following steps.

  When the simulator 2 is activated and the operation of the simulated calculation starts, first, based on the conditions stored in the condition storage unit 22, a virtual work area including a three-dimensional graphic model or the like of the robot hand R is processed by the information processing unit 21. Generate VWA. Then, two-dimensional captured image data captured from the two-dimensional imaging device ID installed at a predetermined position and angle of view in the virtual work area VWA is generated (step S11). The two-dimensional captured image data generated here is obtained (temporarily) stored in the captured data obtaining unit 23. When the two-dimensional captured image data is generated, next, the three-dimensional model of the robot hand R is operated on the workpiece W in the captured tray TR to perform a simulation operation related to the gripping operation of the workpiece W. Execute (step S12). It should be particularly noted that various methods adopted in the technical field of bulk picking can be applied to the simulation operation performed here. More specifically, for example, a three-dimensional measuring device as described in Patent Document 1 is arranged in a virtual working area VWA, and three-dimensional data of the three-dimensional measuring device is used, or input means (not shown) is used. Partial use of teaching by the operator or use of various sensors is permitted. In short, what is important in the simulation calculation process by the simulator 2 is to acquire various data when the workpieces W stacked in the imaged tray TR are successfully grasped. Not limited.

  If the gripping operation fails as a result of the gripping operation described above (No in step S13), the two-dimensional captured image data is deleted (step S14), and two-dimensional captured image data is generated again (step S11). . At the same timing as when the two-dimensional captured image data is erased in step S14, the arrangement of the stacked works W can be changed at random. Thereby, data for the workpieces W in various arrangements can be obtained. In this embodiment, each time the gripping fails, the latest two-dimensional captured image data is deleted and the two-dimensional captured image data is regenerated. However, the two-dimensional captured image data is deleted and regenerated. The simulation calculation (step S12) may be repeatedly performed without performing (that is, without performing the processing shown in step S14 and the subsequent step S11). That is, when the gripping operation has failed, the gripping operation is continuously performed after the arrangement of the work W in the virtual work area VWA is returned to the arrangement of the work W captured by the two-dimensional captured image data generated most recently. You may.

  As a result of the gripping operation described above, when gripping is successful (Yes in step S13), next, three-dimensional coordinate data of the robot hand R at the time of gripping is acquired (step S15). The specific part of the robot hand R for acquiring the three-dimensional coordinate data can be arbitrarily determined. For example, the center of gravity position P (see FIG. 5) of the end effector 54 (see FIG. 5) of the robot hand R can be determined. ) Can be obtained as the three-dimensional coordinate data.

  Next, the data set generation unit 25 converts the three-dimensional coordinate data acquired in step S15 and the latest two-dimensional captured image data acquired and stored in the two-dimensional captured image data acquisition unit 23 in step S11 into one It is specified as a data set (step S16). Thereby, the three-dimensional coordinate data of the robot hand R when the gripping operation of the workpiece W is successful and the two-dimensional captured image data indicating the state of the workpiece W before the start of the gripping operation when the gripping operation is successful are one. The data sets are generated in a one-to-one relationship. Then, the specified data set is transmitted to the machine learning device 3 by the transmission unit 26 (Step S17).

  The above-described series of simulation calculation steps is repeatedly executed until the amount of data sets required in the machine learning device 3 described later. However, since a series of simulation operation steps are executed in the simulator 2, a desired number of data sets can be obtained in an extremely short time as compared with a case where the robot hand R is actually operated to collect data sets. Collection is possible. Since the three-dimensional model of the robot hand R and the like in the simulator 2 does not deteriorate over time, data collection can be performed in a stable environment.

<Machine learning device>
Next, a machine learning device 3 that performs machine learning using a data set generated by the simulator 2 will be described below. The machine learning device 3 includes a data set acquisition unit 31, a data set storage unit 32, a learning unit 33, and a learned model storage unit 34, as shown in FIG.

  The data set acquisition unit 31 acquires a data set transmitted from the transmission unit 26 of the simulator 2. The connection relationship between the transmission unit 26 and the data set acquisition unit 31 can be changed as appropriate, and may be connected locally via wired or wireless communication or via the Internet. Data transmission may be performed via the Internet.

  The data set storage unit 32 is a storage area for storing the data sets acquired by the data set acquisition unit 31.

  The learning section 33 executes machine learning using a plurality of data sets stored in the data set storage section 32 as teacher data. The learned model storage unit 34 is a storage area for storing the learned model generated by the learning unit 33. The machine learning executed by the learning unit 33 will be described below.

  FIG. 3 is a diagram illustrating an example of a neural network model for supervised learning performed in the machine learning device according to the first embodiment of the present invention. The neural network in the neural network model shown in FIG. 3 includes l neurons (x1 to xl) in an input layer, m neurons (y11 to y1m) in a first intermediate layer, and n neurons in a second intermediate layer. , And three neurons (z1 to z3) in the output layer. The first intermediate layer and the second intermediate layer are also called hidden layers, and the neural network may have a plurality of hidden layers in addition to the first intermediate layer and the second intermediate layer. Alternatively, only the first intermediate layer may be used as the hidden layer.

  Nodes connecting neurons between the layers are provided between the input layer and the first intermediate layer, between the first intermediate layer and the second intermediate layer, and between the second intermediate layer and the output layer. Each node is associated with a weight wi (i is a natural number).

  The neural network in the neural network model according to the present embodiment learns the correlation between two-dimensional captured image data and three-dimensional coordinate data using a plurality of data sets stored in the data set storage unit 32. . Specifically, the two-dimensional captured image data is divided into a plurality of data, and the plurality of divided data are used as state variables, each state variable is associated with a neuron in an input layer, and a value of a neuron in an output layer is generally used. Method of calculating the output value of a typical neural network, that is, the value of the output side neuron is connected to the value of the input side neuron connected to the neuron and the node connecting the output side neuron and the input side neuron. The calculation as the sum of the sequence of the multiplied values with the associated weights wi is performed by using a method for all the neurons other than the neurons in the input layer. When associating the state variables with the neurons in the input layer, the form in which the information acquired as the state variables is associated can be appropriately set in consideration of the accuracy of the generated learned model and the like. For example, when associating the two-dimensional captured image data with the input layer as a state variable, it is possible to divide the image data in bit units and associate the color value (eg, RGB value) information of each divided bit with the input layer. it can.

  Then, from the calculated values of the three neurons z1 to z3 in the output layer, that is, in this embodiment, the three-dimensional coordinate data of the robot hand R and the three-dimensional coordinate data of the robot hand R in the data set. The teacher data t1 to t3 are compared with each other to determine an error, and the adjustment of the weight wi associated with each node (back pro vacation) is repeated so that the obtained error is reduced.

  Then, when a predetermined condition such as repeating the series of steps described above a predetermined number of times or that the error is smaller than an allowable value is satisfied, the learning is terminated and the neural network model (of the All weights wi) associated with each of the nodes are stored in the learned model storage unit 34 as a learned model.

  The learned model stored in the learned model storage unit 34 is applied to an actual system via a communication means such as the Internet or a storage medium as required. A specific application mode of the learned model to an actual system (data processing system) will be described later in detail.

<Machine learning method>
In relation to the machine learning device 3 described above, the present invention also provides a machine learning method. FIG. 4 is a flowchart illustrating the machine learning method according to the first embodiment of the present invention. This machine learning method is realized by using a computer, but various types of computers such as a personal computer and a server device can be applied.

  When performing supervised learning as a machine learning method according to the present invention, first, a pre-learning model having an arbitrary initial value weight is prepared (step S21). Next, one of the plurality of data sets stored in the data set storage unit 32 is acquired (step S22), and the two-dimensional captured image data of the acquired data sets is input to the input layers x1 to x1 of the pre-learning model. xl (see FIG. 3) and output layers z1 to z3 (see FIG. 3) are generated (step S23).

  Here, since the three-dimensional coordinate data constituting the output layer generated in step S23 is generated by the pre-learning model, the three-dimensional coordinate data almost always satisfies the user's request, that is, can grasp the work W. Not coordinate data. Therefore, next, machine learning is performed using the three-dimensional coordinate data of the one data set acquired in step S22 and the three-dimensional coordinate data constituting the output layer generated in step S23 (step S23). S24). The machine learning performed here means comparing the three-dimensional coordinate data of the data set with the three-dimensional coordinate data constituting the output layer, detecting an error between the two, and obtaining an output layer in which the error is reduced. For example, machine learning (back pro vacation) is performed by adjusting weights associated with each node in the pre-learning model.

  When machine learning is performed in step S24, it is determined whether or not it is necessary to continue machine learning (step S25). If machine learning is to be continued (No in step S25), the process returns to step S22. When ending the machine learning (Yes in step S25), the process proceeds to step S26. If machine learning is to be continued (No in step S25), the above-described steps S22 to S24 are performed a plurality of times, for example, the same number of times as the number of all data sets stored in the data set storage unit 32. Becomes Usually, the accuracy of the finally generated learned model increases in proportion to the number of times.

  If the machine learning is to be ended (Yes in step S25), the neural network generated by adjusting the weight associated with each node of the learning model through a series of steps is stored in the learned model storage unit 34 as a learned model. This is memorized (step S26), and the series of learning processes ends. The learned model stored here is applied to and used in various data processing systems described later.

  In the learning process and the machine learning method of the machine learning device described above, in order to generate one learned model, a plurality of machine learning processes are repeatedly performed on one neural network (model before learning). Although a learning model is learned and a learned model whose accuracy is improved to an extent applicable to the data processing system is generated, the present invention is not limited to such an acquisition method. For example, a plurality of trained models that have undergone machine learning for a predetermined number of times are stored as one candidate in the trained model storage unit 34, and a data set for validity determination is input to the plurality of trained model groups. Generate (in association with) the output layer (the value of the neuron), compare the three-dimensional coordinate data specified in the output layer with the three-dimensional coordinate data in the data set, and apply the data processing system. The best learned model to be performed may be selected. Note that the data set for determining validity used here may have two-dimensional captured image data and three-dimensional coordinate data and be different data, similarly to the data set used for learning.

  As described above, the learned model generated by the learning process and the machine learning method of the machine learning device 3 described above includes two-dimensional captured image data captured by the two-dimensional imaging device ID as can be understood from a series of steps. When input, one three-dimensional coordinate data of the robot hand R can be output. In other words, when providing a data processing system capable of realizing bulk picking without requiring any human hands using this trained model, the input data to the input layer may be only two-dimensional captured image data. Therefore, in the data processing system using the trained model, since a complicated device such as a three-dimensional measuring device is not required, bulk picking can be realized with a simple configuration, and the technology can be introduced at low cost. it can. Further, by using the simulator 2 to generate a data set used for machine learning, a large amount of data set can be collected in a short period of time, and a desired learned model can be generated in a short time.

  Although the state variables associated with the input layer of the trained model are described as only the two-dimensional captured image data, the form in which the two-dimensional captured image data is associated with the input layer can be appropriately adjusted. For example, it is possible to appropriately adjust the original data of the two-dimensional captured image data captured by the two-dimensional imaging device ID by executing a predetermined preprocessing in order to associate the data with the input layer. The state variable in the machine learning device of the present invention is an important factor that directly affects the generated learned model. However, the state variable in the machine learning device or the like of the present invention is changed only by the above-described two-dimensional captured image data. And is not intended to completely exclude the use of other data. For example, a machine learning apparatus or the like in which data that has a sufficiently small effect on the generated learned model compared to the data used in the above-described embodiment is added as a state variable is substantially equivalent to the technology of the present invention. Since it does not deviate from the idea, it can be said that it is included in the technical scope of the present invention.

  Further, it is preferable that the machine learning method according to the present embodiment be sequentially executed in accordance with a change in an actual work area to which a learned model obtained by the machine learning method is applied. The gripping success rate of bulk picking largely depends on the shape of the work, the angle of view of the two-dimensional imaging device, the function of the robot hand, etc. For example, even if only the shape of the work is changed, the learned model before and after the change If not changed, the success rate of grasping often decreases significantly. Also in this case, since the machine learning system according to the first embodiment of the present invention uses a simulator, it is only necessary to operate the simulator and the machine learning device again when generating a new learned model. Therefore, the user can execute the generation of the learned model by machine learning without worrying about the time and cost for obtaining a new learned model, and can always obtain the optimal learned model in a short time and at low cost. It can be used.

<Data processing system>
Next, a data processing system according to the first embodiment of the present invention will be described. This data processing system uses a machine learning system, a machine learning device or a learned model generated by a machine learning method described above to realize bulk picking in a real work area without human intervention. Is composed. The data processing system is specifically applied to a robot controller 100 for controlling a robot hand R and the like as shown in FIG.

  FIG. 5 is a schematic diagram showing an actual work area to which the data processing system according to the first embodiment of the present invention is applied. As shown in FIG. 5, it is particularly noted that the structures and arrangements of various components in the actual work area RWA are the same as the structures and arrangements defined in the virtual work area VWA generated in the simulator described above. Should. The fact that these are the same means that when performing machine learning using the machine learning system 1, information about the actual work area RWA is collected, this information is acquired as a condition in the simulator 2, and the information is stored in the condition storage unit 22. This is caused by storing and referencing when generating the virtual work area VWA. In this way, by making the real work area RWA and the virtual work area VWA the same, the learned model generated by performing the machine learning based on the virtual work area RWA can be used in the real work area RWA. It can operate very effectively.

  In the actual work area RWA, a tray TR, a plurality of works W, a two-dimensional imaging device ID, a robot hand R, and a robot controller 100 are mainly arranged.

  The tray TR has a box shape, accommodates a plurality of works W therein, and allows access to the works W from an opening formed at an upper portion. The shape of the tray TR is not limited at all. For example, a shape in which the peripheral wall has a tapered shape spreading upward and a shape in which the peripheral wall is sufficiently lower than the work W can be adopted.

  The plurality of works W have a predetermined shape and are randomly arranged in the tray TR. Since various mechanical parts and the like can be applied to the work W, various shapes can be assumed for the shape. FIG. 5 illustrates the work W having a rectangular parallelepiped shape.

  The two-dimensional imaging device ID is an imaging device that can be acquired as two-dimensional imaging image data. For example, the two-dimensional imaging device ID is connected to a network by one WEB camera that can transmit two-dimensional imaging image data to the network. It is realized. The two-dimensional imaging device ID is fixed at a position obliquely above the tray TR and is set to an angle of view that allows the entire inside of the tray TR to be imaged without leakage.

  The robot hand R is disposed near the tray TR, and holds the work W in the tray TR. In the present embodiment, a vertical articulated robot will be described as an example of the robot hand R, but the type of the robot is not limited at all. For example, any robot applicable to picking in bulk, such as a horizontal articulated robot, a parallel link robot, a rectangular coordinate robot, or a cylindrical coordinate robot, can be adopted as appropriate.

The robot hand R includes a base 51, a lower link arm 52, an upper link arm 53, an end effector 54, and a plurality of joints 55 to 57. The robot hand R composed of these components has six axes (the rotation θ ₁ of the base portion 51, the rotations θ _{2 to} θ ₄ of the joints 55 to 57, the rotation θ ₅ of the upper link arm, and the rotation θ of the end effector 54). ₆ ). Further, for example, a gripper can be applied to the end effector 54, but the end effector 54 can be appropriately changed in consideration of the shape of the work W, and for example, a multi-finger hand, a suction type, or the like may be used. Is also possible. In addition, as for the detailed configuration of the robot hand R, since the configuration of a well-known vertical articulated robot can be adopted, the description is omitted here.

  The robot controller 100 is connected to and controls the robot hand R and the two-dimensional imaging device ID. The robot controller 100 includes a two-dimensional imaging device control unit 110, a robot hand control unit 120, a main storage unit 130, and an inference unit 140.

  The two-dimensional imaging device control unit 110 has a configuration corresponding to the acquisition unit in the data processing system of the present invention, and controls the two-dimensional imaging device ID to acquire two-dimensional imaging image data at a desired timing. It is. The two-dimensional captured image data acquired by the two-dimensional imaging device control unit 110 is sent to the inference unit 120 described below, and is associated with the input layer of the learned model.

  The robot hand control unit 120 controls the robot hand R to grip the work W by operating each axis of the robot hand R based on three-dimensional coordinate data specified by the inference unit 140 described below. Things.

  The main storage unit 130 is a storage area for storing various data for realizing the picking work of the work W in the actual work area RWA, and includes at least a learned model storage unit 131 therein. ing. The trained model storage unit 131 is a storage area for storing a trained model generated through the machine learning system, the machine learning device, and the machine learning method according to the above-described first embodiment of the present invention. It is preferable that the learned model storage unit 131 stores a plurality of learned models sequentially created in accordance with a change in the environment of the actual work area RWA. In this case, an inference unit 140 described later is used. In, an appropriate model can be selected and used from the plurality of learned models according to the actual situation of the work area RWA.

  The inference unit 140 uses the two-dimensional captured image data acquired by the two-dimensional imaging device control unit 110 and one learned model in the learned model storage unit 131 to obtain desired three-dimensional coordinates of the robot hand R. Infer data. In detail, the learned model according to the situation of the actual work area RWA is referred to from the learned model storage unit 131, and the two-dimensional captured image data acquired by the two-dimensional imaging device control unit 110 is referred to by the reference 1 The three-dimensional coordinate data of the robot hand R capable of gripping the workpiece W is output to the output layer by associating the three hands with the input layer of the learned models.

  A series of steps when picking the work W by controlling the robot controller 100 having the above configuration will be described below with reference to FIGS. FIG. 6 is a flowchart illustrating the bulk picking process in the actual work area according to the first embodiment of the present invention.

  In the actual work area RWA shown in FIG. 5, when the bulk picking operation is started, first, the imaging device ID is operated by the two-dimensional imaging device control unit 110, and two-dimensional image data obtained by imaging the inside of the tray TR. Is obtained (step S31). Next, the inference unit 140 uses the learned model storage unit 131 based on the input information from the operator from the input unit (not shown), the two-dimensional imaging device ID stored in the main storage unit 130, and the function information on the robot hand R. One learned model is specified from one or a plurality of learned models stored in (step S32).

  When the learned model is specified, the inference unit 140 refers to the specified learned model, and in the input layer of the learned model, the two-dimensional captured image acquired by the two-dimensional imaging device control unit 110 in step S31. The data is correlated, inference (data processing) using the learned model is executed, and three-dimensional coordinate data of one of the robot hands R capable of holding the work W is output as an output layer (step S33). The output three-dimensional coordinate data is sent to the robot hand control unit 120, and the robot hand control unit 120 places a predetermined part of the robot hand R (for example, the center of gravity P of the end effector 54) at the position indicated by the three-dimensional coordinate data. The six axes of the robot hand are driven to try to grip the workpiece W (step S34). It should be noted that since what kind of operation (transportation, etc.) is performed after gripping the work W is relatively simple, if it is appropriately set in the robot hand control unit 120, machine learning or the like can be performed. It can be realized without requiring. Therefore, in the present invention, the description of the work after gripping the work W is omitted.

  As described above, in the bulk picking to which the data processing system according to the first embodiment of the present invention is applied, the work W is performed only from the image data captured by the two-dimensional imaging device ID including a camera having a simple structure. Since it is possible to infer the three-dimensional coordinate data of the robot hand R capable of holding the object, it is possible to realize the bulk picking that does not require human labor with a simple configuration. In the first embodiment, the result of the gripping attempt in step S34 is recognized, and the result, the two-dimensional captured image data used by the inference unit 140 at that time, and the three-dimensional coordinate data output by the inference unit 140 are recognized. Is more preferably used as a learning data set. In this case, the above-described machine learning device 3 is provided inside or outside the robot controller 100, and the machine learning device 3 executes machine learning using the learning data set. Data in the learned model storage unit 131 may be updated. In this way, not only the gripping result using the virtual working area VWA but also the gripping result in the real working area RWA can be used for learning, and the accuracy of the learned model can be further improved. .

<Second embodiment>
In the first embodiment described above, the output layer of the learned model outputs three-dimensional coordinate data. However, depending on the shape of the robot hand R (particularly, the shape of the end effector 54), the gripping of the workpiece W may fail with only the three-dimensional coordinate data. That is, for example, the robot hand R is a robot having a high degree of freedom, such as the vertical articulated robot illustrated in FIG. 5, and the length of the end effector 54 in the horizontal direction or the vertical direction (side of the upper link arm 52). When the robot hand R moves to the three-dimensional coordinate data output by the trained model, for example, when the robot hand R has a long external shape (compared to the vertical or vertical direction), the long end effector 54 May be in contact with the work W, the arrangement of the work W may be changed, and as a result, a situation in which the grip of the work W fails may occur.

  In view of the above, in order to realize the gripping of the workpiece regardless of the shape of the end effector, the following describes, as a second embodiment of the present invention, three-dimensional coordinate data as data output by the output layer. A machine learning system, a machine learning device, a machine learning method, and a data processing system capable of outputting angle data of the robot hand R in addition to the above will be described. FIG. 7 is a schematic diagram showing a machine learning system according to the second embodiment of the present invention. Note that a machine learning system, a machine learning device, a machine learning method, and a data processing system according to a second embodiment described below will be described focusing on the differences from the first embodiment described above. The same reference numerals are given to configurations and the like that are common to the first embodiment, and description thereof will be omitted.

  As shown in FIG. 7, the machine learning system 1A according to the second embodiment of the present application includes a simulator 2A and a machine learning device 3A. The simulator 2A further includes an angle data acquisition unit 27 in addition to the series of components included in the simulator 2 according to the first embodiment.

The angle data acquisition unit 27 determines the robot when the three-dimensional model of the robot hand R in the virtual work area VWA has successfully gripped one (or a plurality of works W) of the works W in the virtual work area VWA as well. This is for acquiring angle (posture) data of a hand portion (a portion formed by the end effector 54 and the upper link arm 52 in some cases) of the hand R. Here, the angle data is the angle RX formed by the hand of the robot hand R with respect to the x-axis shown in FIG. 7, similarly, the angle RY formed by the hand of the robot hand R with respect to the y-axis, and the robot hand R with respect to the z-axis. (RX, RY, RZ) when the angle RZ formed by the hand portion is In the present embodiment, the angle data is described as being composed of (RX, RY, RZ), but instead of such three-dimensional angle data, two-dimensional (for example, only RX and RY) or It is also possible to adopt one-dimensional (for example, RX only) angle data. By reducing the number of numerical values defined as the angle data, it is possible to reduce the number of data sets necessary to obtain a learned model having sufficient accuracy in machine learning described later. Further, in the present embodiment, the angle data is defined as (RX, RY, RZ), but the data is specified by the angle formed by each axis (θ _{1 to} θ ₆ ) of the robot. Instead of the angle data represented by (RX, RY, RZ), an angle formed by a predetermined number of axes of the robot hand R can be used.

  Next, with reference to FIG. 8, a series of simulated calculation steps by the simulator 2A will be described below. FIG. 8 is a flowchart showing a simulated calculation process of the simulator 2A according to the second embodiment of the present invention. As shown in FIG. 8, the simulator 2A according to the second embodiment of the present invention executes the following steps.

  When the simulator 2A is activated and the operation of the simulated operation starts, first, based on the conditions stored in the condition storage unit 22, the information processing unit 21 generates a virtual work area including a three-dimensional graphic model of the robot hand R and the like. Generate VWA. Then, two-dimensional captured image data captured from the two-dimensional imaging device ID installed at a predetermined position and angle of view in the virtual work area VWA is generated (step S41). The two-dimensional captured image data generated here is obtained (temporarily) stored in the captured data obtaining unit 23. When the two-dimensional captured image data is generated, next, the three-dimensional model of the robot hand R is operated on the workpiece W in the captured tray TR to perform a simulation operation related to the gripping operation of the workpiece W. Execute (Step S42).

  As a result of the gripping operation described above, when gripping has failed (No in step S43), the two-dimensional captured image data generated most recently is deleted (step S44), and two-dimensional captured image data is generated again in step S41. Will be reworked. Further, as a result of the gripping operation described above, when gripping is successful (Yes in step S43), next, three-dimensional coordinate data and angle data of the robot hand R at the time of gripping are acquired (steps S45 and S46). ). Note that the execution timing of step S45 and step S46 may be either earlier or simultaneously. Further, the specific part of the robot hand R that acquires the three-dimensional coordinate data and the angle data may be an arbitrary position of the end effector 54 (for example, the center of gravity P of the end effector 54 shown in FIG. 5). Further, as in the first embodiment, step S44 and the subsequent step S41 can be omitted.

  When the three-dimensional coordinate data and the angle data of the robot hand R at the time when the workpiece W is gripped are acquired, the data set generation unit 25 outputs the three-dimensional coordinate data acquired in step S45, the angle data acquired in step S46, and step S41. Then, the latest two-dimensional captured image data acquired and stored by the two-dimensional captured image data acquisition unit 23 is specified as one data set (step S47). Thereby, the three-dimensional coordinate data and the angle data of the robot hand R when the gripping operation of the workpiece W is successful, and the two-dimensional captured image data indicating the state of the workpiece W before the start of the gripping operation when the gripping operation is successful. Are associated in a one-to-one relationship, and a data set is generated. Then, the specified data set is transmitted to the machine learning device 3A by the transmission unit 26 (Step S48).

  Next, a machine learning device 3A that performs machine learning using a data set generated by the simulator 2A will be described below. As can be seen from FIG. 7, the configuration of the machine learning device 3A is the same as that of the machine learning device 3 according to the above-described first embodiment, and only the content of the machine learning and the content of the data related thereto are different. This is different from the first embodiment.

  FIG. 9 is a flowchart illustrating a machine learning method according to the second embodiment of the present invention. The machine learning method shown here is realized by using a computer, and is also executed by the machine learning device 3A according to the above-described second embodiment.

  In the machine learning method according to the second embodiment, as shown in FIG. 9, first, a pre-learning model having an arbitrary initial value weight is prepared (step S51). Next, one of the plurality of data sets stored in the data set storage unit 32 is obtained (step S52), and the two-dimensional captured image data of the obtained data set corresponds to the input layer of the pre-learning model. Then, an output layer is generated (step S53).

  Next, machine learning is performed using the three-dimensional coordinate data and the angle data of the one data set acquired in step S52 and the three-dimensional coordinate data and the angle data constituting the output layer generated in step S53. Perform (Step S54). When the machine learning is performed in step S54, it is determined whether or not it is necessary to continue the machine learning (step S55). When the machine learning is continued as a specific result (No in step S55), the above-described steps S52 to S54 are performed a plurality of times. If the machine learning is to be terminated (Yes in step S55), the neural network model generated by adjusting the weights associated with each node of the learning model through a series of steps is stored as a learned model as a learned model. This is stored in the unit 34 (step S56), and a series of learning processes ends.

  As described above, in the machine learning system, the machine learning device, and the machine learning method according to the second embodiment of the present invention, the machine learning system, the machine learning device, and the machine learning method according to the first embodiment are used. In addition to the effects of the above, the following effects are further exhibited. That is, these machine learning systems, machine learning devices, and machine learning methods can generate a learned model that can output angle data in addition to three-dimensional coordinate data. It is possible to provide a machine learning system, a machine learning device, and a machine learning method that can realize a highly accurate gripping operation regardless of the shape of the robot hand and can be applied to a wide variety of work areas.

  Lastly, a brief description will be given of the bulk picking realized by the machine learning system, the machine learning device, or the data processing system using the learned model generated by the machine learning method according to the above-described second embodiment. . The flow of a series of data processing of bulk picking by the data processing system according to the present embodiment is the same as the process shown in FIG. 6, but the data output by the inference unit 140 in step S33 in FIG. It differs from the data processing system according to the first embodiment in that not only dimensional coordinate data but also angle data is output. In addition to this, in the control process of the robot hand R shown in step S34 of FIG. 6, the angle data is also referred to in addition to the three-dimensional coordinate data, and the angle data is stored at the position indicated by the three-dimensional coordinate data. Are driven by driving the six axes of the robot hand R so that the workpiece W is positioned at the angle (posture) indicated by.

  As described above, in the bulk picking to which the data processing system according to the second embodiment of the present invention is applied, the workpiece W is gripped only from the image data captured by the two-dimensional imaging device ID having no complicated configuration. Since it is possible to infer the three-dimensional coordinate data and the angle data of the possible robot hand R, similar to the data processing system according to the above-described first embodiment, it is possible to easily perform the bulk picking without requiring any human labor. It can be realized using a configuration. In addition to this, by referring to the angle data of the robot hand R, the workpiece W can always be gripped with high accuracy regardless of the shape of the robot hand R, particularly the shape of the end effector 54.

<Third embodiment>
In the first and second embodiments, as a data set generated by the simulators 2 and 2A, three-dimensional coordinate data and two-dimensional captured image data are associated in a one-to-one relationship, or three-dimensional coordinate data. An explanation was given on the data, the angle data, and the two-dimensional captured image data associated with each other in a one-to-one relation. However, since bulk picking is based on the premise that a plurality of workpieces are arranged in a tray, some of the workpieces in the tray can be gripped, In some cases, there may be a plurality of points that can be gripped. Therefore, in many cases, there is not one piece of three-dimensional coordinate data but a plurality of pieces of three-dimensional coordinate data associated with one piece of two-dimensional captured image data. Therefore, hereinafter, as a third embodiment of the present invention, a data set is composed of two-dimensional captured image data and a plurality of three-dimensional coordinate data (or a set of a plurality of three-dimensional coordinate data and angle data). The generated machine learning system, machine learning device, machine learning method, and data processing system will be described.

  FIG. 10 is a schematic diagram illustrating a machine learning system according to a third embodiment of the present invention. Note that a machine learning system, a machine learning device, a machine learning method, and a data processing system according to a third embodiment described below will be described focusing on differences from the first embodiment described above. The same reference numerals are given to configurations and the like that are common to the first embodiment, and description thereof will be omitted.

  As shown in FIG. 10, the machine learning system 1B according to the third embodiment of the present application includes a simulator 2B and a machine learning device 3B. The simulator 2B further includes a two-dimensional captured image data dividing unit 28 in addition to the series of components included in the simulator 2 according to the first embodiment.

  The two-dimensional captured image data division unit 28 constitutes a part of the information processing unit 21 and divides the two-dimensional captured image data acquired by the two-dimensional captured image data acquisition unit 23 into a plurality of regions. . FIG. 11 is a schematic diagram illustrating an example of two-dimensional captured image data generated by the simulator according to the third embodiment of the present invention. Note that FIG. 11 shows an example in which only two works W are arranged in the two-dimensional captured image data for ease of explanation.

  With respect to the two-dimensional captured image data shown in FIG. 11, the two-dimensional captured image data dividing unit 28 partitions an area in the tray TR where the work W is arranged into a predetermined divided area. FIG. 11 illustrates an example in which a total of 16 divided areas DA1 to DA16 having a uniform size are divided and formed in a grid shape with 4 rows × 4 columns. Note that the size and number of the divided areas can be adjusted as appropriate.

  Next, with reference to FIG. 12, a series of simulated calculation steps by the simulator 2B will be described below. FIG. 12 is a flowchart showing a simulated calculation process of the simulator 2B according to the third embodiment of the present invention. As shown in FIG. 12, the simulator 2B according to the third embodiment of the present invention executes the following steps.

  When the simulator 2B is activated and the operation of the simulation operation is started, first, based on the conditions stored in the condition storage unit 22, the information processing unit 21 generates a virtual work area including a three-dimensional graphic model of the robot hand R. Generate VWA. Then, two-dimensional captured image data captured from the two-dimensional imaging device ID installed at a predetermined position and angle of view in the virtual work area VWA is generated (step S61). The two-dimensional captured image data generated here is obtained (temporarily) stored in the captured data obtaining unit 23. When the two-dimensional captured image data is generated, the generated two-dimensional captured image data is divided into predetermined regions by the two-dimensional captured image data dividing unit 28 as shown in FIG. DA1 to DA16 are formed (Step S62).

  Next, a three-dimensional model of the robot hand R is operated on the imaged work W in the tray TR to execute a simulation calculation relating to a gripping operation of the work W. In the present embodiment, The execution of the simulated calculation is performed only for one of the formed divided areas DA1 to DA16 (step S63). More specifically, when the divided area DA1 is selected as the one divided area, the gripping operation of the work W by the three-dimensional model of the robot hand R is performed only on the area within the divided area DA1. The gripping operation of the work W is not performed on an area other than the divided area DA1.

  The above-described gripping operation is performed a plurality of times using various methods. Then, when the grip is successful (Yes in step S64), the three-dimensional coordinate data of the robot hand R at the time of the grip is obtained (step S65). If the gripping has failed (No in step S64), it is determined that there is no grippable portion in the divided region subjected to the simulation calculation, and the three-dimensional coordinate data is not obtained. When the divided area DA1 is selected as one of the divided areas shown in the above example, as can be seen from FIG. 11, since the work W does not exist in the divided area DA1 in the first place, the gripping is not performed in step S64. It is determined that the operation has failed (No in step S64), and it is specified that there is no three-dimensional coordinate that can grip the work in the divided area DA1. Note that the gripping operation performed in the simulation calculation is performed a plurality of times, but the arrangement of the work W is always the same. Therefore, it is necessary to execute control to return the work W to the original arrangement in the simulator 2B every time the gripping operation is performed, but it is not necessary to regenerate the two-dimensional captured image data.

  When the above-described gripping operation is completed, it is specified whether or not the simulation operation can be ended (step S66). Specifically, it is determined whether or not all of the plurality of divided areas DA1 to DA16 divided in step S62 have been selected as the targets of the simulation calculation. As a result, if there is a divided area that is not the target of the simulation operation (No in step S66), the process proceeds to step S67, and if there is no divided area that is not the target of the simulation operation (step S66). If “Yes” in S66, the process moves to step S68. In step S67, one of the divided regions not subjected to the simulation operation is selected, and the selected divided region is specified as the target of the next simulation operation, and then the process returns to step S63.

  When all the simulation calculations for the plurality of divided regions are completed, in step S68, the data set generation unit 25 obtains the three-dimensional coordinate data obtained in step S65, and in step S61 obtains and stores the two-dimensional captured image data acquisition unit 23. The obtained two-dimensional captured image data is specified as one data set. Here, taking the two-dimensional captured image data shown in FIG. 11 as an example, among the plurality of divided areas DA1 to DA16, the work W exists in the divided areas DA6, DA7, and DA10. , DA11, and it is apparent that the gripping is not successful in the simulation calculation for the other divided areas. It is assumed that as a result of executing the gripping operation simulation operation on the four divided areas DA6, DA7, DA10, and DA11, three areas DA7, DA10, and DA11 have been successfully gripped. In this case, the three-dimensional coordinate data obtained through each of the steps S63 to S67 is three three-dimensional coordinate data obtained in each of the three areas DA7, DA10, and DA11. Therefore, the three three-dimensional coordinate data and the two-dimensional captured image data acquired and stored in step S61 are associated in the data set generation unit 25, and are specified as one data set. Then, the specified data set is transmitted to the machine learning device 3B by the transmission unit 26 (Step S69).

  The above-described series of simulation calculations is executed a plurality of times each time the arrangement of the work W is changed, specifically, up to the amount of data set required in the machine learning device 3B. According to the simulated calculation according to the above-described method, the maximum number of three-dimensional coordinate data (same as the number of divided areas) is 16 as three-dimensional coordinate data associated with one two-dimensional captured image data as a data set. Then, the number of three-dimensional coordinate data included in the data set transmitted to and stored in the machine learning device 3B can be variously within the range of 1 to 16.

  Next, a machine learning device 3B that performs machine learning using a data set generated by the simulator 2B and a machine learning method performed by the machine learning device 3B will be described below. First, the configuration of the machine learning device 3B is the same as that of the machine learning device 3 according to the above-described first embodiment, as can be seen from FIG.

  The basic steps of the machine learning method performed by the machine learning device 3B are the same as the steps shown in FIG. However, in the machine learning method according to the present embodiment, the number of three-dimensional coordinate data output by the learning model in the learning unit 33 is limited to one as in the case of the data sets stored in the data set storage unit 32. I can't. However, unless the number of three-dimensional coordinate data to be output is simply limited, there is a possibility that a large amount of three-dimensional coordinate data is output, particularly in a learning model at the beginning of learning. It is preferable to determine the maximum value of the data in advance. The maximum value is preferably the same as the number of divisions of the two-dimensional captured image data defined by the two-dimensional captured image data division unit 28 of the simulator 2B. For example, by analyzing the two-dimensional captured image data, Further, it can be specified by various methods such as limiting the numerical value, and the present invention is not limited to the specifying method.

  Further, when there are a plurality of three-dimensional coordinate data output by the learning model, the accuracy (probability of successful grasping) for each data is usually not uniform. Further, when the learned model generated by the machine learning device 3B is applied to the data processing system, when actually controlling the robot hand R, an optimal one out of the plurality of output three-dimensional coordinate data is used. It is necessary to select three-dimensional coordinate data. Therefore, in the learning model according to the present embodiment, when outputting the three-dimensional coordinate data, it is set so that the certainty of the three-dimensional coordinate data, that is, the reliability of the data is also output. . This credit is numerical data represented by, for example, a ratio (%) or 0 to 1. The trustworthiness is determined, for example, in addition to the coordinates and the number of the three-dimensional coordinate data as the teacher data in the data set including the two-dimensional image capturing data associated with the input layer, as well as the two-dimensional captured image data division. The estimation can be performed by performing inference using a trained model that has been learned by comprehensively considering the number of divisions in the unit 28 and information on the divided regions for which the three-dimensional coordinate data has not been acquired. It is specified based on the estimation result. Note that the method of specifying the detailed numerical value of the credit is arbitrary as long as the probability of successful grasping can be estimated, and thus the detailed description is omitted here.

  Next, a data processing system according to a third embodiment of the present invention will be described. This data processing system is for realizing bulk stacking picking in a real work area without using a human, using a machine learning system 1B, a machine learning device 3B, or a learned model generated by a machine learning method. Make up the system. The data processing system is specifically applied to a robot controller 100B for controlling a robot hand R and the like as shown in FIG.

  FIG. 13 is a schematic diagram showing an actual work area to which the data processing system according to the third embodiment of the present invention is applied. As shown in FIG. 13, the structure and arrangement of various components in the actual work area RWA according to the present embodiment are the same as those shown in FIG. The robot controller 100B further includes a specifying unit 150 in addition to a series of components included in the robot controller 100 according to the first embodiment.

  When there are a plurality of three-dimensional coordinate data output by the inference unit 140, the specifying unit 150 specifies one optimal three-dimensional coordinate data from the plurality of output three-dimensional coordinate data. Things. In this specification, the numerical value of the degree of trust associated with each output three-dimensional coordinate data is taken into consideration, and for example, the three-dimensional coordinate data having the highest numerical value of the degree of trust is specified as one optimal three-dimensional coordinate data.

  A series of steps when picking the workpiece W by controlling the robot controller 100B having the above configuration will be described below with reference to FIGS. FIG. 14 is a flowchart showing a bulk picking process in an actual work area according to the third embodiment of the present invention.

  In the actual work area RWA shown in FIG. 13, when the bulk picking operation is started, first, the imaging device ID is operated by the two-dimensional imaging device control unit 110, and two-dimensional image data obtained by imaging the inside of the tray TR. Is obtained (step S71). Next, the inference unit 140 uses the learned model storage unit 131 based on the input information from the operator from the input unit (not shown), the two-dimensional imaging device ID stored in the main storage unit 130, and the function information on the robot hand R. One learned model is specified from one or a plurality of learned models stored in (step S72).

  When the learned model is specified, the inference unit 140 refers to the specified learned model, and in the input layer of the learned model, the two-dimensional captured image acquired by the two-dimensional imaging device control unit 110 in step S71. The data is associated, inference (data processing) using the learned model is performed, and three-dimensional coordinate data of the robot hand R capable of holding the workpiece W is output as an output layer (step S73). At this time, in the machine-learned system generated by the machine learning system 1B, the machine learning device 3B, and the machine learning method according to the third embodiment, in most cases, a plurality of three-dimensional coordinate data are output. Then, after the inference unit 140 outputs the plurality of three-dimensional coordinate data, the identifying unit 150 considers the numerical value of the credit associated with each of the plurality of three-dimensional coordinate data, and for example, determines the three-dimensional coordinate associated with the highest credit. The coordinate data is specified as one three-dimensional coordinate data actually used for controlling the robot hand R (step S74). Then, the one specified three-dimensional coordinate data is sent to the robot hand control unit 120, and the robot hand control unit 120 places a predetermined portion of the robot hand R (for example, the center of gravity of the end effector 54) at the position indicated by the three-dimensional coordinate data. The six axes of the robot hand R are driven to try to grip the workpiece W to position the position P) (step S75). If there is only one piece of three-dimensional coordinate data output in step S73, it is not necessary to specify the three-dimensional coordinate data by the specifying unit 150, so the output three-dimensional coordinate data can be directly sent to the robot hand control unit 120. I just need.

  As described above, in the bulk picking to which the data processing system according to the third embodiment of the present invention is applied, the work W is performed only from image data captured by the two-dimensional imaging device ID including a camera having a simple structure. Can be inferred from the three-dimensional coordinate data of the robot hand R capable of gripping, and bulk picking without requiring any manpower can be realized with a simple configuration. Further, after inferring a plurality of three-dimensional coordinate data from the two-dimensional captured image data, the most suitable three-dimensional coordinate data is specified and adopted for the control of the robot hand R, so that the reliability is high, in other words, It is possible to use three-dimensional coordinate data that has a high possibility of successful gripping, thereby realizing a more accurate gripping operation.

  In the data processing system according to the third embodiment, as shown in the second embodiment, the output layer of the trained model includes the three-dimensional coordinate data and the angle data of the robot hand R. Can also be output. In this case, since the angle data is specified in a one-to-one relationship with the three-dimensional coordinate data, the three-dimensional coordinate data and the angle data are always the same in the data set generated by the simulator 3B. It will be easily understood by those skilled in the art that the specifying unit 150 specifies the corresponding angle data by specifying the three-dimensional coordinate data.

  The present invention is not limited to the above-described embodiment, and can be implemented with various modifications without departing from the gist of the present invention. And, they are all included in the technical idea of the present invention.

1, 1A, 1B Machine learning system 2, 2A, 2B Simulator 3, 3A, 3B Machine learning device 21 Information processing unit 23 Two-dimensional captured image data acquisition unit 24 Three-dimensional coordinate data acquisition unit 25 Data set generation unit 27 Angle data acquisition Unit 28 two-dimensional captured image data division unit 31 data set acquisition unit 32 data set storage unit 33 learning unit 34 learned model storage unit 100, 100B robot controller (data processing system)
110 Two-dimensional imaging device control unit (acquisition unit)
120 Robot hand control unit 130 Main storage unit 131 Learned model storage unit 140 Inference unit 150 Specific unit R Robot hand W Work ID 2D imaging device TR Tray RWA Real work area (predetermined area in the real world)
VWA virtual work area (predetermined area of simulator)

Claims (14)

One or more three-dimensional coordinate data of the robot hand when the robot hand on the simulator grips any one or a plurality of workpieces arranged in a predetermined area of the simulator through gripping operation and succeeds in gripping. And a two-dimensional image of the one or more works arranged in the predetermined area from a predetermined angle of view by a two-dimensional imaging device on the simulator before the gripping operation when the robot hand succeeds in gripping. A data set storage unit for acquiring a learning data set including captured image data from the simulator and storing a plurality of sets;
By inputting a plurality of sets of the learning data sets, one or a plurality of workpieces arranged in a predetermined area in the real world are captured by a two-dimensional imaging apparatus from a two-dimensional captured image captured from the same angle of view as the predetermined angle of view. A learning unit for learning a learning model for inferring three-dimensional coordinates of the robot hand in the real world;
A learned model storage unit that stores the learning model learned by the learning unit.
Machine learning device. A machine learning system comprising a simulator and a machine learning device, wherein:
One or a plurality of three-dimensional robot hands when the simulator succeeds in grasping through a grasping operation by a robot hand on the simulator that grasps one or a plurality of works arranged in a predetermined area. Coordinate data and one or a plurality of workpieces arranged in the predetermined area are captured from a predetermined angle of view by a two-dimensional imaging device on the simulator before the gripping operation when the robot hand succeeds in gripping. A function of generating a learning data set including three-dimensional captured image data,
A data set storage unit for acquiring and storing a plurality of sets of the learning data sets from the simulator; and inputting a plurality of sets of the learning data sets to be arranged in a predetermined area in the real world. A learning unit that learns a learning model that infers three-dimensional coordinates of the robot hand in the real world from a two-dimensional captured image obtained by capturing one or more workpieces from the same angle of view as the predetermined angle of view using a two-dimensional imaging device; A learned model storage unit that stores the learning model learned by the learning unit.
Machine learning system. A three-dimensional coordinate data of the robot hand when the robot hand for gripping either one of the deployed multiple workpieces in a predetermined region is to successfully grip through the gripping operation, when the robot hand is to be successful gripping data a plurality of sets stored training data set and a 2-dimensional sensed image data captured from a predetermined angle in the predetermined area two-dimensional imaging device arranged multiple workpieces in front of the gripping operation Set storage unit;
By inputting a plurality of sets of the learning data sets, one or two of a plurality of workpieces in the two-dimensional captured image of the robot hand are grasped from a two-dimensional captured image including a plurality of workpieces. A learning unit for learning a learning model for inferring a plurality of three-dimensional coordinates;
A learned model storage unit that stores the learning model learned by the learning unit.
Machine learning device. An acquisition unit configured to acquire a two-dimensional captured image of one or a plurality of workpieces arranged in a predetermined area, which is captured from a predetermined angle of view by a two-dimensional imaging device;
The three-dimensional coordinates of the robot hand by inputting the two-dimensional captured image acquired by the acquiring unit into a learned model generated by the machine learning device according to any one of claims 1 to 3. And an inference unit for inferring
Data processing system. The inference unit further includes a specifying unit that specifies a predetermined one of the plurality of three-dimensional coordinates when a plurality of three-dimensional coordinates are inferred,
The data processing system according to claim 4. A computer-based machine learning method that:
One or more three-dimensional coordinate data of the robot hand when the robot hand on the simulator grips any of one or a plurality of works arranged in a predetermined area of the simulator and succeeds in gripping through a gripping operation. And a two-dimensional imaging in which one or a plurality of works arranged in the predetermined area is imaged from a predetermined angle of view by a two-dimensional imaging device on the simulator before the gripping operation when the robot hand succeeds in gripping. Acquiring a learning data set including image data from the simulator and storing a plurality of sets;
By inputting a plurality of sets of the learning data sets, one or a plurality of workpieces arranged in a predetermined area in the real world are captured by a two-dimensional imaging apparatus from a two-dimensional captured image captured from the same angle of view as the predetermined angle of view. Learning a learning model for inferring the three-dimensional coordinates of the robot hand in the real world ;
Storing the learned learning model.
Machine learning method. A computer-based machine learning method that:
A three-dimensional coordinate data of the robot hand when the robot hand for gripping either one of the deployed multiple workpieces in a predetermined region is to successfully grip through the gripping operation, when the robot hand is to be successful gripping a step of plural sets stored training data set and a 2-dimensional sensed image data captured from a predetermined angle in the predetermined area two-dimensional imaging device arranged multiple workpieces in front of the gripping operation ;
By inputting a plurality of sets of the learning data sets, one or two of a plurality of workpieces in the two-dimensional captured image of the robot hand are grasped from a two-dimensional captured image including a plurality of workpieces. Learning a learning model for inferring a plurality of three-dimensional coordinates;
Storing the learned learning model.
Machine learning method. One or more three-dimensional coordinate data of the robot hand when the robot hand on the simulator grips any one or a plurality of workpieces arranged in a predetermined area of the simulator through gripping operation and succeeds in gripping. And one or a plurality of workpieces arranged in the predetermined area were imaged from a predetermined angle of view by the two-dimensional imaging device on the simulator before the gripping operation when the robot hand succeeded in gripping. A data set storage unit for acquiring a learning data set including two-dimensional captured image data from the simulator and storing a plurality of sets;
By inputting a plurality of sets of the learning data sets, one or a plurality of workpieces arranged in a predetermined area in the real world are captured by a two-dimensional imaging apparatus from a two-dimensional captured image captured from the same angle of view as the predetermined angle of view. A learning unit for learning a learning model for inferring three-dimensional coordinates and angles of the robot hand in the real world;
A learned model storage unit that stores the learning model learned by the learning unit.
Machine learning device. A machine learning system comprising a simulator and a machine learning device, wherein:
One or a plurality of three-dimensional robot hands when the simulator succeeds in grasping through a grasping operation by a robot hand on the simulator that grasps one or a plurality of works arranged in a predetermined area. Coordinate data and angle data, and one or a plurality of works arranged in the predetermined area before the gripping operation when the robot hand succeeds in gripping the robot hand from a predetermined angle of view by a two-dimensional imaging device on the simulator. A function of generating a learning data set including captured two-dimensional captured image data,
A data set storage unit for acquiring and storing a plurality of sets of the learning data sets from the simulator; and inputting a plurality of sets of the learning data sets to be arranged in a predetermined area in the real world. A learning unit that learns a learning model that infers three-dimensional coordinates and angles of the robot hand in the real world from a two-dimensional captured image obtained by capturing one or more workpieces from the same angle of view as the predetermined angle of view using a two-dimensional imaging device. And a learned model storage unit that stores the learning model learned by the learning unit.
Machine learning system. And the three-dimensional coordinate data and the angle data of the robot hand when the robot hand for gripping either one of the deployed multiple workpieces in a predetermined region is to successfully grip through the gripping operation, the robot hand to grip a plurality of sets of training data set and a 2-dimensional sensed image data captured from a predetermined angle by said gripping the predetermined area two-dimensional imaging device arranged with multiple work in before the operation when a successful A data set storage unit for storing;
By inputting a plurality of sets of the learning data sets, one or two of a plurality of workpieces in the two-dimensional captured image of the robot hand are grasped from a two-dimensional captured image including a plurality of workpieces. A learning unit for learning a learning model for inferring a plurality of three-dimensional coordinates and angles;
A learned model storage unit that stores the learning model learned by the learning unit.
Machine learning device. An acquisition unit configured to acquire two-dimensional captured image data obtained by capturing one or a plurality of works arranged in a predetermined region from a predetermined angle of view by a two-dimensional imaging device;
A three-dimensional robot hand by inputting the two-dimensional captured image data acquired by the acquisition unit to a learned model generated by the machine learning device according to any one of claims 8 to 10. An inference unit that infers coordinates and angles.
Data processing system. The inference unit further includes, when a plurality of three-dimensional coordinates and angles are inferred, a specifying unit that specifies a predetermined one of the plurality of three-dimensional coordinates and angles.
The data processing system according to claim 11 . A computer-based machine learning method that:
One or more three-dimensional coordinate data of the robot hand when the robot hand on the simulator grips any of one or a plurality of works arranged in a predetermined area of the simulator and succeeds in gripping through a gripping operation. 1 and a plurality of workpieces arranged in the predetermined area are imaged from a predetermined angle of view by the two-dimensional imaging device on the simulator before the gripping operation when the robot hand succeeds in gripping. Acquiring from the simulator a learning data set including three-dimensional captured image data, and storing a plurality of learning data sets;
By inputting a plurality of sets of the learning data sets, one or a plurality of workpieces arranged in a predetermined area in the real world are captured by a two- dimensional imaging apparatus from a two-dimensional captured image captured from the same angle of view as the predetermined angle of view. a step of learning a learning model for inferring the three-dimensional coordinates及beauty angles of the robot hand in the real world;
Storing the learned learning model.
Machine learning method. A computer-based machine learning method that:
And the three-dimensional coordinate data and the angle of the robot hand when the robot hand for gripping either one of the deployed multiple workpieces in a predetermined region is to successfully grip through the gripping operation, the robot hand successful gripping wherein the 2-dimensional sensed image data captured from a predetermined angle by the gripping said predetermined area two-dimensional imaging device arranged with multiple work in the previous operation, a plurality of sets stored training data set comprising a time to Performing the steps;
By inputting a plurality of sets of the learning data sets, one or two of a plurality of workpieces in the two-dimensional captured image of the robot hand are grasped from a two-dimensional captured image including a plurality of workpieces. a step of learning a learning model for inferring a plurality of three-dimensional coordinates及beauty angles;
Storing the learned learning model.
Machine learning method.

Images