JP2020168719A - Robot system and control method of robot - Google Patents

Abstract

To provide a machine learning device that selects an optimal robot operation without human intervention when a workpiece is taken out, a robot system, and a machine learning method.SOLUTION: A machine learning device includes: a state amount observation part 21 that observes output data of a three-dimensional measuring instrument 15 for measuring at least a three-dimensional map of each of workpieces 12 placed in a random state including a bulk-loaded state; an operation result acquisition part 26 that acquires a taking-out operation result of a robot 14 that takes out the workpieces by a hand part 13; and a learning part 22 that learns a workpiece taking-out operation by receiving output of the state amount observation part and the operation result acquisition part. The state amount observation part further observes the output data of a coordinate calculation part that calculates a three-dimensional position for each of the workpieces on the basis of the output of the three-dimensional measuring instrument. The learning part includes: a reward calculation part 23 that calculates a reward on the basis of a determination result of a workpiece taking-out success that is output of the operation result acquisition part; and a value function update part 24 that has a value function for determining a value of the workpiece taking-out operation and updates a value function according to the reward.SELECTED DRAWING: Figure 1

Description

The present invention relates to a machine learning device, a robot system, and a machine learning method for learning a movement of taking out works placed in a disorderly manner, including a state in which they are stacked in pieces.

Conventionally, for example, a robot system has been known in which workpieces stacked in a basket-shaped box are gripped and transported by a hand portion of the robot (see, for example, Patent Documents 1 and 2). In such a robot system, for example, the position information of a plurality of workpieces is acquired by using a three-dimensional measuring instrument installed above the basket-shaped box, and the workpieces are individually loaded one by one based on the position information. It is taken out by the hand part.

Japanese Patent No. 5642738 Japanese Patent No. 5670397

However, in the above-mentioned conventional robot system, for example, how to extract the work to be taken out from the distance images of a plurality of works measured by the three-dimensional measuring instrument and at what position the work is to be taken out are determined in advance. Must be set to. In addition, it is necessary to program in advance how to operate the hand part of the robot when taking out the work. Specifically, for example, it is necessary for a human to use a teaching pendant to teach a robot how to take out a work.

Therefore, if the setting for extracting the work to be taken out from the distance images of the plurality of works is not appropriate or the operation program of the robot is not properly created, the success rate when the robot takes out the work and carries it is lowered. In addition, in order to increase the success rate, it is necessary for human beings to improve the work detection setting and the robot operation program while searching for the optimum operation of the robot through trial and error.

Therefore, an object of the present invention is a machine learning device capable of learning the optimum operation of a robot when taking out works placed in a disorderly manner, including a state of being piled up in pieces, without human intervention in view of the above-mentioned actual situation. , To provide robot systems and machine learning methods.

According to the first configuration example of the first embodiment according to the present invention, a machine learning device that learns the operation of a robot that takes out the work by a hand unit from a plurality of works placed randomly, including a state in which they are stacked separately. The operation of acquiring the result of the take-out operation of the robot that takes out the work by the state quantity observation unit that observes the output data of the three-dimensional measuring instrument that measures at least the three-dimensional map for each work and the hand unit. The state quantity observation unit further includes a result acquisition unit, a learning unit that receives an output from the state quantity observation unit and an output from the operation result acquisition unit, and learns the extraction operation of the work. Based on the output of the three-dimensional measuring instrument, the output data of the coordinate calculation unit that calculates the three-dimensional position for each work is also observed, and the learning unit takes out the work which is the output of the operation result acquisition unit. A reward calculation unit that calculates a reward based on the success / failure determination result of the work, a value function update unit that has a value function that determines the value of the extraction operation of the work, and updates the value function according to the reward. A machine learning device is provided. According to the second configuration example of the first embodiment according to the present invention, a machine learning device that learns the operation of a robot that takes out the work by a hand unit from a plurality of works placed randomly, including a state in which they are stacked separately. The operation of acquiring the result of the take-out operation of the robot that takes out the work by the state quantity observing unit that observes the output data of the three-dimensional measuring instrument that measures at least the three-dimensional map for each work and the hand unit. The state quantity observation unit further includes a result acquisition unit, a learning unit that receives an output from the state quantity observation unit and an output from the operation result acquisition unit, and learns the extraction operation of the work. Based on the output of the three-dimensional measuring instrument, the output data of the coordinate calculation unit that calculates the three-dimensional position for each work is also observed, and the learning unit has a learning model for learning the extraction operation of the work. Then, the result of determining the success or failure of taking out the work, which is the output of the operation result acquisition unit, the error calculation unit that calculates the error based on the learning model, and the learning that updates the learning model according to the error. A machine learning device including a model update unit is provided. It is preferable that the machine learning device further includes a decision-making unit that determines command data for instructing the robot to take out the work by referring to the output from the learning unit.

According to the second embodiment of the present invention, it is a machine learning device that learns the operation of a robot that takes out the work by a hand unit from a plurality of randomly placed works including a state in which they are stacked separately. The state quantity observing unit for observing the state quantity of the robot including the output data of the three-dimensional measuring instrument for measuring the three-dimensional map for each work, and the hand unit for taking out the work are acquired as a result of the taking-out operation of the robot. The operation amount including the measurement parameters of the three-dimensional measuring instrument, which receives the output from the operation result acquisition unit, the output from the state amount observation unit, and the output from the operation result acquisition unit, is the state amount of the robot and the extraction operation. A machine learning device including a learning unit for learning in association with the result of the above is provided. It is preferable that the machine learning device further includes a decision-making unit that determines the measurement parameters of the three-dimensional measuring instrument with reference to the operation amount learned by the learning unit.

The state quantity observing unit can also observe the state quantity of the robot including the output data of the coordinate calculation unit that calculates the three-dimensional position for each work based on the output of the three-dimensional measuring instrument. The coordinate calculation unit may further calculate the posture of each work and output the calculated three-dimensional position and posture data of each work. The operation result acquisition unit can use the output data of the three-dimensional measuring instrument. The machine learning device further includes a preprocessing unit that processes the output data of the three-dimensional measuring instrument before inputting to the state quantity observing unit, and the state quantity observing unit processes the output data of the preprocessing unit. It is preferable to receive it as the state quantity of the robot. The preprocessing unit can make the direction and height of each work in the output data of the three-dimensional measuring instrument constant. The operation result acquisition unit can acquire at least one of the success or failure of taking out the work, the damaged state of the work, and the degree of achievement when the taken out work is passed to a subsequent process.

The learning unit has a reward calculation unit that calculates a reward based on the output of the operation result acquisition unit, and a value function that determines the value of the extraction operation of the work, and updates the value function according to the reward. It can be provided with a value function update unit. The learning unit has a learning model that learns the extraction operation of the work, an error calculation unit that calculates an error based on the output of the operation result acquisition unit and the output of the learning model, and the error. It is also possible to provide a learning model update unit that updates the learning model accordingly. The machine learning device preferably has a neural network.

According to the third embodiment of the present invention, the machine learning device learns the operation of a robot that takes out the work by a hand unit from a plurality of randomly placed works including a state in which they are stacked separately. The state quantity observing unit for observing the state quantity of the robot including the output data of the three-dimensional measuring instrument for measuring the three-dimensional map for each work, and the hand unit for taking out the work are acquired as a result of the taking-out operation of the robot. The operation amount including the command data that receives the operation result acquisition unit, the output from the state quantity observation unit, and the output from the operation result acquisition unit, and commands the robot to take out the work, is the robot. A robot system including a machine learning device including a learning unit that learns in association with a state quantity and the result of the retrieval operation, wherein the robot, the three-dimensional measuring instrument, the robot, and the three-dimensional measuring instrument are provided. A robot system including a control device for controlling each of the above is provided.

According to the fourth embodiment of the present invention, the machine learning device learns the operation of a robot that takes out the work by a hand unit from a plurality of randomly placed works including a state in which they are stacked separately. The state quantity observing unit for observing the state quantity of the robot including the output data of the three-dimensional measuring instrument for measuring the three-dimensional map for each work, and the hand unit for taking out the work are acquired as a result of the taking out operation of the robot. The operation amount including the measurement parameters of the three-dimensional measuring instrument, which receives the output from the operation result acquisition unit, the output from the state amount observation unit, and the output from the operation result acquisition unit, is the state amount of the robot and the extraction operation. A robot system including a machine learning device including a learning unit that learns in association with the result of the above, and is a control device that controls the robot, the three-dimensional measuring instrument, the robot, and the three-dimensional measuring instrument, respectively. And, a robot system is provided.

The robot system includes a plurality of the robots, the machine learning device is provided for each robot, and the plurality of machine learning devices provided on the plurality of robots mutually data via a communication medium. It is preferable to share or exchange. The machine learning device may exist on a cloud server.

According to the fifth embodiment of the present invention, the machine learning method is a machine learning method for learning the operation of a robot that takes out the work by a hand unit from a plurality of randomly placed works including a state in which they are stacked separately. Observe the state quantity of the robot including the output data of the three-dimensional measuring instrument that measures the three-dimensional map for each work, acquire the result of the take-out operation of the robot that takes out the work by the hand unit, and observe the state quantity. The operation amount including the output from the unit and the output from the operation result acquisition unit and the command data for instructing the robot to take out the work is associated with the state amount of the robot and the result of the take-out operation. A machine learning method for learning is provided.

According to the machine learning device, the robot system, and the machine learning method according to the present invention, it is possible to learn the optimum movement of the robot when taking out the randomly placed workpieces, including the stacked state, without human intervention. It works.

FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. FIG. 2 is a diagram schematically showing a model of a neuron. FIG. 3 is a diagram schematically showing a three-layer neural network configured by combining the neurons shown in FIG. 2. FIG. 4 is a flowchart showing an example of the operation of the machine learning device shown in FIG. FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention. FIG. 6 is a diagram for explaining an example of processing of the preprocessing unit in the robot system shown in FIG. FIG. 7 is a block diagram showing a modified example of the robot system shown in FIG.

Hereinafter, examples of the machine learning device, the robot system, and the machine learning method according to the present invention will be described in detail with reference to the accompanying drawings. Here, in each drawing, the same member is designated by the same reference numeral. Further, those having the same reference numerals in different drawings mean that they are components having the same function. The scales of these drawings have been changed as appropriate for ease of understanding.

FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. The robot system 10 of the present embodiment is a three-dimensional measuring instrument that measures a three-dimensional map of the surface of the work 12 and the robot 14 to which the hand portion 13 that grips the work 12 that is stacked separately in the basket-shaped box 11 is attached. A control device 16 for controlling a robot 14 and a three-dimensional measuring instrument 15, a coordinate calculation unit 19, and a machine learning device 20 are provided.

Here, the machine learning device 20 includes a state quantity observation unit 21, an operation result acquisition unit 26, a learning unit 22, and a decision-making unit 25. As will be described in detail later, the machine learning device 20 learns and outputs operation amounts such as command data for instructing the robot 14 to take out the work 12 or measurement parameters of the three-dimensional measuring instrument 15.

The robot 14 is, for example, a 6-axis articulated robot, and the drive axes of the robot 14 and the hand unit 13 are controlled by the control device 16. Further, the robot 14 is used to take out the works 12 one by one from the boxes 11 installed at predetermined positions and sequentially move them to a designated place, for example, a conveyor or a workbench (not shown).

By the way, when the work 12 stacked separately is taken out from the box 11, the hand portion 13 or the work 12 may collide with or come into contact with the wall of the box 11. Alternatively, the hand portion 13 or the work 12 may be caught by another work 12. In such a case, a function of detecting the force acting on the hand portion 13 is required so that the overload applied to the robot 14 can be immediately avoided. Therefore, a 6-axis force sensor 17 is provided between the tip of the arm portion of the robot 14 and the hand portion 13. Further, the robot system 10 of the present embodiment also has a function of estimating the force acting on the hand portion 13 based on the current value of a motor (not shown) that drives the drive shaft of each joint portion of the robot 14. ..

Further, since the force sensor 17 can detect the force acting on the hand portion 13, it can also determine whether or not the hand portion 13 actually grips the work 12. That is, when the hand portion 13 grips the work 12, the weight of the work 12 acts on the hand portion 13, so that the detection value of the force sensor 17 exceeds a predetermined threshold value after the work 12 is taken out. Therefore, it can be determined that the hand portion 13 is holding the work 12. Regarding the determination of whether or not the hand unit 13 grips the work 12, for example, shooting data of a camera used for the three-dimensional measuring instrument 15, a photoelectric sensor (not shown) attached to the hand unit 13, or the like is used. It can also be judged by the output. Further, the judgment may be made based on the data of the pressure gauge of the suction type hand described later.

Here, the hand portion 13 may have various forms as long as the work 12 can be held. For example, the hand portion 13 is provided with a form in which the work 12 is gripped by opening and closing two or a plurality of claw portions, or an electromagnet or a negative pressure generator that generates an attractive force with respect to the work 12. You may. That is, in FIG. 1, the hand portion 13 is depicted as gripping the work by two claw portions, but it goes without saying that this is not limited.

The three-dimensional measuring instrument 15 is provided at a predetermined position above the plurality of works 12 by the support portion 18 in order to measure the plurality of works 12. As the three-dimensional measuring instrument 15, for example, a three-dimensional visual sensor that acquires three-dimensional position information by image processing the image data of the work 12 taken by two cameras (not shown) can be used. it can. Specifically, by applying the triangular measurement method, the optical cutting method, the Time-of-flight method, the Depth from Defocus method, or a method using these methods in combination, a three-dimensional map (multiple workpieces stacked separately) is applied. 12 surface positions) are measured.

The coordinate calculation unit 19 calculates (measures) the positions of the surfaces of the plurality of workpieces 12 stacked separately by inputting the three-dimensional map obtained by the three-dimensional measuring instrument 15. That is, using the output of the three-dimensional measuring instrument 15, the three-dimensional position data (x, y, z) or the three-dimensional position data (x, y, z) and the posture data (w) for each work 12 are used. , P, r) can be obtained. Here, the state quantity observing unit 21 receives both the three-dimensional map from the three-dimensional measuring instrument 15 and the position data (attitude data) from the coordinate calculation unit 19, and observes the state quantity of the robot 14. For example, it is possible to receive only the three-dimensional map from the three-dimensional measuring instrument 15 and observe the state quantity of the robot 14. Further, as will be described later with reference to FIG. 5, a preprocessing unit 50 is added, and the preprocessing unit 50 causes the tertiary from the three-dimensional measuring instrument 15 before inputting to the state quantity observation unit 21. It is also possible to process (preprocess) the original map and input it to the state quantity observation unit 21.

It is assumed that the correlation position between the robot 14 and the three-dimensional measuring instrument 15 is determined in advance by calibration. Further, in the three-dimensional measuring instrument 15 of the present invention, a laser distance measuring instrument can be used instead of the three-dimensional visual sensor. That is, by measuring the distance from the position where the three-dimensional measuring instrument 15 is installed to the surface of each work 12 by laser scanning, or by using various sensors such as a monocular camera and a tactile sensor, a plurality of pieces are stacked separately. The three-dimensional position data and the posture (x, y, z, w, p, r) of the work 12 may be acquired.

That is, in the present invention, for example, as long as the data (x, y, z, w, p, r) of each work 12 can be acquired, the three-dimensional measuring instrument 15 to which any three-dimensional measuring method is applied can be applied. be able to. Further, the mode in which the three-dimensional measuring instrument 15 is installed is not particularly limited, and may be fixed to, for example, a floor or a wall, or may be attached to an arm portion or the like of the robot 14.

The three-dimensional measuring instrument 15 acquires a three-dimensional map of a plurality of works 12 separately stacked in the box 11 by a command from the control device 16, and the coordinate calculation unit 19 acquires a plurality of three-dimensional maps based on the three-dimensional map. The data of the three-dimensional position (attitude) of the work 12 is acquired (calculated), and the data is output to the control device 16 and the state quantity observation unit 21 and the operation result acquisition unit 26 of the machine learning device 20 described later. It has become. In particular, in the coordinate calculation unit 19, for example, the boundary between a certain work 12 and another work 12 and the boundary between the work 12 and the box 11 are estimated based on the image data of a plurality of captured works 12. , Data of the three-dimensional position for each work 12 is acquired.

The three-dimensional position data for each work 12 is obtained, for example, by estimating the existence position and the holdable position of each work 12 from the positions of a plurality of points on the surface of the plurality of works 12 stacked separately. Refers to the data that has been created. Of course, the data of the three-dimensional position of each work 12 may include the data of the posture of the work 12.

Further, the acquisition of the three-dimensional position and posture data for each work 12 in the coordinate calculation unit 19 includes the use of a machine learning method. For example, it is also possible to apply object recognition or angle estimation from an input image or a laser distance measuring device using a method such as supervised learning described later.

Then, when the three-dimensional position data for each work 12 is input from the three-dimensional measuring instrument 15 to the control device 16 via the coordinate calculation unit 19, the control device 16 takes out a certain work 12 from the box 11. Controls the operation of 13. At this time, based on the command value (operation amount) corresponding to the optimum position, posture, and take-out direction of the hand unit 13 obtained by the machine learning device 20 described later, the motors of each axis of the hand unit 13 and the robot 14 ( (Not shown) is driven.

Further, the machine learning device 20 is a variable of shooting conditions of the camera used for the three-dimensional measuring instrument 15 (measurement parameters of the three-dimensional measuring instrument 15: for example, an exposure time adjusted at the time of shooting using an exposure meter, and a subject to be photographed. It is also possible to learn (such as the illuminance of the lighting system that illuminates the object) and control the three-dimensional measuring instrument 15 via the control device 16 based on the learned measurement parameter operation amount. Here, the variables of the position / posture estimation condition used by the three-dimensional measuring instrument 15 to estimate the existing position / posture of each work 12 and the position / posture that can be held from the measured positions of the plurality of works 12 are , May be included in the output data of the above-mentioned three-dimensional measuring instrument 15.

Further, the output data from the three-dimensional measuring instrument 15 is processed in advance by the preprocessing unit 50 or the like which will be described in detail later with reference to FIG. 5, and the processed data (image data) is processed by the state quantity observation unit 21. As mentioned above, it is also possible to give to. The operation result acquisition unit 26 can acquire, for example, the result of extracting the work 12 by the hand unit 13 of the robot 14 from the output data (output data of the coordinate calculation unit 19) from the three-dimensional measuring instrument 15. However, in addition to that, for example, the degree of achievement when the taken-out work 12 is passed to the subsequent process, and the operation result such as whether or not there is a state change such as damage of the taken-out work 12 can be obtained by other means (for example,). Needless to say, it can also be acquired via a camera, sensor, etc. provided in the subsequent process. In the above, the state quantity observation unit 21 and the operation result acquisition unit 26 are functional blocks, and it goes without saying that they can be regarded as achieving both functions by one block.

Next, the machine learning device 20 shown in FIG. 1 will be described in detail. The machine learning device 20 extracts useful rules, knowledge representations, judgment criteria, etc. in the set of data input to the device by analysis, outputs the judgment result, and learns knowledge (machine learning). ) Has the function of performing. There are various methods of machine learning, but they can be roughly divided into, for example, "supervised learning", "unsupervised learning", and "reinforcement learning". Furthermore, in order to realize these methods, there is a method called "deep learning" that learns the extraction of the feature amount itself. Note that these machine learning (machine learning device 20) may use a general-purpose computer or processor, but if GPGPU (General-Purpose computing on Graphics Processing Units), a large-scale PC cluster, or the like is applied, the speed becomes higher. It is possible to process.

First, supervised learning is a model in which a large number of sets of data of a certain input and a result (label) are given to the machine learning device 20, the features in those data sets are learned, and the result is estimated from the input. That is, the relationship is acquired in a recursive manner. When this supervised learning is applied to the present embodiment, it can be used, for example, as a part for estimating a work position from a sensor input or a part for estimating the acquisition success probability for a work candidate. For example, it can be realized by using an algorithm such as a neural network described later.

In unsupervised learning, a large amount of input data is given to the learning device to learn how the input data is distributed, and even if the corresponding teacher output data is not given, the input data is subjected to learning. It is a method of learning with a device that performs compression, classification, shaping, etc. For example, the features in those datasets can be clustered among similar people. Using this result, it is possible to realize the prediction of the output by assigning the output so as to set some standard and optimize it.

In addition, as an intermediate problem setting between unsupervised learning and supervised learning, there is also what is called semi-supervised learning, for example, there is a set of input and output data only partially, and other than that Corresponds to the case where the data is input only. In the present embodiment, learning can be performed efficiently by using data (image data, simulation data, etc.) that can be acquired without actually moving the robot in unsupervised learning. ..

Next, reinforcement learning will be described. First, consider the following as a problem setting for reinforcement learning.
・ The robot observes the state of the environment and decides the action.
・ The environment changes according to some rules, and my actions may change the environment.
・ Every time you act, a reward signal will be returned.
・ What we want to maximize is the total amount of (discounted) rewards in the future.
・ Learning starts from a state in which the consequences of behavior are completely unknown or incompletely known. That is, the robot can obtain the result as data only after actually acting. In other words, it is necessary to search for the optimum behavior through trial and error.
-It is also possible to start learning from a good starting point by setting the state of pre-learning (the above-mentioned supervised learning and reverse reinforcement learning) as the initial state so as to imitate human movements.

Here, reinforcement learning is not only for judgment and classification, but also for learning appropriate behavior based on the interaction that behavior gives to the environment, that is, to maximize the reward that can be obtained in the future. Learn how to learn. This means that in the present embodiment, it is possible to acquire an action that affects the future, for example, by breaking the mountain of the work 12 and making it easier to take the work 12 in the future. Hereinafter, the explanation will be continued in the case of Q-learning as an example, but the description is not limited to Q-learning.

Q-learning is a method of learning the value Q (s, a) of selecting an action a under a certain environmental state s. That is, in a certain state s, the action a having the highest value Q (s, a) may be selected as the optimum action. However, at first, the correct value of the value Q (s, a) is not known at all for the combination of the state s and the action a. Therefore, the agent (action subject) selects various actions a under a certain state s, and a reward is given to the action a at that time. As a result, the agent learns a better action choice, that is, the correct value Q (s, a).

The results of behavioral, we want to maximize the sum of the rewards future, finally Q (s, a) = E aims to _{^{[Σ (γ t) r t}} ] become so. Here E [] denotes the expected value, t is the time, parameter γ is called the discount rate to be described later, is r _t compensation at time t, sigma is the sum by the time t. The expected value in this equation is taken when the state changes according to the optimum behavior, and since it is not known, it is learned while searching. Such an update formula for the value Q (s, a) can be expressed by, for example, the following formula (1).

In the above formula (1), s _t represents the state of the environment at time t, a _t represents the action at time t. By the action a _t, the state changes to s _{t + 1.} r _{t + 1} represents the reward obtained by changing the state. Further, the term with max is obtained by multiplying the Q value when the action a having the highest Q value known at that time is selected under the state _{st + 1} by γ. Here, γ is a parameter of 0 <γ ≦ 1 and is called a discount rate. Further, α is a learning coefficient and is in the range of 0 <α ≦ 1.

The above-mentioned formula (1) as a result of the trial a _t, based on the reward r _{t + 1} came back, represents a method for updating the evaluation value Q of the action a _t in state _{s t (s t, a t} ) ing. In other words, the evaluation value Q (s _t, a _t) of the action a in the state s than, reward r _{t + 1} and the evaluation value Q of the best action max a in the next state by the action a (s _t + 1, max if is larger of the sum of _{a t + 1), Q (} s t, a larger a _t), is smaller in the opposite, Q (s _t, has been shown to reduce the a _t). That is, the value of an action in one state is brought closer to the value of the resulting immediate return of action and the best action of that action in the next state.

Here, the expression method of Q (s, a) on the computer is a method of holding the value as a table for all the state action pairs (s, a) and Q (s, a). There is a way to prepare a function that approximates. In the latter method, the above equation (1) can be realized by adjusting the parameters of the approximate function by a method such as the stochastic gradient descent method. As the approximation function, a neural network described later can be used.

In addition, a neural network can be used as a learning model for supervised learning and unsupervised learning, or as an approximation algorithm for value functions in reinforcement learning. FIG. 2 is a diagram schematically showing a model of neurons, and FIG. 3 is a diagram schematically showing a three-layer neural network configured by combining the neurons shown in FIG. 2. That is, the neural network is composed of, for example, an arithmetic unit and a memory that imitate a neuron model as shown in FIG.

As shown in FIG. 2, the neuron outputs an output (result) y for a plurality of inputs x (in FIG. 2, as an example, inputs x1 to inputs x3). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3) corresponding to this input x. As a result, the neuron outputs the result y expressed by the following equation (2). The input x, the result y, and the weight w are all vectors. Further, in the following equation (2), θ is a bias and f _k is an activation function.

A three-layer neural network constructed by combining the neurons shown in FIG. 2 will be described with reference to FIG. As shown in FIG. 3, a plurality of inputs x (here, as an example, inputs x1 to input x3) are input from the left side of the neural network, and a result y (here, as an example, results y1 to input y3) are input from the right side. ) Is output. Specifically, the inputs x1, x2, and x3 are input with corresponding weights applied to each of the three neurons N11 to N13. The weights applied to these inputs are collectively labeled W1.

The neurons N11 to N13 output z11 to z13, respectively. In FIG. 3, these z11 to z13 are collectively referred to as a feature vector Z1, and can be regarded as a vector obtained by extracting the feature amount of the input vector. This feature vector Z1 is a feature vector between the weights W1 and W2. z11 to z13 are input with corresponding weights applied to each of the two neurons N21 and N22. The weights applied to these feature vectors are collectively labeled W2.

Neurons N21 and N22 output z21 and z22, respectively. In FIG. 3, these z21 and z22 are collectively marked as the feature vector Z2. This feature vector Z2 is a feature vector between the weights W2 and W3. z21 and z22 are input with corresponding weights applied to each of the three neurons N31 to N33. The weights applied to these feature vectors are collectively labeled W3.

Finally, the neurons N31 to N33 output the result y1 to the result y3, respectively. The operation of the neural network includes a learning mode and a value prediction mode. For example, in the learning mode, the weight W is learned using the learning data set, and the behavior of the robot is determined in the prediction mode using the parameters. For convenience, I wrote it as prediction, but it goes without saying that various tasks such as detection, classification, and inference are possible.

Here, the data obtained by actually moving the robot in the prediction mode is immediately learned and reflected in the next action (online learning), or the learning is summarized using the data group collected in advance. After that, the detection mode can be performed with that parameter (batch learning). Alternatively, it is also possible to insert a learning mode every time data is accumulated to some extent in the middle.

Further, the weights W1 to W3 can be learned by the error back propagation method (error reverse propagation method: backpropagation). The error information enters from the right side and flows to the left side. The error back propagation method is a method of adjusting (learning) the weights of each neuron so as to reduce the difference between the output y when the input x is input and the true output y (teacher).

Such a neural network can have three or more layers and more layers (called deep learning). It is also possible to automatically acquire an arithmetic unit that returns the results by stepwise extracting the features of the input from only the teacher data.

Therefore, in order to carry out the above-mentioned Q-learning, the machine learning device 20 of the present embodiment has a state quantity observation unit 21, an operation result acquisition unit 26, a learning unit 22, and a decision-making unit, as shown in FIG. It has 25. However, as described above, the machine learning method applied to the present invention is not limited to Q-learning. That is, various methods such as "supervised learning", "unsupervised learning", "semi-supervised learning", and "reinforcement learning" that can be used in machine learning devices can be applied. A general-purpose computer or processor may be used for these machine learning (machine learning device 20), but if a GPGPU, a large-scale PC cluster, or the like is applied, processing can be performed at a higher speed.

That is, according to the present embodiment, it is a machine learning device that learns the operation of a robot 14 that takes out a work 12 by a hand unit 13 from a plurality of randomly placed works 12 including a state in which they are stacked separately. The state of the robot 14 including the output data of the three-dimensional measuring instrument 15 for measuring the three-dimensional position (x, y, z) for each twelve or the three-dimensional position and the posture (x, y, z, w, p, r). The state quantity observation unit 21 for observing the amount, the operation result acquisition unit 26 for acquiring the result of the extraction operation of the robot 14 for extracting the work 12 by the hand unit 13, and the output and operation result acquisition unit 26 from the state quantity observation unit 21. A learning unit 22 that receives an output from the robot 14 and learns an operation amount including command data for instructing the robot 14 to take out the work 12 in association with the state amount of the robot 14 and the result of the take-out operation.

The state quantity observed by the state quantity observing unit 21 may include, for example, state variables for setting the position, posture, and taking-out direction of the hand portion 13 when a certain work 12 is taken out from the box 11. Further, the learned operation amount includes, for example, command values such as torque, speed, and rotation position given to each drive shaft of the robot 14 and the hand unit 13 from the control device 16 when the work 12 is taken out from the box 11. You may.

Then, when the learning unit 22 takes out one of the plurality of works 12 stacked separately, the learning unit 22 learns by associating the above state variable with the result of the taking-out operation of the work 12 (output of the operation result acquisition unit 26). .. That is, the control device 16 randomly sets the output data of the three-dimensional measuring instrument 15 (coordinate calculation unit 19) and the command data of the hand unit 13, or randomly sets them based on a predetermined rule and hands. The work 12 is taken out by the unit 13. Here, as the above-mentioned predetermined rule, for example, among a plurality of works 12 stacked separately, the work having the highest height (z) direction is taken out in order. As a result, the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 correspond to the act of taking out a certain work. Then, success and failure of taking out the work 12 occur, and each time such success and failure occur, the learning unit 22 is a state variable composed of the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13. Will be evaluated.

Further, the learning unit 22 stores the output data of the three-dimensional measuring instrument 15 when the work 12 is taken out and the command data of the hand unit 13 in association with the evaluation of the result of the taking-out operation of the work 12. As an example of failure, there is a case where the hand portion 13 cannot grip the work 12, or a case where the work 12 collides with or comes into contact with the wall of the box 11 even if the work 12 can be gripped. Further, the success or failure of taking out the work 12 is determined based on the detected value of the force sensor 17 and the imaging data by the three-dimensional measuring instrument. Here, the machine learning device 20 can also perform learning by using, for example, a part of the command data of the hand unit 13 output from the control device 16.

Here, the learning unit 22 of the present embodiment preferably includes a reward calculation unit 23 and a value function update unit 24. For example, the reward calculation unit 23 calculates a reward, for example, a score based on the success or failure of taking out the work 12 due to the above state variable. The reward is high for the successful removal of the work 12, and the reward is low for the failure of the removal of the work 12. Further, the reward may be calculated based on the number of times the work 12 is successfully taken out within a predetermined time. Further, when calculating this reward, for example, the reward is paid according to each stage of taking out the work 12, such as successful gripping by the hand unit 13, successful transportation by the hand unit 13, successful placement operation of the work 12. You may calculate.

Then, the value function update unit 24 has a value function that determines the value of the retrieval operation of the work 12, and updates the value function according to the above reward. To update this value function, the update formula of value Q (s, a) as described above is used. Furthermore, it is preferable to create an action value table at the time of this update. The action value table referred to here is the output data of the three-dimensional measuring instrument 15 when the work 12 is taken out, the command data of the hand unit 13, and the value function updated according to the take-out result of the work 12 at that time (that is, the value function). Evaluation value) and recorded in association with each other.

As this action value table, it is also possible to use a function approximated by using the above-mentioned neural network, which is particularly effective when the amount of information of the state s is enormous such as image data. Further, the above value function is not limited to one type. For example, a value function for evaluating the success or failure of gripping the work 12 by the hand unit 13 and a value function for evaluating the time (cycle time) required for the hand unit 13 to grip and transport the work 12 can be considered.

Further, as the above value function, a value function for evaluating the interference between the box 11 and the hand unit 13 or the work 12 at the time of taking out the work may be used. In order to calculate the reward used for updating the value function, the state quantity observing unit 21 preferably observes the force acting on the hand unit 13, for example, the value detected by the force sensor 17. Then, when the amount of change in force detected by the force sensor 17 exceeds a predetermined threshold value, it can be estimated that the above interference has occurred. Therefore, the reward in that case is set to, for example, a negative value, and the value determined by the value function is low. It is preferable to be.

Further, according to the present embodiment, it is also possible to learn the measurement parameters of the three-dimensional measuring instrument 15 as the operation amount. That is, according to the present embodiment, it is a machine learning device that learns the operation of a robot 14 that takes out a work 12 by a hand unit 13 from a plurality of randomly placed works 12 including a state in which they are stacked separately. The state of the robot 14 including the output data of the three-dimensional measuring instrument 15 that measures the three-dimensional position (x, y, z) for each twelve or the three-dimensional position and the posture (x, y, z, w, p, r). The state quantity observation unit 21 for observing the amount, the operation result acquisition unit 26 for acquiring the result of the extraction operation of the robot 14 for extracting the work 12 by the hand unit 13, and the output and operation result acquisition unit 26 from the state quantity observation unit 21. The learning unit 22 receives the output from the device and learns the operation amount including the measurement parameters of the three-dimensional measuring instrument 15 in association with the state amount of the robot 14 and the result of the extraction operation.

Further, the robot system 10 of the present embodiment may be provided with an automatic hand exchange device (not shown) for exchanging the hand unit 13 attached to the robot 14 with the hand unit 13 of another embodiment. In that case, it is preferable that the value function updating unit 24 has the above-mentioned value function for each hand unit 13 having a different form, and updates the value function of the exchanged hand unit 13 according to the reward. As a result, the optimum operation of the hand unit 13 can be learned for each of the plurality of hands 13 having different forms, so that the hand unit 13 having a higher value function can be selected by the automatic hand exchange device.

Subsequently, the decision-making unit 25 selects, for example, the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 corresponding to the highest evaluation value by referring to the action value table created as described above. It is preferable to do so. After that, the decision-making unit 25 outputs the optimum data of the selected hand unit 13 and the three-dimensional measuring instrument 15 to the control device 16.

Then, the control device 16 controls the three-dimensional measuring instrument 15 and the robot 14 by using the optimum data of the hand unit 13 and the three-dimensional measuring instrument 15 output by the learning unit 22, and takes out the work 12. For example, the control device 16 operates each drive axis of the hand unit 13 and the robot 14 based on the state variables that set the optimum position, posture, and take-out direction of the hand unit 13 obtained by the learning unit 22. Is preferable.

As shown in FIG. 1, the robot system 10 of the above-described embodiment is provided with one machine learning device 20 for one robot 14. However, in the present invention, the number of each of the robot 14 and the machine learning device 20 is not limited to one. For example, the robot system 10 may include a plurality of robots 14, and one or more machine learning devices 20 may be provided corresponding to each robot 14. Then, it is preferable that the robot system 10 shares or mutually exchanges the optimum state variables of the three-dimensional measuring instrument 15 and the hand unit 13 acquired by the machine learning device 20 of each robot 14 by a communication medium such as a network. As a result, even if the operating rate of one robot 14 is lower than the operating rate of another robot 14, the optimum operation result acquired by the machine learning device 20 provided in another robot 14 is used for the operation of the certain robot 14. be able to. In addition, the time required for learning can be shortened by sharing the learning model among a plurality of robots, or by sharing the operation amount including the measurement parameters of the three-dimensional measuring instrument 15 and the state amount of the robot 14 and the result of the extraction operation. Can be done.

Further, the machine learning device 20 may be inside the robot 14 or outside the robot 14. Alternatively, the machine learning device 20 may exist in the control device 16 or in a cloud server (not shown).

Further, when the robot system 10 includes a plurality of robots 14, the work of taking out the work 12 to the hand portion of another robot 14 is performed while the work 12 gripped by the hand portion 13 is carried by one robot 14. It is possible to make it. Then, the value function updating unit 24 can update the value function by utilizing the time between the switching of the robot 14 that takes out the work 12. Further, the machine learning device 20 has a state variable of a plurality of hand models, performs a retrieval simulation with the plurality of hand models during the retrieval operation of the work 12, and performs a retrieval simulation with the plurality of hand models according to the result of the retrieval simulation. It is also possible to learn the state variable of the above in association with the result of the extraction operation of the work 12.

In the machine learning device 20 described above, the output data of the three-dimensional measuring instrument 15 when the data of the three-dimensional map for each work 12 is acquired is transmitted from the three-dimensional measuring instrument 15 to the state quantity observation unit 21. It has become like. Since such transmitted data does not necessarily include abnormal data, the machine learning device 20 has a filtering function for abnormal data, that is, data from the three-dimensional measuring instrument 15 is stored in the state quantity observing unit 21. It is possible to have a function that can select whether or not to input to. As a result, the learning unit 22 of the machine learning device 20 can efficiently learn the optimum operation of the hand unit 13 by the three-dimensional measuring instrument 15 and the robot 14.

Further, in the machine learning device 20 described above, the output data from the learning unit 22 is input to the control device 16, but the output data from the learning unit 22 does not include abnormal data. Therefore, it may have a function of filtering abnormal data, that is, a function of selecting whether or not to output the data from the learning unit 22 to the control device 16. As a result, the control device 16 enables the robot 14 to more safely execute the optimum operation of the hand unit 13.

The above-mentioned abnormal data can be detected by the following procedure. That is, the probability distribution of the input data is estimated, the probability of occurrence of new input is derived using the probability distribution, and if the probability of occurrence is below a certain level, it is regarded as abnormal data that greatly deviates from the typical behavior. Data can be detected.

Next, an example of the operation of the machine learning device 20 provided in the robot system 10 of the present embodiment will be described. FIG. 4 is a flowchart showing an example of the operation of the machine learning device shown in FIG. As shown in FIG. 4, when the learning operation (learning process) starts in the machine learning device 20 shown in FIG. 1, the three-dimensional measuring instrument 15 performs and outputs three-dimensional measurement (step S11 in FIG. 4). .. That is, in step S11, for example, a three-dimensional map (output data of the three-dimensional measuring instrument 15) for each of the randomly placed works 12 including the separately stacked states is acquired and output to the state quantity observation unit 21. At the same time, the coordinate calculation unit 19 receives the three-dimensional map for each work 12, calculates the three-dimensional position (x, y, z) for each work 12, and calculates the state quantity observation unit 21, the operation result acquisition unit 26, and the control device 16. Output to. Here, the coordinate calculation unit 19 may calculate and output the posture (w, p, r) for each work 12 from the output of the three-dimensional measuring instrument 15.

As will be described with reference to FIG. 5, the output (three-dimensional map) of the three-dimensional measuring instrument 15 is state quantity observation via the preprocessing unit 50 that is processed before being input to the state quantity observation unit 21. It may be input to the unit 21. Further, as described with reference to FIG. 7, only the output of the three-dimensional measuring instrument 15 may be input to the state quantity observing unit 21, and further, only the output of the three-dimensional measuring instrument 15 may input the preprocessing unit 50. It may be input to the state quantity observation unit 21 via. As described above, the execution and output of the three-dimensional measurement in step S11 can include various things.

Specifically, in the case of FIG. 1, the state quantity observation unit 21 has a three-dimensional map for each work 12 from the three-dimensional measuring instrument 15, and a three-dimensional position (x) for each work 12 from the coordinate calculation unit 19. , Y, z) and state quantities (output data of the three-dimensional measuring instrument 15) such as the posture (w, p, r) are observed. The operation result acquisition unit 26 acquires the result of the extraction operation of the robot 14 that extracts the work 12 by the hand unit 13 from the output data of the three-dimensional measuring instrument 15 (output data of the coordinate calculation unit 19). In addition to the output data of the three-dimensional measuring instrument, the operation result acquisition unit 26 also acquires the result of the extraction operation such as the degree of achievement when the extracted work 12 is passed to the subsequent process and the damage of the extracted work 12. be able to.

Further, for example, the machine learning device 20 determines the optimum operation based on the output data of the three-dimensional measuring instrument 15 (step S12 in FIG. 4), and the control device 16 is the hand unit 13 (robot 14). The command data (operation amount) of the above is output, and the work 12 is taken out (step S13 in FIG. 4). Then, the work extraction result is acquired by the operation result acquisition unit 26 described above (step S14 in FIG. 4).

Next, the success or failure of taking out the work 12 is determined from the output from the operation result acquisition unit 26 (step S15 in FIG. 4), and if the work 12 is taken out successfully, a positive reward is set (in FIG. 4). In step S16), if the extraction of the work 12 fails, a negative reward is set (step S17 in FIG. 4), and the action value table (value function) is updated (step S18 in FIG. 4).

Here, the success / failure determination of the removal of the work 12 can be made based on, for example, the output data of the three-dimensional measuring instrument 15 after the removal operation of the work 12. Further, the success / failure determination of the removal of the work 12 is not limited to the evaluation of the success / failure of the removal of the work 12, for example, the degree of achievement when the removed work 12 is passed to the subsequent process, the damage of the removed work 12 and the like. It may be evaluated whether or not there is a change in the state of the work 12 or the time (cycle time) and energy (electric energy) required to grip and transport the work 12 by the hand unit 13.

The reward calculation unit 23 calculates the reward value based on the success / failure determination of the retrieval of the work 12, and the value function update unit 24 updates the action value table. That is, when the learning unit 22 succeeds in taking out the work 12, it sets a positive reward to the reward in the above-mentioned update formula of the value Q (s, a) (S16), and fails to take out the work 12. If so, a negative reward is set for the reward in the renewal formula (S17). Then, the learning unit 22 updates the above-mentioned action value table each time the work 12 is taken out (S18). By repeating the above steps S11 to S18, the learning unit 22 continues (learns) to update the action value table.

In the above, the data input to the state quantity observation unit 21 is not limited to the output data of the three-dimensional measuring instrument 15, and may include, for example, data such as the output of another sensor, and further, the control device 16 It is also possible to use a part of the command data from. In this way, the control device 16 causes the robot 14 to execute the take-out operation of the work 12 by using the command data (operation amount) output from the machine learning device 20. The learning by the machine learning device 20 is not limited to the extraction operation of the work 12, and may be, for example, the measurement parameter of the three-dimensional measuring instrument 15 as described above.

As described above, according to the robot system 10 provided with the machine learning device 20 of the present embodiment, the robot that takes out the work 12 by the hand unit 13 from a plurality of randomly placed works 12 including the state of being stacked separately. 14 movements can be learned. As a result, the robot system 10 can learn the selection of the optimum operation of the robot 14 for taking out the work 12 stacked in pieces without human intervention.

FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention, and shows a robot system to which supervised learning is applied. As is clear from the comparison between FIG. 5 and FIG. 1 described above, the robot system 10'applied with supervised learning shown in FIG. 5 is relative to the robot system 10 to which Q learning (reinforcement learning) shown in FIG. 1 is applied. Further, a data recording unit 40 with a result (label) is provided. The robot system 10'shown in FIG. 5 further includes a preprocessing unit 50 that preprocesses the output data of the three-dimensional measuring instrument 15. Needless to say, the preprocessing unit 50 can also be provided for the robot system 10 shown in FIG. 1, for example.

As shown in FIG. 5, the machine learning device 30 in the robot system 10'applied with supervised learning includes a state quantity observation unit 31, an operation result acquisition unit 36, a learning unit 32, a decision-making unit 35, and the like. To be equipped. The learning unit 32 includes an error calculation unit 33 and a learning model update unit 34. Also in the robot system 10'of the present embodiment, the machine learning device 30 learns the operation amount such as the command data for instructing the robot 14 to take out the work 12 or the measurement parameters of the three-dimensional measuring instrument 15. Output.

That is, in the robot system 10'to which the supervised learning shown in FIG. 5 is applied, the error calculation unit 33 and the learning model update unit 34 are the reward calculation unit 23 and the reward calculation unit 23 in the robot system 10 to which the Q learning shown in FIG. Corresponds to the value function update unit 24. The other configurations, for example, the configurations of the three-dimensional measuring instrument 15, the control device 16, the robot 14, and the like are the same as those in FIG. 1, and the description thereof will be omitted.

The error calculation unit 33 calculates the error between the result (label) output from the operation result acquisition unit 36 and the output of the learning model implemented in the learning unit. Here, the data recording unit 40 with a result (label) is obtained, for example, by the day before a predetermined day when the robot 14 is made to perform the work when the shape of the work 12 and the processing by the robot 14 are the same. ) Is held, and the result (labeled) data held in the result (labeled) data recording unit 40 can be provided to the error calculation unit 33 on the predetermined day. Alternatively, data obtained by a simulation or the like performed outside the robot system 10'or data with a result (label) of another robot system is calculated by using a memory card or a communication line to calculate the error of the robot system 10'. It is also possible to provide to the unit 33. Further, the data recording unit 40 with a result (label) is composed of a non-volatile memory such as a flash memory (Flash Memory), and the data recording unit (non-volatile memory) 40 with a result (label) is built in the learning unit 32. The data with the result (label) held in the data recording unit 40 with the result (label) can be used as it is in the learning unit 32.

FIG. 6 is a diagram for explaining an example of processing of the preprocessing unit in the robot system shown in FIG. 5, and FIG. 6A shows three-dimensional positions of a plurality of works 12 separately stacked in the box 11 (FIG. 6). An example of the data of the posture), that is, the output data of the three-dimensional measuring instrument 15, is shown, and FIGS. 6 (b) to 6 (d) perform preprocessing on the workpieces 121 to 123 in FIG. 6 (a). An example of the image data after this is shown.

Here, the work 12 (121 to 123) is assumed to be a cylindrical metal part, and the hand (13) is not gripped by two claws, but is, for example, a cylindrical work 12 It is assumed that the suction pad absorbs the central part of the length of the cylinder with negative pressure. Therefore, for example, if the position of the longitudinal central portion of the work 12 is known, the work 12 can be taken out by moving the suction pad (13) with respect to the position and sucking the work 12. The numerical values in FIGS. 6 (a) to 6 (d) are represented by [mm] and indicate the x direction, the y direction, and the z direction, respectively. In the z direction, the height (depth) of image data obtained by capturing a box 11 in which a plurality of works 12 are stacked separately by a three-dimensional measuring instrument 15 (for example, having two cameras) provided above. Corresponds to the direction.

As is clear from the comparison between FIGS. 6 (a) and 6 (b) to 6 (d), as an example of the processing of the preprocessing unit 50 in the robot system 10'shown in FIG. 5, a three-dimensional measuring instrument is used. From the output data (three-dimensional image) of 15, the work 12 (for example, three works 121 to 123) of interest is rotated and processed so that the height of the center becomes “0”.

That is, the output data of the three-dimensional measuring instrument 15 includes, for example, information on the three-dimensional position (x, y, z) and posture (w, p, r) of the longitudinal central portion of each work 12. .. At this time, as shown in FIGS. 6 (b), 6 (c) and 6 (d), the three workpieces 121, 122 and 123 of interest are rotated by −r and subtracted by z, respectively. All of them are designed to meet the same conditions. By performing such preprocessing, it becomes possible to reduce the load on the machine learning device 30.

Here, the three-dimensional image shown in FIG. 6A is not the output data of the three-dimensional measuring instrument 15 itself, but from, for example, an image obtained by a program that defines the extraction order of the work 12 that has been carried out from before. , The threshold value for selection is lowered, and this process itself can also be performed by the preprocessing unit 50. Needless to say, the processing by the pretreatment unit 50 can be variously changed depending on various conditions such as the shape of the work 12 and the type of the hand 13.

In this way, the output data (three-dimensional map for each work 12) of the three-dimensional measuring instrument 15 processed by the preprocessing unit 50 before being input to the state quantity observation unit 31 is input to the state quantity observation unit 31. Will be done. Again, referring to FIG. 5, the error calculation unit 33 that receives the result (label) output from the operation result acquisition unit 36 actually, for example, when the output of the neural network shown in FIG. 3 as a learning model is y. When the work 12 is taken out and succeeded, it is considered that there is an error of -log (y), and when it fails, it is considered that there is an error of -log (1-y), and this error should be minimized. Process with the goal of. The input of the neural network shown in FIG. 3 includes, for example, the image data of the works 121 to 123 of interest after performing the preprocessing as shown in FIGS. 6 (b) to 6 (d), and The data of the three-dimensional position and the posture (x, y, z, w, p, r) for each of the works 121 to 123 of interest will be given.

FIG. 7 is a block diagram showing a modified example of the robot system shown in FIG. As is clear from the comparison between FIG. 7 and FIG. 1, in the modified example of the robot system 10 shown in FIG. 7, the coordinate calculation unit 19 is deleted, and the state quantity observation unit 21 is three-dimensional from the three-dimensional measuring instrument 15. Only the map is received and the state quantity of the robot 14 is observed. Needless to say, the control device 16 can be provided with a configuration corresponding to the coordinate calculation unit 19. Further, the configuration shown in FIG. 7 can be applied to, for example, the robot system 10'to which the supervised learning described with reference to FIG. 5 is applied. That is, in the robot system 10'shown in FIG. 5, the preprocessing unit 50 may be deleted, and the state quantity observing unit 31 may receive only the three-dimensional map from the three-dimensional measuring instrument 15 and observe the state quantity of the robot 14. It is possible. Thus, each of the above embodiments can be modified and modified in various ways.

As described in detail above, according to the present embodiment, a machine learning device capable of learning the optimum operation of a robot when taking out works placed in a disorderly manner, including a state in which they are stacked separately, without human intervention. It will be possible to provide robot systems and machine learning methods. The machine learning devices 20 and 30 in the present invention are not limited to those to which reinforcement learning (for example, Q-learning) or supervised learning is applied, and various machine learning algorithms can be applied.

Although the embodiments have been described above, all the examples and conditions described here are described for the purpose of assisting the understanding of the concept of the invention applied to the invention and the technology, and the examples and conditions described in particular are described. It is not intended to limit the scope of the invention. Nor does such description in the specification indicate the advantages and disadvantages of the invention. Although embodiments of the invention have been described in detail, it should be understood that various modifications, replacements and modifications can be made without departing from the spirit and scope of the invention.

10, 10'Robot system 11 Box 12 Work 13 Hand part 14 Robot 15 Three-dimensional measuring instrument 16 Control device 17 Force sensor 18 Support part 19 Coordinate calculation part 20, 30 Machine learning device 21, 31 State quantity observation unit 22, 32 Learning Part 23 Reward calculation part 24 Value function update part 25,35 Decision part 26,36 Operation result acquisition part 33 Error calculation part 34 Learning model update part 40 Data recording part with result (label) 50 Preprocessing part

Claims (19)

It is a machine learning device that learns the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
The state quantity observing unit (21, 31) for observing the output data of the three-dimensional measuring instrument (15) for measuring at least the three-dimensional map for each work (12).
The operation result acquisition unit (26, 36) for acquiring the result of the extraction operation of the robot (14) for extracting the work (12) by the hand unit (13), and
The learning unit (22, 32) that receives the output from the state quantity observation unit (21, 31) and the output from the operation result acquisition unit (26, 36) and learns the extraction operation of the work (12). And with
The state quantity observation unit (21, 31) further outputs data of the coordinate calculation unit (19) that calculates a three-dimensional position for each work (12) based on the output of the three-dimensional measuring instrument (15). Also observed,
The learning unit (22)
A reward calculation unit (23) that calculates a reward based on a determination result of success or failure of taking out the work, which is an output of the operation result acquisition unit (26).
It has a value function that determines the value of the retrieval operation of the work (12), and includes a value function update unit (24) that updates the value function according to the reward.
A machine learning device characterized by that. It is a machine learning device that learns the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
The state quantity observing unit (21, 31) for observing the output data of the three-dimensional measuring instrument (15) for measuring at least the three-dimensional map for each work (12).
The operation result acquisition unit (26, 36) for acquiring the result of the extraction operation of the robot (14) for extracting the work (12) by the hand unit (13), and
With the learning unit (22, 32) that receives the output from the state quantity observation unit (21, 31) and the output from the operation result acquisition unit (26, 36) and learns the extraction operation of the work (12). With,
The state quantity observation unit (21, 31) further outputs data of the coordinate calculation unit (19) that calculates a three-dimensional position for each work (12) based on the output of the three-dimensional measuring instrument (15). Also observed,
The learning unit (32) has a learning model for learning the extraction operation of the work (12).
An error calculation unit (33) that calculates an error based on the result of determining the success or failure of taking out the work, which is the output of the operation result acquisition unit (26), and the learning model.
A learning model update unit (34) that updates the learning model according to the error is provided.
A machine learning device characterized by that. further,
A decision-making unit (25, 35) for determining command data for instructing the robot (14) to take out the work (12) with reference to the output from the learning unit (22, 32) is provided.
The machine learning apparatus according to claim 1 or 2. It is a machine learning device that learns the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
State quantity observation units (21, 31) that observe the output data of the three-dimensional measuring instrument (15) that measures at least the three-dimensional map for each work (12) and the measurement parameters of the three-dimensional measuring instrument (15). When,
The operation result acquisition unit (26, 36) for acquiring the result of the extraction operation of the robot (14) for extracting the work (12) by the hand unit (13), and
The learning unit (22, 32) that receives the output from the state quantity observation unit (21, 31) and the output from the operation result acquisition unit (26, 36) and learns the extraction operation of the work (12). And with
The state quantity observation unit (21, 31) further outputs data of the coordinate calculation unit (19) that calculates a three-dimensional position for each work (12) based on the output of the three-dimensional measuring instrument (15). Also observed,
The learning unit (22)
A reward calculation unit (23) that calculates a reward based on a determination result of success or failure of taking out the work, which is an output of the operation result acquisition unit (26).
It has a value function that determines the value of the retrieval operation of the work (12), and includes a value function update unit (24) that updates the value function according to the reward.
A machine learning device characterized by that. It is a machine learning device that learns the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
State quantity observation units (21, 31) that observe the output data of the three-dimensional measuring instrument (15) that measures at least the three-dimensional map for each work (12) and the measurement parameters of the three-dimensional measuring instrument (15). When,
The operation result acquisition unit (26, 36) for acquiring the result of the extraction operation of the robot (14) for extracting the work (12) by the hand unit (13), and
The learning unit (22, 32) that receives the output from the state quantity observation unit (21, 31) and the output from the operation result acquisition unit (26, 36) and learns the extraction operation of the work (12). And with
The state quantity observation unit (21, 31) further outputs data of the coordinate calculation unit (19) that calculates a three-dimensional position for each work (12) based on the output of the three-dimensional measuring instrument (15). Also observed,
The learning unit (32) has a learning model for learning the extraction operation of the work (12).
An error calculation unit (33) that calculates an error based on the result of determining the success or failure of taking out the work, which is the output of the operation result acquisition unit (26), and the learning model.
A learning model update unit (34) that updates the learning model according to the error is provided.
A machine learning device characterized by that. further,
With reference to the output from the learning units (22, 32), the command data for instructing the robot (14) to take out the work (12) and the measurement parameters of the three-dimensional measuring instrument (15) are determined. Equipped with decision-making departments (25, 35)
The machine learning apparatus according to claim 4 or 5. The coordinate calculation unit (19) further
The posture for each work (12) is calculated, and the calculated three-dimensional position and posture data for each work (12) is output.
The machine learning device according to any one of claims 1 to 6, wherein the machine learning device is characterized in that. The operation result acquisition unit (26, 36) uses the output data of the three-dimensional measuring instrument (15).
The machine learning device according to any one of claims 1 to 7, wherein the machine learning device is characterized. further,
A preprocessing unit (50) that processes the output data of the three-dimensional measuring instrument (15) before inputting it to the state quantity observation unit (21, 31) is provided.
The state quantity observation unit (21, 31) receives the output data of the preprocessing unit (50).
The machine learning device according to any one of claims 1 to 8, wherein the machine learning device is characterized by the above. The preprocessing unit (50) is used for each work (12) in the three-dimensional position data (x, y, z) and posture data (w, p, r) output from the three-dimensional measuring instrument (15). Align the direction and height to a certain level,
The machine learning device according to claim 9. The operation result acquisition unit (26, 36)
The success or failure of taking out the work (12), the damaged state of the work (12), and the time and energy required to grip and carry the work (12) by the hand portion (13), or the above. Obtain at least one of those evaluated for time and said energy,
The machine learning device according to any one of claims 1 to 10. The machine learning device
The machine learning device according to any one of claims 1 to 11, which has a neural network. A robot system (10, 10') including the machine learning device (20, 30) according to any one of claims 1 to 12.
With the robot (14)
With the three-dimensional measuring instrument (15)
A control device (16) for controlling the robot (14) and the three-dimensional measuring instrument (15), respectively.
A robot system characterized by that. The robot system (10, 10') includes a plurality of the robots (14).
The machine learning devices (20, 30) are provided for each robot (14), respectively.
The plurality of machine learning devices (20) provided in the plurality of robots (14) are adapted to share or exchange data with each other via a communication medium.
13. The robot system according to claim 13. The machine learning device (20, 30) exists on the cloud server.
The robot system according to claim 14. It is a machine learning method for learning the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
Observe the output data of the 3D measuring instrument (15) that measures at least the 3D map for each work (12).
The result of the take-out operation of the robot (14) that takes out the work (12) by the hand unit (13) is acquired.
Upon receiving at least a three-dimensional map for each of the observed work (12) and the acquired result of the retrieval operation of the robot (14), the retrieval operation of the work (12) is learned.
The observation of the output data of the three-dimensional measuring instrument (15) also observes the three-dimensional position data obtained by calculating the three-dimensional position of each work (12) based on the output of the three-dimensional measuring instrument (15). And
The learning of the extraction operation is
The reward is calculated based on the result of determining the success or failure of taking out the work, which is the result of the taken-out operation of the robot (14) acquired.
The value function that determines the value of the retrieval operation of the work (12) is updated according to the reward.
A machine learning method characterized by that. It is a machine learning method for learning the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
Observe the output data of the 3D measuring instrument (15) that measures at least the 3D map for each work (12).
The result of the take-out operation of the robot (14) that takes out the work (12) by the hand unit (13) is acquired.
Upon receiving at least a three-dimensional map for each of the observed work (12) and the acquired result of the retrieval operation of the robot (14), the retrieval operation of the work (12) is learned.
The observation of the output data of the three-dimensional measuring instrument (15) also observes the three-dimensional position data obtained by calculating the three-dimensional position of each work (12) based on the output of the three-dimensional measuring instrument (15). And
The learning of the extraction operation is
The error is calculated based on the acquired result of the retrieval operation of the robot (14) and the learning model for learning the retrieval operation of the work (12).
The learning model is updated according to the error.
A machine learning method characterized by that. It is a machine learning method for learning the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
Observe the output data of the three-dimensional measuring instrument (15) that measures at least the three-dimensional map for each work (12), and the measurement parameters of the three-dimensional measuring instrument (15).
The result of the take-out operation of the robot (14) that takes out the work (12) by the hand unit (13) is acquired.
Upon receiving at least a three-dimensional map for each of the observed work (12) and the acquired result of the retrieval operation of the robot (14), the retrieval operation of the work (12) is learned.
The observation of the output data of the three-dimensional measuring instrument (15) also observes the three-dimensional position data obtained by calculating the three-dimensional position of each work (12) based on the output of the three-dimensional measuring instrument (15). And
The learning of the extraction operation is
The reward is calculated based on the result of determining the success or failure of taking out the work, which is the result of the taken-out operation of the robot (14) acquired.
The value function that determines the value of the retrieval operation of the work (12) is updated according to the reward.
A machine learning method characterized by that. It is a machine learning method for learning the operation of a robot (14) that takes out the work (12) by a hand unit (13) from a plurality of randomly placed works (12) including a state of being piled up in pieces.
Observe the output data of the three-dimensional measuring instrument (15) that measures at least the three-dimensional map for each work (12), and the measurement parameters of the three-dimensional measuring instrument (15).
The result of the take-out operation of the robot (14) that takes out the work (12) by the hand unit (13) is acquired.
Upon receiving at least a three-dimensional map for each of the observed work (12) and the acquired result of the retrieval operation of the robot (14), the retrieval operation of the work (12) is learned.
The observation of the output data of the three-dimensional measuring instrument (15) also observes the three-dimensional position data obtained by calculating the three-dimensional position of each work (12) based on the output of the three-dimensional measuring instrument (15). And
The learning of the extraction operation is
The error is calculated based on the acquired result of the retrieval operation of the robot (14) and the learning model for learning the retrieval operation of the work (12).
The learning model is updated according to the error.
A machine learning method characterized by that.