JP2017030135A - Machine learning apparatus, robot system, and machine learning method for learning workpiece take-out motion - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a machine learning apparatus, a robot system, and a machine learning method capable of selecting an optimum motion of a robot without a human operator when the robot takes out workpieces placed in a disorderly fashion including a bulk state.SOLUTION: A machine learning apparatus learning a motion of a robot 14 that takes out a workpiece 12 by a hand section 13 from a plurality of workpieces 12 placed in a disorderly fashion including a bulk state, comprises: a state variable observation unit 21 observing a state variable of the robot including output data from a three-dimensional measuring tool 15 that acquires a three-dimensional map for each of the workpieces; a motion result acquisition unit 26 acquiring a result of a take-out motion of the robot that takes out the workpiece by the hand section; and a learning unit 22 receiving an output from the state variable observation unit and an output from the motion result acquisition unit, and learning a manipulated variable including command data for commanding the robot to make the take-out motion of the workpiece to be associated with the state variable of the robot and the result of the take-out motion.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a machine learning apparatus, a robot system, and a machine learning method for learning a picking-up operation of a randomly placed work including a state in which they are stacked.

  Conventionally, for example, a robot system is known in which a workpiece stacked in a basket-like box is gripped and transported by a robot hand (see, for example, Patent Documents 1 and 2). In such a robot system, for example, position information of a plurality of workpieces is acquired using a three-dimensional measuring device installed above a cage-like box, and workpieces are moved one by one based on the position information. It is taken out by the hand part.

Japanese Patent No. 5642738 Japanese Patent No. 5670397

  However, in the above-described conventional robot system, for example, how to extract a workpiece to be extracted from a distance image of a plurality of workpieces measured by a three-dimensional measuring instrument, and which position to extract the workpiece in advance. It is necessary to set to. In addition, it is necessary to program in advance how to operate the hand portion of the robot when the workpiece is taken out. Specifically, for example, it is necessary for a human to teach a robot to take out a workpiece using a teaching pendant.

  Therefore, if the setting for extracting the workpiece to be taken out from the distance images of a plurality of workpieces is not appropriate, or if the robot operation program is not properly created, the success rate when the robot takes out and transports the workpiece decreases. In order to increase the success rate, it is necessary for humans to improve the work detection setting and the robot operation program while searching for the optimal operation of the robot through trial and error.

  Accordingly, an object of the present invention is to provide a machine learning device capable of learning the optimum operation of a robot when taking out a randomly placed workpiece, including a stacked state, in view of the above-described situation without human intervention. It is to provide a robot system and a machine learning method.

  According to the first embodiment of the present invention, there is provided a machine learning device that learns the operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces, including a state in which the workpieces are stacked. A state quantity observing unit for observing the state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of the robot taking out the work by the hand unit. An operation result acquisition unit, an output from the state quantity observation unit and an output from the operation result acquisition unit, and an operation amount including command data for instructing the robot to take out the workpiece, A machine learning device comprising: a learning unit that learns in association with a state quantity and a result of the extraction operation. The machine learning device preferably further includes a decision making unit that determines the command data to be commanded to the robot with reference to the operation amount learned by the learning unit.

  According to a second embodiment of the present invention, there is provided a machine learning device that learns the operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces, including a stacked state. A state quantity observing unit for observing the state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of the robot taking out the work by the hand unit. An operation result acquisition unit, an output from the state quantity observation unit and an output from the operation result acquisition unit are received, and an operation amount including a measurement parameter of the three-dimensional measuring instrument is determined as the state amount and the extraction operation of the robot. And a learning unit that learns in association with the result of the above. It is preferable that the machine learning device further includes a decision making unit that determines the measurement parameter of the three-dimensional measuring device with reference to the operation amount learned by the learning unit.

  The state quantity observation unit can also observe a state quantity of the robot including output data of a coordinate calculation unit that calculates a three-dimensional position for each workpiece based on an output of the three-dimensional measuring instrument. The coordinate calculation unit may further calculate a posture for each workpiece and output data of the calculated three-dimensional position and posture for each workpiece. The operation result acquisition unit can use output data of the three-dimensional measuring instrument. The machine learning device further includes a preprocessing unit that processes output data of the three-dimensional measuring instrument before input to the state quantity observation unit, and the state quantity observation unit receives output data of the preprocessing unit. It is preferably received as a state quantity of the robot. The said pre-processing part can arrange | equalize the direction and height for every said workpiece | work in the output data of the said three-dimensional measuring device uniformly. The operation result acquisition unit can acquire at least one of success or failure of the removal of the workpiece, a damaged state of the workpiece, and an achievement level when the removed workpiece is passed to a subsequent process.

  The learning unit has a reward calculation unit that calculates a reward based on an output of the operation result acquisition unit, and a value function that determines a value of the picking-up operation of the work, and updates the value function according to the reward And a value function updating unit to perform. The learning unit includes a learning model that learns the take-out operation of the workpiece, an error calculation unit that calculates an error based on an output of the operation result acquisition unit, and an output of the learning model, and the error And a learning model updating unit that updates the learning model accordingly. The machine learning device preferably has a neural network.

  According to a third embodiment of the present invention, there is provided a machine learning device that learns an operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces including a state in which the workpieces are piled up. A state quantity observing unit for observing the state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of the robot taking out the work by the hand unit. An operation result acquisition unit, an output from the state quantity observation unit and an output from the operation result acquisition unit, and an operation amount including command data for instructing the robot to take out the workpiece, And a learning unit that learns in association with a state quantity and a result of the take-out operation. , And the robot, and the three-dimensional measuring instrument, a robot system comprising a control device, the controlling the robot and the three-dimensional measuring instrument, respectively is provided.

  According to a fourth embodiment of the present invention, there is provided a machine learning device that learns the operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces, including a stacked state. A state quantity observing unit for observing the state quantity of the robot including output data of a three-dimensional measuring instrument for measuring a three-dimensional map for each work, and a result of the robot taking out the work by the hand unit. An operation result acquisition unit, an output from the state quantity observation unit and an output from the operation result acquisition unit are received, and an operation amount including a measurement parameter of the three-dimensional measuring instrument is determined as the state amount and the extraction operation of the robot. A learning unit comprising: a learning unit that learns in association with a result of the robot, wherein the robot and the three-dimensional And measuring device, a robot system and a control device for controlling each said robot and said three-dimensional measuring device is provided.

  The robot system includes a plurality of the robots, the machine learning device is provided for each of the robots, and the plurality of machine learning devices provided in the plurality of robots exchange data with each other via a communication medium. Is preferably shared or exchanged. The machine learning device may exist on a cloud server.

  According to a fifth embodiment of the present invention, there is provided a machine learning method for learning an operation of a robot that takes out the workpiece by a hand unit from a plurality of randomly placed workpieces including a state in which the workpieces are piled up. Observe the state quantity of the robot including output data of a three-dimensional measuring instrument that measures a three-dimensional map for each work, obtain the result of the robot's take-out operation to take out the work by the hand unit, and observe the state quantity The operation amount including the command data for receiving the output from the unit and the output from the operation result acquisition unit and instructing the robot to take out the workpiece is related to the state quantity of the robot and the result of the extraction operation. A machine learning method is provided.

  According to the machine learning device, the robot system, and the machine learning method according to the present invention, it is possible to learn the optimal operation of the robot when taking out a randomly placed workpiece including a stacked state without human intervention. There is an effect.

FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. FIG. 2 is a diagram schematically illustrating a neuron model. FIG. 3 is a diagram schematically showing a three-layer neural network configured by combining the neurons shown in FIG. FIG. 4 is a flowchart showing an example of the operation of the machine learning apparatus shown in FIG. FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention. FIG. 6 is a diagram for explaining an example of processing of the preprocessing unit in the robot system shown in FIG. FIG. 7 is a block diagram showing a modification of the robot system shown in FIG.

  Hereinafter, embodiments of a machine learning device, a robot system, and a machine learning method according to the present invention will be described in detail with reference to the accompanying drawings. Here, in each drawing, the same reference numeral is given to the same member. Moreover, what attached | subjected the same referential mark in a different drawing shall mean that it is a component which has the same function. In order to facilitate understanding, the scales of these drawings are appropriately changed.

  FIG. 1 is a block diagram showing a conceptual configuration of a robot system according to an embodiment of the present invention. The robot system 10 of this embodiment includes a robot 14 to which a hand unit 13 that holds a workpiece 12 stacked in a basket-like box 11 is attached, and a three-dimensional measuring instrument that measures a three-dimensional map of the surface of the workpiece 12. 15, a control device 16 that controls the robot 14 and the three-dimensional measuring instrument 15, a coordinate calculation unit 19, and a machine learning device 20.

  Here, the machine learning device 20 includes a state quantity observation unit 21, an operation result acquisition unit 26, a learning unit 22, and a decision determination unit 25. As will be described in detail later, the machine learning device 20 learns and outputs operation data such as command data for instructing the robot 14 to take out the workpiece 12 or measurement parameters of the three-dimensional measuring instrument 15.

  The robot 14 is, for example, a 6-axis articulated robot, and the drive axes of the robot 14 and the hand unit 13 are controlled by the control device 16. The robot 14 is used to take out the workpieces 12 one by one from the box 11 installed at a predetermined position and sequentially move them to a designated place, for example, a conveyor or a work table (not shown).

  By the way, when the work 12 stacked in bulk is taken out from the box 11, the hand unit 13 or the work 12 may collide with or come into contact with the wall of the box 11. Alternatively, the hand unit 13 or the work 12 may be caught by another work 12. In such a case, a function for detecting the force acting on the hand unit 13 is required so that an overload applied to the robot 14 can be immediately avoided. Therefore, a six-axis force sensor 17 is provided between the tip of the arm part of the robot 14 and the hand part 13. The robot system 10 of this embodiment also has a function of estimating the force acting on the hand unit 13 based on the current value of a motor (not shown) that drives the drive shaft of each joint unit of the robot 14. .

  Furthermore, since the force sensor 17 can detect the force acting on the hand portion 13, it can also determine whether or not the hand portion 13 is actually gripping the workpiece 12. That is, when the hand unit 13 grips the workpiece 12, the weight of the workpiece 12 acts on the hand unit 13, and therefore, after the workpiece 12 is taken out, the detection value of the force sensor 17 exceeds a predetermined threshold value. Then, it can be determined that the hand unit 13 is holding the workpiece 12. In addition, about the judgment whether the hand part 13 is holding the workpiece | work 12, about the imaging | photography data of the camera used for the three-dimensional measuring device 15, the photoelectric sensor etc. which are not shown attached to the hand part 13, etc., for example It can also be judged from the output. Further, the determination may be made based on the data of the pressure gauge of the suction type hand described later.

  Here, as long as the hand part 13 can hold | maintain the workpiece | work 12, it may have various forms. For example, the hand unit 13 is configured to grip the workpiece 12 by opening or closing two or more claw units, or includes an electromagnet or a negative pressure generator that generates an attractive force with respect to the workpiece 12. May be. That is, in FIG. 1, the hand portion 13 is depicted as holding a workpiece by two claw portions, but it is needless to say that this is not a limitation.

  The three-dimensional measuring instrument 15 is provided at a predetermined position above the plurality of workpieces 12 by the support unit 18 in order to measure the plurality of workpieces 12. As the three-dimensional measuring instrument 15, for example, a three-dimensional visual sensor that acquires three-dimensional position information by performing image processing on the image data of the workpiece 12 photographed from two cameras (not shown) may be used. it can. Specifically, by applying the triangulation measurement method, light cutting method, time-of-flight method, depth from defocus method, or a combination of these methods, a 3D map 12 surface positions) are measured.

  The coordinate calculation unit 19 calculates (measures) the positions of the surfaces of the plurality of workpieces 12 stacked in bulk, using the three-dimensional map obtained by the three-dimensional measuring instrument 15 as an input. That is, using the output of the three-dimensional measuring instrument 15, three-dimensional position data (x, y, z) for each workpiece 12, or three-dimensional position data (x, y, z) and posture data (w , P, r). Here, the state quantity observation unit 21 receives both the three-dimensional map from the three-dimensional measuring device 15 and the position data (posture data) from the coordinate calculation unit 19 and observes the state quantity of the robot 14. For example, only the three-dimensional map from the three-dimensional measuring instrument 15 can be received and the state quantity of the robot 14 can be observed. Further, as described later with reference to FIG. 5, a pre-processing unit 50 is added, and this pre-processing unit 50 performs the tertiary processing from the three-dimensional measuring instrument 15 before input to the state quantity observation unit 21. It is also possible to process (pre-process) the original map and input it to the state quantity observation unit 21.

  It is assumed that the correlation position between the robot 14 and the three-dimensional measuring instrument 15 is determined in advance by calibration. The three-dimensional measuring instrument 15 of the present invention can use a laser distance measuring instrument instead of the three-dimensional visual sensor. That is, by measuring the distance from the position where the three-dimensional measuring instrument 15 is installed to the surface of each workpiece 12 by laser scanning, or by using various sensors such as a monocular camera and a tactile sensor, a plurality of stacked units are collected. The three-dimensional position data and posture (x, y, z, w, p, r) of the workpiece 12 may be acquired.

  That is, in the present invention, for example, any three-dimensional measuring device 15 to which any three-dimensional measuring method is applied is applicable as long as data (x, y, z, w, p, r) of each workpiece 12 can be acquired. be able to. The manner in which the three-dimensional measuring instrument 15 is installed is not particularly limited, and may be fixed to a floor, a wall, or the like, or may be attached to an arm portion of the robot 14 or the like.

  The three-dimensional measuring instrument 15 acquires a three-dimensional map of the plurality of workpieces 12 stacked in the box 11 according to a command from the control device 16, and the coordinate calculation unit 19 uses a plurality of the three-dimensional maps based on the three-dimensional map. Data of the three-dimensional position (posture) of the workpiece 12 is acquired (calculated), and the data is output to the control device 16 and a state quantity observation unit 21 and an operation result acquisition unit 26 of the machine learning device 20 described later. It has become. In particular, in the coordinate calculation unit 19, for example, the boundary between one workpiece 12 and another workpiece 12 or the boundary between the workpiece 12 and the box 11 is estimated based on the image data of a plurality of photographed workpieces 12. The data of the three-dimensional position for each workpiece 12 is acquired.

  The three-dimensional position data for each work 12 is obtained by, for example, estimating the existence position and the holdable position of each work 12 from the positions of a plurality of points on the surface of the plurality of works 12 stacked in bulk. Points to the generated data. Of course, the three-dimensional position data for each workpiece 12 may include data on the posture of the workpiece 12.

  Furthermore, the acquisition of the three-dimensional position and orientation data for each workpiece 12 in the coordinate calculation unit 19 includes using a machine learning method. For example, it is also possible to apply object recognition or angle estimation from an input image or a laser distance measuring device using a method such as supervised learning described later.

  When the data of the three-dimensional position for each workpiece 12 is input from the three-dimensional measuring instrument 15 to the control device 16 via the coordinate calculation unit 19, the control device 16 takes out a certain workpiece 12 from the box 11. 13 operations are controlled. At this time, on the basis of the command value (operation amount) corresponding to the optimum position, posture and take-out direction of the hand unit 13 obtained by the machine learning device 20 described later, the motors of the axes of the hand unit 13 and the robot 14 ( (Not shown) is driven.

  In addition, the machine learning device 20 uses a camera imaging condition variable used for the three-dimensional measuring instrument 15 (measurement parameters of the three-dimensional measuring instrument 15; for example, exposure time adjusted at the time of shooting using an exposure meter, The illuminance of the illumination system that illuminates the object is learned, and the three-dimensional measuring instrument 15 can be controlled via the control device 16 based on the learned measurement parameter operation amount. Here, the variable of the position / posture estimation condition used by the three-dimensional measuring instrument 15 to estimate the existence position / posture of each work 12 and the holdable position / posture from the measured positions of the plurality of works 12 is as follows. , It may be included in the output data of the three-dimensional measuring instrument 15 described above.

  Furthermore, the output data from the three-dimensional measuring instrument 15 is processed in advance by a preprocessing unit 50 or the like which will be described in detail later with reference to FIG. 5, and the processed data (image data) is processed by the state quantity observation unit 21. As described above, it is also possible to give to the above. The operation result acquisition unit 26 can acquire, for example, the result of taking out the workpiece 12 by the hand unit 13 of the robot 14 from the output data from the three-dimensional measuring instrument 15 (output data of the coordinate calculation unit 19). In addition, for example, the degree of achievement when the taken-out workpiece 12 is passed to the subsequent process, and the operation result such as whether there is no state change such as breakage of the taken-out workpiece 12 are obtained by other means (for example, Needless to say, it can also be obtained via a camera, sensor, etc. provided in a post-process. In the above, the state quantity observation unit 21 and the operation result acquisition unit 26 are functional blocks, and it is needless to say that both functions can be achieved by one block.

  Next, the machine learning device 20 shown in FIG. 1 will be described in detail. The machine learning device 20 extracts useful rules, knowledge expressions, judgment criteria, and the like from the set of data input to the device by analysis, outputs the judgment results, and learns knowledge (machine learning). ). There are various machine learning methods, but they can be roughly classified into “supervised learning”, “unsupervised learning”, and “reinforcement learning”. Furthermore, in order to realize these methods, there is a method called “deep learning” that learns the extraction of the feature quantity itself. These machine learnings (machine learning device 20) may use general-purpose computers or processors. However, when GPGPU (General-Purpose computing on Graphics Processing Units) or a large-scale PC cluster is applied, the machine learning is faster. Can be processed.

  First, supervised learning is a model in which a large number of data sets of certain inputs and results (labels) are given to the machine learning device 20 to learn features in those data sets and to estimate the results from the inputs. That is, the relationship is acquired inductively. When this supervised learning is applied to the present embodiment, it can be used, for example, as a part for estimating a work position from a sensor input or a part for estimating an acquisition success probability for a work candidate. For example, it can be realized using an algorithm such as a neural network described later.

  In addition, unsupervised learning means that only a large amount of input data is given to the learning device to learn how the input data is distributed. This is a technique for learning with a device that performs compression, classification, shaping, and the like. For example, features in those data sets can be clustered among similar people. By using this result, output prediction can be realized by assigning outputs so as to optimize some of them by providing some criteria.

  In addition, as an intermediate problem setting between unsupervised learning and supervised learning, there is what is called semi-supervised learning. For example, only a part of input and output data sets exist, and other than that, This corresponds to the case of input-only data. In this embodiment, it is possible to efficiently perform learning by using unsupervised learning (data such as image data and simulation data) that can be acquired without actually moving the robot. .

Next, reinforcement learning will be described. First, consider the following as a problem setting for reinforcement learning.
-The robot observes the state of the environment and decides the action.
・ Environment changes according to some rules, and your actions may change the environment.
-Every time you act, a reward signal comes back.
• What we want to maximize is the sum of future (discounted) rewards.
・ Learning starts from a state of not knowing the consequences of the action at all or knowing incompletely. That is, the robot can obtain the result as data only after actually acting. In other words, it is necessary to search for the optimum action through trial and error.
-Learning can be started from a good starting point with the state of prior learning (a method such as supervised learning or reverse reinforcement learning described above) as an initial state so as to imitate human movement.

  Reinforcement learning is not only judgment and classification, but also learning behavior, learning appropriate behavior based on the interaction of behavior on the environment, that is, maximizing the reward that can be obtained in the future. Learn how to learn. This indicates that, in the present embodiment, for example, an action that affects the future, such as breaking the mountain of the work 12 and making it easier to take the work 12 in the future, can be obtained. Hereinafter, as an example, the description will be continued in the case of Q learning, but is not limited to Q learning.

  Q learning is a method of learning a value Q (s, a) for selecting an action a under a certain environmental state s. That is, in a certain state s, the action a having the highest value Q (s, a) may be selected as the optimum action. However, at first, the correct value Q (s, a) is not known at all for the combination of the state s and the action a. Therefore, the agent (action subject) selects various actions a under a certain state s, and a reward is given to the action a at that time. Thereby, the agent learns the selection of a better action, that is, the correct value Q (s, a).

The results of behavioral, we want to maximize the sum of the rewards future, finally Q (s, a) = E aims to _{^{[Σ (γ t) r t}} ] become so. Here E [] denotes the expected value, t is the time, parameter γ is called the discount rate to be described later, is r _t compensation at time t, sigma is the sum by the time t. The expected value in this equation is assumed when the state changes according to the optimum behavior, and since it is not known, it is learned while searching. Such an update formula of the value Q (s, a) can be expressed by the following formula (1), for example.

In the above formula (1), s _t represents the state of the environment at time t, a _t represents the action at time t. By the action a _t, the state changes to s _{t + 1.} r _{t + 1} represents a reward obtained by the change of the state. The term with max is a value obtained by multiplying the Q value when the action a having the highest Q value known at that time is selected under the state s _{t + 1} by γ. Here, γ is a parameter of 0 <γ ≦ 1, and is called a discount rate. In addition, α is a learning coefficient and is in a range of 0 <α ≦ 1.

The above-mentioned formula (1) as a result of the trial a _t, based on the reward r _{t + 1} came back, represents a method for updating the evaluation value Q of the action a _t in state _{s t (s t, a t} ) ing. That is, the evaluation value Q (s _{t + 1} , max) of the best action max a in the next state by the reward r _{t + 1} and the action a, rather than the evaluation value Q (s _t , a _t ) of the action a in the state s. If the sum of a _{t + 1} ) is larger, Q (s _t , a _t ) is increased, and if it is smaller, Q (s _t , a _t ) is decreased. That is, the value of a certain action in a certain state is brought close to the reward that immediately returns as a result and the value of the best action in the next state by that action.

  Here, the expression method of Q (s, a) on the computer includes a method of holding the values as a table for all the state action pairs (s, a), and Q (s, a). There is a method to prepare a function that approximates. In the latter method, the above-described equation (1) can be realized by adjusting the parameters of the approximate function by a method such as the probability gradient descent method. Note that a neural network described later can be used as the approximate function.

  A neural network can be used as a learning model for supervised learning, unsupervised learning, or an approximation algorithm for a value function in reinforcement learning. FIG. 2 is a diagram schematically showing a neuron model, and FIG. 3 is a diagram schematically showing a three-layer neural network configured by combining the neurons shown in FIG. That is, the neural network includes, for example, an arithmetic device that simulates a neuron model as shown in FIG.

As shown in FIG. 2, the neuron outputs an output (result) y for a plurality of inputs x (in FIG. 2, as an example, inputs x1 to x3). Each input x (x1, x2, x3) is multiplied by a weight w (w1, w2, w3) corresponding to this input x. Thereby, the neuron outputs a result y expressed by the following equation (2). Note that the input x, the result y, and the weight w are all vectors. In the following equation (2), θ is a bias, and f _k is an activation function.

  With reference to FIG. 3, a three-layer neural network configured by combining the neurons shown in FIG. 2 will be described. As shown in FIG. 3, a plurality of inputs x (here, as an example, inputs x1 to x3) are inputted from the left side of the neural network, and results y (here, as an example, results y1 to y3 are taken as examples). ) Is output. Specifically, the inputs x1, x2, and x3 are input with corresponding weights applied to each of the three neurons N11 to N13. The weights applied to these inputs are collectively labeled w1.

  The neurons N11 to N13 output z11 to z13, respectively. In FIG. 3, these z11 to z13 are collectively described as a feature vector z1, and can be regarded as a vector obtained by extracting the feature quantity of the input vector. The feature vector z1 is a feature vector between the weight w1 and the weight w2. z11 to z13 are inputted to each of the two neurons N21 and N22 with corresponding weights multiplied. The weights applied to these feature vectors are collectively labeled w2.

  The neurons N21 and N22 output z21 and z22, respectively. In FIG. 3, these z21 and z22 are collectively denoted as a feature vector z2. The feature vector z2 is a feature vector between the weight w2 and the weight w3. z21 and z22 are input with corresponding weights applied to each of the three neurons N31 to N33. The weights applied to these feature vectors are collectively labeled w3.

  Finally, the neurons N31 to N33 output the results y1 to y3, respectively. The operation of the neural network includes a learning mode and a value prediction mode. For example, in the learning mode, the weight W is learned using the learning data set, and the behavior of the robot is determined in the prediction mode using the parameters. For convenience, the word “prediction” is used, but it goes without saying that various tasks such as detection, classification, and inference are possible.

  Here, it is possible to immediately learn the data obtained by actually moving the robot in the prediction mode and reflect it in the next action (online learning). From then on, the detection mode can be performed with the parameters (batch learning). Alternatively, it is also possible to sandwich the learning mode every time data is accumulated to some extent.

  Further, the weights w1 to w3 can be learned by the error back propagation method (error reverse propagation method: backpropagation). The error information enters from the right side and flows to the left side. The error back-propagation method is a method of adjusting (learning) the weight of each neuron so as to reduce the difference between the output y when the input x is input and the true output y (teacher).

  Such a neural network can have more layers than three layers (referred to as deep learning). It is also possible to automatically acquire an arithmetic unit that performs input feature extraction step by step and regresses the result from only teacher data.

  Therefore, the machine learning device 20 of the present embodiment performs the above-described Q learning, as shown in FIG. 1, the state quantity observation unit 21, the operation result acquisition unit 26, the learning unit 22, and the decision making unit. 25. However, as described above, the machine learning method applied to the present invention is not limited to the Q learning. That is, various methods such as “supervised learning”, “unsupervised learning”, “semi-supervised learning”, “reinforcement learning”, and the like that can be used in the machine learning apparatus are applicable. These machine learning (machine learning device 20) may use a general-purpose computer or processor, but can be processed at higher speed by applying GPGPU, a large-scale PC cluster, or the like.

  That is, according to the present embodiment, a machine learning device that learns the operation of a robot 14 that takes out a workpiece 12 by a hand unit 13 from a plurality of randomly placed workpieces 12 including a stacked state. The state of the robot 14 including the output data of the three-dimensional measuring device 15 that measures the three-dimensional position (x, y, z) every twelve or the three-dimensional position and posture (x, y, z, w, p, r). A state quantity observing unit 21 for observing the quantity, an operation result obtaining unit 26 for obtaining the result of the take-out operation of the robot 14 for taking out the workpiece 12 by the hand unit 13, and an output from the state quantity observing unit 21 and the operation result obtaining unit 26. The operation amount including the command data for receiving the output from the robot 14 and instructing the robot 14 to take out the workpiece 12 is related to the state quantity of the robot 14 and the result of the take-out operation. Comprising a learning unit 22 for learning and Te.

  Note that the state quantity observed by the state quantity observation unit 21 may include, for example, state variables for setting the position, posture, and take-out direction of the hand unit 13 when a certain work 12 is taken out from the box 11. In addition, the learned operation amount includes, for example, command values such as torque, speed, and rotational position given from the control device 16 to each drive shaft of the robot 14 and the hand unit 13 when the workpiece 12 is taken out of the box 11. May be.

  When the learning unit 22 takes out one of the plurality of stacked workpieces 12, the learning unit 22 learns by associating the state variable with the result of the workpiece 12 take-out operation (output of the operation result acquisition unit 26). . That is, the control device 16 sets the output data of the three-dimensional measuring instrument 15 (coordinate calculation unit 19) and the command data of the hand unit 13 at random, or sets them randomly based on a predetermined rule. The operation of taking out the workpiece 12 by the unit 13 is performed. Here, as said predetermined rule, there exists a thing taken out in order from the workpiece | work with a high height (z) direction among the some workpiece | work 12 piled up in bulk, for example. Thereby, the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 correspond to the action of taking out a certain workpiece. Then, success and failure of taking out the workpiece 12 occur, and each time such success and failure occur, the learning unit 22 is a state variable composed of the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13. Will be evaluated.

  Further, the learning unit 22 stores the output data of the three-dimensional measuring instrument 15 when the workpiece 12 is taken out, the command data of the hand unit 13 and the evaluation of the result of the taking-out operation of the workpiece 12 in association with each other. Examples of the failure include a case where the hand unit 13 cannot grip the workpiece 12 or a case where the workpiece 12 collides with or contacts the wall of the box 11 even if the workpiece 12 can be gripped. Further, whether or not the workpiece 12 is taken out is determined based on the detection value of the force sensor 17 and the image data obtained by the three-dimensional measuring instrument. Here, the machine learning device 20 can also perform learning by using a part of the command data of the hand unit 13 output from the control device 16, for example.

  Here, the learning unit 22 of the present embodiment preferably includes a reward calculation unit 23 and a value function update unit 24. For example, the reward calculation unit 23 calculates a reward, for example, a score based on the success or failure of taking out the workpiece 12 caused by the state variable. The reward is increased for the successful removal of the workpiece 12, and the reward is decreased for the failure to extract the workpiece 12. Further, the reward may be calculated based on the number of successful removal of the workpiece 12 within a predetermined time. Furthermore, when calculating the reward, for example, the reward is determined according to each stage of the removal of the workpiece 12 such as successful gripping by the hand unit 13, successful transport by the hand unit 13, and successful placement operation of the workpiece 12. You may calculate.

  And the value function update part 24 has a value function which determines the value of the taking-out operation | work of the workpiece | work 12, and updates a value function according to said reward. For updating the value function, the update formula of the value Q (s, a) as described above is used. Furthermore, it is preferable to create an action value table at the time of this update. The action value table here is a value function updated according to the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 when the work 12 is taken out, and the take-out result of the work 12 at that time (that is, The evaluation value is recorded in association with each other.

  Note that a function approximated using the above-described neural network can also be used as this action value table, which is particularly effective when the amount of information of the state s is enormous, such as image data. The value function is not limited to one type. For example, a value function for evaluating the success or failure of gripping the workpiece 12 by the hand unit 13 or a value function for evaluating the time (cycle time) required to grip and transport the workpiece 12 by the hand unit 13 can be considered.

  Furthermore, as the above value function, a value function for evaluating the interference between the box 11 and the hand unit 13 or the work 12 at the time of taking out the work may be used. In order to calculate a reward used for updating the value function, the state quantity observation unit 21 preferably observes a force acting on the hand unit 13, for example, a value detected by the force sensor 17. If the amount of change in the force detected by the force sensor 17 exceeds a predetermined threshold, it can be estimated that the interference has occurred. Therefore, the reward in that case is set to a negative value, for example, and the value defined by the value function is low. It is preferable to do so.

  Moreover, according to this embodiment, it is also possible to learn the measurement parameter of the three-dimensional measuring instrument 15 as the operation amount. That is, according to the present embodiment, a machine learning device that learns the operation of a robot 14 that takes out a workpiece 12 by a hand unit 13 from a plurality of randomly placed workpieces 12 including a stacked state. The state of the robot 14 including the output data of the three-dimensional measuring device 15 that measures the three-dimensional position (x, y, z) every twelve or the three-dimensional position and posture (x, y, z, w, p, r). A state quantity observing unit 21 for observing the quantity, an operation result obtaining unit 26 for obtaining the result of the take-out operation of the robot 14 for taking out the workpiece 12 by the hand unit 13, and an output from the state quantity observing unit 21 and the operation result obtaining unit 26. A learning unit 22 that receives an output from the robot and learns an operation amount including a measurement parameter of the three-dimensional measuring instrument 15 in association with a state amount of the robot 14 and a result of the extraction operation; Provided.

  Furthermore, in the robot system 10 of the present embodiment, an automatic hand exchange device (not shown) that exchanges the hand unit 13 attached to the robot 14 with another type of hand unit 13 may be provided. In that case, the value function updating unit 24 preferably has the value function for each hand unit 13 having a different form, and updates the value function of the exchanged hand unit 13 according to the reward. Thereby, since the optimal operation | movement of the hand part 13 can be learned for every several hand 13 from which a form differs, it becomes possible to make the automatic hand changer select the hand part 13 with a higher value function.

  Subsequently, the decision making unit 25 selects, for example, the output data of the three-dimensional measuring instrument 15 and the command data of the hand unit 13 corresponding to the highest evaluation value with reference to the action value table created as described above. It is preferable to do. Thereafter, the decision making unit 25 outputs optimum data of the selected hand unit 13 and the three-dimensional measuring instrument 15 to the control device 16.

  And the control apparatus 16 takes out the workpiece | work 12 by controlling the three-dimensional measuring device 15 and the robot 14, respectively, using the optimal data of the hand part 13 and the three-dimensional measuring device 15 which the learning part 22 outputs. For example, the control device 16 operates each drive shaft of the hand unit 13 or the robot 14 based on the state variables for setting the optimum position, posture, and extraction direction of the hand unit 13 obtained by the learning unit 22. Is preferred.

  The robot system 10 according to the above-described embodiment includes one machine learning device 20 for one robot 14 as shown in FIG. However, in the present invention, the number of each of the robot 14 and the machine learning device 20 is not limited to one. For example, the robot system 10 may include a plurality of robots 14, and one or more machine learning devices 20 may be provided corresponding to each robot 14. The robot system 10 preferably shares or exchanges the optimum state variables of the three-dimensional measuring instrument 15 and the hand unit 13 acquired by the machine learning device 20 of each robot 14 with a communication medium such as a network. Thereby, even if the operation rate of a certain robot 14 is lower than the operation rate of another robot 14, the optimum operation result acquired by the machine learning device 20 provided in the other robot 14 is used for the operation of the certain robot 14. be able to. Also, the learning time can be shortened by sharing the learning model among a plurality of robots, or by sharing the operation amount including the measurement parameters of the three-dimensional measuring instrument 15, the state amount of the robot 14, and the result of the extraction operation. Can do.

  Further, the machine learning device 20 may be inside the robot 14 or outside the robot 14. Or the machine learning apparatus 20 may exist in the control apparatus 16, and may exist in a cloud server (not shown).

  Further, when the robot system 10 includes a plurality of robots 14, an operation of taking out the workpiece 12 to the hand portion of another robot 14 is performed while a certain robot 14 transports the workpiece 12 gripped by the hand portion 13. It is possible to make it. And the value function update part 24 can also update a value function using the time during which the robot 14 which takes out such a workpiece | work 12 switches. Further, the machine learning device 20 has state variables of a plurality of hand models, performs a pick-up simulation with a plurality of hand models during the picking-up operation of the workpiece 12, and a plurality of hand models according to the result of the pick-up simulation. It is also possible to learn the state variable in association with the result of the workpiece 12 take-out operation.

  In the machine learning device 20 described above, the output data of the three-dimensional measuring instrument 15 when acquiring the three-dimensional map data for each workpiece 12 is transmitted from the three-dimensional measuring instrument 15 to the state quantity observation unit 21. It is like that. Since such transmission data does not always include abnormal data, the machine learning device 20 uses the abnormal data filtering function, that is, the data from the three-dimensional measuring instrument 15 as the state quantity observation unit 21. It is possible to provide a function capable of selecting whether or not to input to. Thereby, the learning unit 22 of the machine learning device 20 can efficiently learn the optimum operation of the hand unit 13 by the three-dimensional measuring instrument 15 and the robot 14.

  Furthermore, in the machine learning device 20 described above, output data from the learning unit 22 is input to the control device 16, but the output data from the learning unit 22 does not include abnormal data. Since it is not limited, a filtering function for abnormal data, that is, a function capable of selecting whether to output data from the learning unit 22 to the control device 16 may be provided. Thereby, the control device 16 can cause the robot 14 to more safely execute the optimum operation of the hand unit 13.

  The abnormal data described above can be detected by the following procedure. That is, the probability distribution of the input data is estimated, the probability distribution of the new input is derived using the probability distribution, and if the probability of occurrence is below a certain level, it is regarded as abnormal data that deviates significantly from typical behavior. Data can be detected.

  Next, an example of operation | movement of the machine learning apparatus 20 with which the robot system 10 of this embodiment is provided is demonstrated. FIG. 4 is a flowchart showing an example of the operation of the machine learning apparatus shown in FIG. As shown in FIG. 4, when the learning operation (learning process) is started in the machine learning device 20 shown in FIG. 1, the three-dimensional measuring device 15 performs three-dimensional measurement and outputs it (step S11 in FIG. 4). . That is, in step S 11, for example, a three-dimensional map (output data of the three-dimensional measuring instrument 15) for each work 12 placed in a messy state including the stacked state is acquired and output to the state quantity observation unit 21. At the same time, the coordinate calculation unit 19 receives a three-dimensional map for each workpiece 12 and calculates a three-dimensional position (x, y, z) for each workpiece 12 to calculate the state quantity observation unit 21, the operation result acquisition unit 26, and the control device 16. Output to. Here, the coordinate calculation unit 19 may calculate and output the posture (w, p, r) for each workpiece 12 from the output of the three-dimensional measuring instrument 15.

  As will be described with reference to FIG. 5, the output (three-dimensional map) of the three-dimensional measuring instrument 15 is subjected to state quantity observation via a preprocessing unit 50 that is processed before being input to the state quantity observation unit 21. It may be input to the unit 21. Further, as will be described with reference to FIG. 7, only the output of the three-dimensional measuring device 15 may be input to the state quantity observation unit 21, and only the output of the three-dimensional measuring device 15 may be connected to the preprocessing unit 50. It may be input to the state quantity observation unit 21 via this. Thus, the implementation and output of the three-dimensional measurement in step S11 can include various things.

  Specifically, in the case of FIG. 1, the state quantity observation unit 21 includes a three-dimensional map for each workpiece 12 from the three-dimensional measuring instrument 15 and a three-dimensional position (x for each workpiece 12 from the coordinate calculation unit 19. , Y, z) and state quantities (output data of the three-dimensional measuring instrument 15) such as posture (w, p, r) are observed. The operation result acquisition unit 26 acquires the result of the extraction operation of the robot 14 that extracts the workpiece 12 by the hand unit 13 based on the output data of the three-dimensional measuring instrument 15 (output data of the coordinate calculation unit 19). In addition to the output data of the three-dimensional measuring instrument, the operation result acquisition unit 26 also acquires the result of the extraction operation such as the degree of achievement when the extracted workpiece 12 is passed to the subsequent process and the damage of the extracted workpiece 12. be able to.

  Further, for example, the machine learning device 20 determines an optimal operation based on the output data of the three-dimensional measuring instrument 15 (step S12 in FIG. 4), and the control device 16 also determines the hand unit 13 (robot 14). Command data (operation amount) is output, and the workpiece 12 is taken out (step S13 in FIG. 4). Then, the workpiece removal result is acquired by the operation result acquisition unit 26 described above (step S14 in FIG. 4).

  Next, whether or not the workpiece 12 is successfully taken out is determined based on the output from the operation result acquisition unit 26 (step S15 in FIG. 4). If the workpiece 12 is successfully taken out, a positive reward is set (see FIG. 4). Step S16) If the removal of the workpiece 12 fails, a negative reward is set (step S17 in FIG. 4), and the action value table (value function) is updated (step S18 in FIG. 4).

  Here, the success / failure determination of the removal of the workpiece 12 can be made based on, for example, the output data of the three-dimensional measuring instrument 15 after the workpiece 12 is removed. Moreover, the success / failure determination of the removal of the workpiece 12 is not limited to the evaluation of the success / failure of the removal of the workpiece 12, for example, the degree of achievement when the removed workpiece 12 is passed to the subsequent process, the damage of the removed workpiece 12, etc. It may be evaluated whether there is no change in the state, or the time (cycle time) or energy (electric energy) required for gripping and transporting the workpiece 12 by the hand unit 13.

  The reward value calculation based on the success / failure determination of the removal of the workpiece 12 is performed by the reward calculation unit 23, and the behavior value table is updated by the value function updating unit 24. That is, when the learning unit 22 succeeds in picking up the workpiece 12, the learning unit 22 sets a positive reward to the reward in the above-described update formula of the value Q (s, a) (S16), and fails to take out the workpiece 12 If so, a negative reward is set as the reward in the update formula (S17). And the learning part 22 updates the action value table mentioned above whenever the workpiece | work 12 is taken out (S18). By repeating the above steps S11 to S18, the learning unit 22 continues (learns) updating of the behavior value table.

  In the above, the data input to the state quantity observation unit 21 is not limited to the output data of the three-dimensional measuring instrument 15 and may include, for example, data such as the output of other sensors. It is also possible to use part of the command data from In this way, the control device 16 causes the robot 14 to perform an operation of taking out the workpiece 12 using the command data (operation amount) output from the machine learning device 20. Note that the learning by the machine learning device 20 is not limited to the operation of taking out the workpiece 12, and for example, as described above, the measurement parameter of the three-dimensional measuring instrument 15 may be used.

  As described above, according to the robot system 10 including the machine learning device 20 according to the present embodiment, the robot 12 picks up the workpiece 12 by the hand unit 13 from the plurality of randomly placed workpieces 12 including the stacked state. 14 actions can be learned. Thereby, the robot system 10 can learn selection of the optimal operation | movement of the robot 14 which takes out the workpiece | work 12 piled up without a human intervention.

  FIG. 5 is a block diagram showing a conceptual configuration of a robot system according to another embodiment of the present invention, and shows a robot system to which supervised learning is applied. As is clear from a comparison between FIG. 5 and FIG. 1 described above, the robot system 10 ′ to which supervised learning shown in FIG. 5 is applied is different from the robot system 10 to which Q learning (reinforcement learning) shown in FIG. In addition, a data recording unit 40 with a result (label) is provided. Note that the robot system 10 ′ illustrated in FIG. 5 further includes a preprocessing unit 50 that preprocesses the output data of the three-dimensional measuring instrument 15. Needless to say, the pre-processing unit 50 can be provided for the robot system 10 shown in FIG. 1, for example.

  As shown in FIG. 5, the machine learning device 30 in the robot system 10 ′ to which supervised learning is applied includes a state quantity observation unit 31, an operation result acquisition unit 36, a learning unit 32, a decision making unit 35, Is provided. The learning unit 32 includes an error calculation unit 33 and a learning model update unit 34. In the robot system 10 ′ of the present embodiment, the machine learning device 30 learns operation amounts such as command data for instructing the robot 14 to take out the workpiece 12 or measurement parameters of the three-dimensional measuring instrument 15. Output.

  That is, in the robot system 10 ′ to which supervised learning shown in FIG. 5 is applied, the error calculation unit 33 and the learning model update unit 34 are respectively a reward calculation unit 23 in the robot system 10 to which Q learning shown in FIG. This corresponds to the value function updating unit 24. Other configurations, for example, the configurations of the three-dimensional measuring instrument 15, the control device 16, the robot 14, and the like are the same as those in FIG.

  The error calculation unit 33 calculates an error between the result (label) output from the operation result acquisition unit 36 and the output of the learning model installed in the learning unit. Here, the result (label) -attached data recording unit 40, for example, if the shape of the workpiece 12 and the processing by the robot 14 are the same, the result (label) obtained up to the day before a predetermined day for the robot 14 to perform the work. ) And data with results (label) held in the data recording unit with results (label) 40 can be provided to the error calculator 33 on the predetermined date. Alternatively, data obtained by a simulation or the like performed outside the robot system 10 ′ or data with a result (label) of another robot system is calculated using a memory card or a communication line to calculate the error of the robot system 10 ′. It can also be provided to the unit 33. Further, the data recording unit 40 with the result (label) is configured by a non-volatile memory such as a flash memory, and the data recording unit (non-volatile memory) 40 with the result (label) is built in the learning unit 32. The data with the result (label) held in the data recording unit with result (label) 40 can be used as it is by the learning unit 32.

  FIG. 6 is a diagram for explaining an example of the processing of the preprocessing unit in the robot system shown in FIG. 5. FIG. 6A shows the three-dimensional positions (a plurality of workpieces 12 stacked in the box 11 ( FIG. 6 (b) to FIG. 6 (d) perform preprocessing on the workpieces 121 to 123 in FIG. 6 (a). The example of the image data after is shown.

  Here, the workpiece 12 (121 to 123) is assumed to be a cylindrical metal part, and the hand (13) does not hold the workpiece with two claw portions, but, for example, the cylindrical workpiece 12 It is assumed that the suction pad absorbs the longitudinal center of the substrate with negative pressure. Therefore, for example, if the position of the longitudinal center portion of the workpiece 12 is known, the workpiece 12 can be taken out by moving the suction pad (13) to the position and sucking it. Also, the numerical values in FIGS. 6A to 6D are expressed in [mm] and indicate the x direction, the y direction, and the z direction, respectively. Note that, in the z direction, the height (depth) of image data obtained by imaging the box 11 in which a plurality of workpieces 12 are stacked by a three-dimensional measuring instrument 15 (for example, having two cameras) provided above. Corresponds to the direction.

  As is clear from a comparison between FIG. 6 (a) and FIGS. 6 (b) to 6 (d), as an example of the processing of the preprocessing unit 50 in the robot system 10 ′ shown in FIG. From the 15 output data (three-dimensional images), the work 12 of interest (for example, the three works 121 to 123) is rotated and processed so that the center height becomes “0”.

  That is, the output data of the three-dimensional measuring instrument 15 includes, for example, information on the three-dimensional position (x, y, z) and posture (w, p, r) of the longitudinal center portion of each workpiece 12. . At this time, as shown in FIGS. 6 (b), 6 (c) and 6 (d), the three workpieces 121, 122, 123 of interest are respectively rotated by −r and subtracted by z. Are all set to the same conditions. By performing such preprocessing, the load on the machine learning device 30 can be reduced.

  Here, the three-dimensional image shown in FIG. 6A is not the output data itself of the three-dimensional measuring instrument 15 but, for example, from an image obtained by a program that prescribes the order in which the workpieces 12 are taken out. The threshold value for selection is lowered, and this processing itself can also be performed by the preprocessing unit 50. In addition, it cannot be overemphasized that it can change variously according to various conditions as a process by such a pre-processing part 50 including the shape of the workpiece | work 12, the kind of hand 13, etc. FIG.

  As described above, the output data (three-dimensional map for each work 12) of the three-dimensional measuring instrument 15 processed by the preprocessing unit 50 before being input to the state quantity observation unit 31 is input to the state quantity observation unit 31. Will be. Referring to FIG. 5 again, the error calculation unit 33 that receives the result (label) output from the operation result acquisition unit 36, when the output of the neural network shown in FIG. If the workpiece 12 is successfully taken out, it is assumed that there is an error of -log (y), and if it is unsuccessful, an error of -log (1-y) is assumed, and this error is minimized. Process with the goal. As the input of the neural network shown in FIG. 3, for example, image data of the works 121 to 123 to be noticed after performing the preprocessing as shown in FIGS. 6B to 6D, and Data of the three-dimensional position and posture (x, y, z, w, p, r) for each of the workpieces 121 to 123 to which attention is paid is given.

  FIG. 7 is a block diagram showing a modification of the robot system shown in FIG. As is clear from comparison between FIG. 7 and FIG. 1, in the modification of the robot system 10 shown in FIG. 7, the coordinate calculation unit 19 is deleted, and the state quantity observation unit 21 is Only the map is received and the state quantity of the robot 14 is observed. It goes without saying that a configuration corresponding to the coordinate calculation unit 19 can be provided for the control device 16. Further, the configuration shown in FIG. 7 can be applied to, for example, the robot system 10 ′ to which supervised learning described with reference to FIG. 5 is applied. That is, in the robot system 10 ′ shown in FIG. 5, the preprocessing unit 50 is deleted, and the state quantity observation unit 31 receives only the three-dimensional map from the three-dimensional measuring device 15 and observes the state quantity of the robot 14. Is possible. As described above, the embodiments described above can be variously changed and modified.

  As described above in detail, according to the present embodiment, the machine learning device that can learn the optimal operation of the robot when taking out a randomly placed work, including a stacked state, without human intervention, It becomes possible to provide a robot system and a machine learning method. The machine learning devices 20 and 30 in the present invention are not limited to those using reinforcement learning (for example, Q learning) or supervised learning, and various machine learning algorithms can be applied.

  Although the embodiment has been described above, all examples and conditions described herein are described for the purpose of helping understanding of the concept of the invention applied to the invention and the technology. It is not intended to limit the scope of the invention. Nor does such a description of the specification indicate an advantage or disadvantage of the invention. Although embodiments of the invention have been described in detail, it should be understood that various changes, substitutions and modifications can be made without departing from the spirit and scope of the invention.

DESCRIPTION OF SYMBOLS 10,10 'Robot system 11 Box 12 Work 13 Hand part 14 Robot 15 Three-dimensional measuring device 16 Control apparatus 17 Force sensor 18 Support part 19 Coordinate calculation part 20, 30 Machine learning apparatus 21, 31 State quantity observation part 22, 32 Learning Unit 23 Reward calculation unit 24 Value function update unit 25, 35 Decision making unit 26, 36 Operation result acquisition unit 33 Error calculation unit 34 Learning model update unit 40 Data recording unit with result (label) 50 Pre-processing unit

With reference to FIG. 3, a three-layer neural network configured by combining the neurons shown in FIG. 2 will be described. As shown in FIG. 3, a plurality of inputs x (here, as an example, inputs x1 to x3) are inputted from the left side of the neural network, and results y (here, as an example, results y1 to y3 are taken as examples). ) Is output. Specifically, the inputs x1, x2, and x3 are input with corresponding weights applied to each of the three neurons N11 to N13. The weights applied to these inputs are collectively labeled W1 .

The neurons N11 to N13 output z11 to z13, respectively. 3, these z11~z13 are collectively is labeled as a feature vector Z 1 and can be considered to have extracts a feature quantity of the input vector vector. This feature vector Z 1 is a feature vector between the weight W 1 and the weight W 2. z11 to z13 are inputted to each of the two neurons N21 and N22 with corresponding weights multiplied. The weights applied to these feature vectors are collectively labeled W2 .

The neurons N21 and N22 output z21 and z22, respectively. 3, these z21, Z22 are collectively are labeled as feature vector Z 2. The feature vector Z 2 is a feature vector between the weight W 2 and the weight W 3. z21 and z22 are input with corresponding weights applied to each of the three neurons N31 to N33. The weights applied to these feature vectors are collectively labeled W3 .

Further, the weights W 1 to W 3 can be learned by the error back propagation method (error reverse propagation method: backpropagation). The error information enters from the right side and flows to the left side. The error back-propagation method is a method of adjusting (learning) the weight of each neuron so as to reduce the difference between the output y when the input x is input and the true output y (teacher).

According to the first configuration example of the first embodiment of the present invention, a machine learning device that learns the operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces including a state in which the workpieces are piled up. a is, acquires the state quantity monitoring unit which observations the output data of the three-dimensional measuring device for measuring at least three-dimensional map of each of the work, the result of the extraction operation of the robot for taking out the workpiece by the hand unit an operation result acquisition unit, taking receives the output from the output and the operation result acquisition unit from the state quantity monitoring unit, and a learning unit for learning the operation of taking out the work, the learning section, wherein A reward calculation unit that calculates a reward based on a determination result of success or failure of the removal of the workpiece, which is an output of the operation result acquisition unit, and a value of the extraction operation of the workpiece. It has a value function that the value function update unit for updating the value function in accordance with the compensation, the machine learning device comprising a are provided. According to the second configuration example of the first embodiment of the present invention, a machine learning device that learns the operation of a robot that picks up a workpiece by a hand unit from a plurality of randomly placed workpieces including a state in which the workpieces are piled up. A state quantity observing unit for observing output data of a three-dimensional measuring instrument that measures at least a three-dimensional map for each workpiece, and an operation for acquiring a result of an extraction operation of the robot that takes out the workpiece by the hand unit. A learning unit that receives a result acquisition unit and an output from the state quantity observation unit and an output from the operation result acquisition unit to learn the take-out operation of the workpiece; A learning model for learning the picking-up operation; a determination result of success or failure of picking up the workpiece, which is an output of the operation result acquisition unit; and the learning model An error calculator for calculating an error based on Le, a learning model updating unit for updating the learning model in response to the error, the machine learning device comprising a are provided. The machine learning apparatus further refers to the output from the learning section, provided with a decision section for determining a directive data you command the operation of taking out the workpiece to the robot are preferred.

Claims (18)

A machine learning device that learns the operation of a robot (14) that picks up the workpiece (12) by a hand unit (13) from a plurality of randomly placed workpieces (12), including a state of being stacked apart,
A state quantity observation unit (21, 31) for observing a state quantity of the robot (14) including output data of a three-dimensional measuring instrument (15) for obtaining a three-dimensional map for each work (12);
An operation result acquisition unit (26, 36) for acquiring a result of the extraction operation of the robot (14) for extracting the workpiece (12) by the hand unit (13);
Command data for receiving the output from the state quantity observation unit (21, 31) and the output from the operation result acquisition unit (26, 36) and instructing the robot (14) to take out the workpiece (12). A learning unit (22, 32) that learns an operation amount including the relationship between the state amount of the robot (14) and a result of the extraction operation,
A machine learning device characterized by that. further,
A decision making unit (25, 35) for determining the command data to be commanded to the robot (14) with reference to the operation amount learned by the learning unit (22, 32);
The machine learning device according to claim 1. A machine learning device that learns the operation of a robot (14) that picks up the workpiece (12) by a hand unit (13) from a plurality of randomly placed workpieces (12), including a state of being stacked apart,
A state quantity observation unit (21, 31) for observing a state quantity of the robot (14) including output data of a three-dimensional measuring instrument (15) for measuring a three-dimensional map for each work (12);
An operation result acquisition unit (26, 36) for acquiring a result of the extraction operation of the robot (14) for extracting the workpiece (12) by the hand unit (13);
An output from the state quantity observation unit (21, 31) and an output from the operation result acquisition unit (26, 36) are received, and an operation amount including a measurement parameter of the three-dimensional measuring device (15) is obtained as the robot. A learning unit (22, 32) that learns in association with the state quantity of (14) and the result of the extraction operation,
A machine learning device characterized by that. further,
A decision-making unit (25, 35) for determining the measurement parameter of the three-dimensional measuring device (15) with reference to the operation amount learned by the learning unit (22, 32);
The machine learning apparatus according to claim 3. The state quantity observation unit (21, 31) further outputs output data of a coordinate calculation unit (19) that calculates a three-dimensional position for each workpiece (12) based on the output of the three-dimensional measuring device (15). Observing a state quantity of the robot (14) including:
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. The coordinate calculation unit (19) further includes:
Calculating the posture of each workpiece (12) and outputting the calculated three-dimensional position and posture data for each workpiece (12);
The machine learning device according to claim 5. The operation result acquisition unit (26, 36) uses output data of the three-dimensional measuring instrument (15).
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. further,
A preprocessing unit (50) for processing output data of the three-dimensional measuring instrument (15) before input to the state quantity observation unit (21, 31);
The state quantity observation unit (21, 31) receives output data of the preprocessing unit (50) as a state quantity of the robot (14).
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. The pre-processing unit (50) uniformly aligns the direction and height of each workpiece (12) in the output data of the three-dimensional measuring instrument (15).
The machine learning device according to claim 8. The operation result acquisition unit (26, 36)
Obtaining at least one of success or failure of removal of the workpiece (12), a damaged state of the workpiece (12), and a degree of achievement when the removed workpiece (12) is passed to a subsequent process;
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. The learning unit (22)
A reward calculation unit (23) for calculating a reward based on the output of the operation result acquisition unit (26);
A value function update unit (24) having a value function for determining the value of the take-out operation of the work (12) and updating the value function according to the reward,
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. The learning unit (32) has a learning model for learning the take-out operation of the work (12),
An error calculation unit (33) that calculates an error based on the output of the operation result acquisition unit (26) and the output of the learning model;
A learning model update unit (34) that updates the learning model according to the error,
The machine learning apparatus according to claim 1, wherein the machine learning apparatus is a machine learning apparatus. The machine learning device includes:
The machine learning device according to claim 1, comprising a neural network. A robot system (10, 10 ') comprising the machine learning device (20, 30) according to any one of claims 1 to 13,
The robot (14);
The three-dimensional measuring instrument (15);
A control device (16) for controlling the robot (14) and the three-dimensional measuring device (15), respectively.
A robot system characterized by this. The robot system (10, 10 ′) includes a plurality of the robots (14),
The machine learning device (20, 30) is provided for each robot (14),
The plurality of machine learning devices (20) provided in the plurality of robots (14) are configured to share or exchange data with each other via a communication medium.
The robot system according to claim 14. The machine learning device (20, 30) exists on a cloud server,
The robot system according to claim 15. A machine learning method for learning an operation of a robot (14) that takes out the workpiece (12) by a hand unit (13) from a plurality of randomly placed workpieces (12) including a state in which the pieces are stacked.
Observing a state quantity of the robot (14) including output data of a three-dimensional measuring instrument (15) that measures a three-dimensional position of each workpiece (12);
The result of the picking-up operation of the robot (14) for picking up the work (12) by the hand part (13) is obtained,
It includes command data for receiving the observed state quantity of the robot (14) and the acquired result of the removal operation of the robot (14) and instructing the robot (14) to perform the removal operation of the workpiece (12). Learning an operation amount in association with the state amount of the robot (14) and the result of the extraction operation;
A machine learning method characterized by that. A machine learning method for learning an operation of a robot (14) that takes out the workpiece (12) by a hand unit (13) from a plurality of randomly placed workpieces (12) including a state in which the pieces are stacked.
Observing a state quantity of the robot (14) including output data of a three-dimensional measuring instrument (15) for measuring a three-dimensional map for each workpiece (12);
The result of the picking-up operation of the robot (14) for picking up the work (12) by the hand part (13) is obtained,
The observed state quantity of the robot (14) and the acquired result of the extraction operation of the robot (14) are received, and the operation amount including the measurement parameters of the three-dimensional measuring instrument (15) is obtained as the operation quantity of the robot (14). Learning in association with the state quantity of and the result of the extraction operation,
A machine learning method characterized by that.