DE102018004330A1 - Control and machine learning device - Google Patents

Abstract

A machine learning apparatus of a controller observes data of a moving speed of each motor of a robot and a moving speed adjusting amount, a target speed of a tip end of the robot, and a moving path near the tip end of the robot as state variables expressing a current state of an environment, and acquires determination data indicating a fitness determination result of the moving speed of the tip end of the robot. Then, using the observed state variables and the determined determination data, the machine learning device learns the target speed data, the movement speed data, and the movement path data in association with the adjustment amount of the moving speed of each of the motors of the robot.

Description

GENERAL PRIOR ART

Field of the invention

The present invention relates to a controller and a machine learning apparatus, and more particularly to a controller and a machine learning apparatus which optimize a teaching speed.

Description of the Prior Art

General industrial robots are driven or driven in accordance with a previously prepared operation program to go through a teach-in point previously learned by a teaching pendant or the like at a teaching speed. That is, the robots are driven at a predetermined speed along a predetermined path. For example, the laid-open discloses Japanese Patent Application No. 6-285402 As a related art that deals with the teach-in operation of a robot, a technique by which the robot is taught a sealing process (such as a teach-in point and a teach-in speed) to cause the robot to perform a seal.

According to the disclosed in the Japanese Patent Application No. 6-285402 In the disclosed technology, a rotational speed of the motor of a sealant supply pump is controlled according to a moving speed of a robot to supply a sealant to a sealant gun, an amount of the sealant to be applied per unit distance of an object is kept constant regardless of a moving speed of the robot and a film thickness of a bead is kept constant , However, a pump having such a pressure regulating function is expensive, which causes a cause for increasing the cost of an entire system. In order to reduce the cost of the entire system, it is envisaged to carry out a control such that the sealant gun provided at the tip of the robot passes through a teaching point, keeping its movement at a predetermined speed. If such a control is acceptable, the cost of the entire system can be reduced by using a low cost pump that can only control ON / OFF state. However, a trajectory or a moving speed of the tip end of the robot is determined to be encompassed with the movement of a plurality of motors, and a change in the movement of the tip end of the robot upon changing the motion of one motor depends on a moving state or an acceleration / deceleration state of another motor , Therefore, a skilled worker also has difficulty in tuning the tip end of the robot to move along a trajectory while keeping its moving speed constant, and it is necessary to repeatedly perform the adjustment by trial and error. As a result, the worker faces the problem that he must make enormous efforts to carry out the vote.

BRIEF SUMMARY OF THE INVENTION

In view of the above problem, the present invention has an object to provide a controller and a machine learning device capable of adjusting the teaching speed of the tip end of a robot to a predetermined target speed.

In order to solve the above problem, a controller according to the present invention performs machine learning of a tuning amount of a moving speed of each motor of a robot with respect to a target speed of the robot tip, a current speed of each motor of the robot, and a trajectory of the tip of the robot Robot, and performs a control such that a moving speed of the tip end of the robot corresponds to a target speed when the robot moves to a teaching position based on a result of the machine learning.

A controller according to a first embodiment of the present invention tunes the moving speed of each motor of a robot that performs the coating with a sealing material. The controller includes a machine learning device that learns a tuning amount of the moving speed of the individual motors of the robot. The machine learning device has a state observation section indicative of state variables expressing a current state of an environment, training speed adjustment amount data indicating the tuning amount of the moving speed of each of the motors of the robot, target speed data indicating a target speed of a tip end of the robot, moving speed data indicate the moving velocity of each of the motors of the robot, and observe trajectory data indicating a trajectory near the tip end of the robot, a determination data detecting section that detects determination data indicating a fitness determination result of the moving speed of the tip end of the robot, and a learning section the Target speed data, the moving speed data and the trajectory data in association with the timing of adjusting the moving speed of each of the motors of the robot using the state variables and the determination data.

The determination data may include, in addition to the fitness determination result of the moving speed of the tip end of the robot, a fitness determination result of a position of the tip end of the robot.

The learning section may include a reward calculation section that calculates a reward associated with the aptitude determination result, and a value function update section that updates a function using the reward that includes a value of the trim amount of the movement speed of each of the motors of the robot with respect to the target speed of the tip end of the robot, expressing the moving speed of each of the motors of the robot and the trajectory near the tip end of the robot.

The session may perform the calculation of the state variables and the determination data based on a multi-layered structure.

The controller may further include a decision making section that outputs a target value based on the tuning amount of the moving speed of each of the motors of the robot based on a learning result of the learning section.

The learning section may learn the tuning amount of the moving speed of each of the motors of the robot in each of a plurality of robots by using the state variables and destination data obtained for each of the plurality of robots.

The machine learning device may be present in a cloud server.

A machine learning apparatus according to a second embodiment of the present invention learns a tuning amount of the moving speed of each motor of a robot that performs the coating with a sealing material. The machine learning apparatus includes: a state observation section that expresses state variables that express a current state of an environment, a state observation section that expresses state variables of an environment, learn-in velocity adjustment amount data, and the tuning amount of the movement speed of each of the motors of the robot specify target speed data indicating a target speed of a tip end of the robot, movement speed data indicating the moving speed of each of the motors of the robot, and trajectory data indicating a trajectory near the tip end of the robot; a determination data acquiring section that acquires determination data indicating a fitness determination result of the moving speed of the tip end of the robot; and a learning section that learns the target speed data, the movement speed data and the trajectory data in association with the adjustment amount of the moving speed of each of the motors of the robot using the state variables and the determination data.

According to an embodiment of the present invention, by adjusting a learning speed of the robot based on a learning result, it is possible to keep a moving speed of the tip end of a robot constant and to keep a film thickness of a bead constant without using an expensive pump.

list of figures

• 1 FIG. 10 is a schematic hardware configuration diagram of a controller according to a first embodiment; FIG.
• 2 Fig. 10 is a schematic functional block diagram of the controller according to the first embodiment;
• 3 Fig. 10 is a schematic functional block diagram showing an embodiment of the controller;
• 4 Fig. 10 is a schematic flowchart showing the embodiment of a machine learning method;
• 5A is a diagram for describing a neuron;
• 5B is a diagram for describing a neural network;
• 6 Fig. 10 is a schematic functional block diagram of a controller according to a second embodiment;
• 7 Fig. 10 is a schematic functional block diagram showing one embodiment of a system with controls; and
• 8th Fig. 10 is a schematic functional block diagram showing another embodiment of a system with a controller.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

1 FIG. 12 is a schematic hardware configuration diagram showing a control and the essential parts of a machine tool controlled by the controller according to a first embodiment. FIG.

A controller 1 For example, it may be mounted as a controller for controlling an industrial robot (not shown) that performs coating by a sealing material or the like. A central processing unit (CPU) 11 the controller 1 According to the embodiment, a processor that is the controller 1 completely controls. The CPU 11 reads in a read-only memory (ROM) 12 stored system program via a bus 20 and controls the entire controller 1 according to the system program. A random access memory (RAM) 13 temporarily stores temporary calculation data or display data and various data or the like which is input from an operator via a hand-held programmer described later 60 be entered.

A non-volatile memory 14 is designed as a memory, its memory state, for example, by data backup or the like with a battery (not shown) even with the controller off 1 maintains. The non-volatile memory 14 saves those from the programming pendant 60 via an interface 18 input learning data, a program (not shown) via an interface, the robot controlling program, or the like. Programs or various data stored in non-volatile memory 14 may be stored in execution / use in the RAM 13 be developed. Furthermore, the ROM stores 12 beforehand various system programs (a system program comprising communication with a machine learning device 100 which will be described later) for the current processing for the control of a robot or the teaching of a teaching position or the like.

An axis control circuit 30 for controlling the axis of a joint or the like of a robot receives from the CPU 11 a motion set amount of the axis and gives a command to move the axis to a servo amplifier 40 out. The servo amplifier 40 controls a servomotor when receiving the command 50 on, which moves the axis of the robot. The servomotor 50 for the shaft includes a position / speed detecting device and outputs a position / speed feedback signal from the position / speed detecting device to the axis control circuit 30 back to perform a position / speed control. It is noted that the axis control circuit 30 , the servo amplifier 40 and the servomotor 50 in the hardware configuration diagram of 1 are individually shown, but are actually provided according to the number of axes of a robot to be controlled.

For example, in a robot with six axes, the axis control loop is for each of the six axes 30 , the servo amplifier 40 and the servomotor 50 intended.

The programming pendant 60 is a manual data input device with a display, a handle, a hardware key or the like. The programming pendant 60 receives over the interface 18 Information from the controller 1 to display these, and gives pulses, commands and various data inputs from the handle, the hardware key or the like to the CPU 11 further.

A pump 70 conveys a sealing material to a sealant gun (not shown) which is held at the tip of a robot. Based on a command from the CPU 11 via an interface 19 is the pump 70 able to turn on and off the supply of sealing material.

An interface 21 is an interface to the controller 1 and the machine learning device 100 to connect with each other. The machine learning device 100 includes a processor 101 , the whole machine learning device 100 controls, a ROM 102 storing a system program or the like, a RAM 103 which caches data in each processing associated with the machine learning, and a nonvolatile memory 104 used for storing a learning model or the like. The machine learning device 100 can any information (eg position information or speed information of the servomotor 50 , a current value and setting information on a current program, training information or the like stored in the RAM 13 or the like stored) by the controller 1 over the interface 21 can be detected. Furthermore, the controller performs 1 upon receipt of commands to control the servomotor 50 and the peripheral device of a robot used by the machine learning device 100 output, the compensation or the like of a command for controlling the robot based on a program or Einlerndaten by.

2 is a schematic functional block diagram of the controller 1 and the machine learning device 100 according to the first embodiment.

The machine learning device 100 includes software (such as a learning algorithm) and hardware (such as the processor 101 for spontaneously learning a tuning amount of a moving speed of each of the motors of a robot with respect to a target speed of the tip end of the robot, a current speed of each of the motors of the robot, and a trajectory of the tip end of the robot by so-called machine learning. One from the machine learning device 100 the controller 1 The object to be learned corresponds to a model structure expressing the correlation between a target speed of the tip end of a robot, a current rotational speed of each of the motors of the robot, a trajectory of the tip end of the robot, and a trim amount of a moving velocity of each of the motors of the robot.

As in the function block of 2 includes the machine learning device 100 the controller 1 a state observation section 106 , a determination data acquiring section 108 and a session 110 , The state observation section 106 observes state variables S expressing the current state of an environment, the teaching speed tuning amount data S1 indicative of a tuning amount of a moving speed of each motor of a robot in the control of the robot based on training data, target speed data S2 indicative of a target speed of the tip end of the robot, moving speed data S3 indicative of a moving speed of each of the motors of the robot and trajectory data S4 comprising a trajectory near the tip end of the robot. The determination data acquiring section 108 records the determination data D indicating the moving speed designation data D1 indicative of a fitness determination result of the moving speed of the tip end of the robot when tuning a learning speed of each of the motors. Using the state variables S and the determination data D the session learns 110 the target speed of the tip end of the robot, a current speed of each of the motors of the robot, and the moving path of the tip end of the robot in association with the teaching speed tuning amount data S1 ,

The state observation section 106 for example, as one of the functions of the processor 101 or as in ROM 102 stored software for operating the processor 101 be configured. Among the state variables S, the state observation section 106 can be observed, the teaching speed tuning amount data S1 are detected as a set of tuning amounts with respect to a moving speed of each motor of a robot. Here, the adjustment amount of the moving speed of each of the motors of the robot includes a direction (a positive / negative value) in which the moving speed of the motor is tuned.

As the teaching speed tuning amount data S1 For example, a tuning amount of a moving speed of each motor of a robot or a tuning amount of moving speed of each motor may be used from a simulation-simulated result reported by a skilled worker and the controller 1 was handed over to be used at the start of learning. Further, as a teaching speed tuning amount data S1 a tuning amount of one through the machine learning device 100 in the previous learning cycle based on a learning outcome of the session 110 certain speed of movement of each motor of a robot can be used when learning has progressed to a degree. In such a case, the machine learning device 100 in advance, for each learning cycle, a certain amount of tuning a moving speed of each motor of a robot in the RAM 103 Caching, so the state observation section 106 from the RAM 103 the amount of tuning by the machine learning device 100 detected in the previous learning cycle movement speed of each motor of the robot.

Furthermore, as target speed data S2 under the state variables S For example, a learning speed set by a worker and included in the training data may be used. Since a teach-in speed set by a worker is a value set by the worker as a target value, it can be used as a target speed.

As movement speed data S3 under the state variables S For example, a moving speed in the current cycle of each motor (ie, the servomotor 50 ) of a robot. The movement speed data S3 can be detected using a position and speed detecting device mounted on a motor.

As trajectory data S4 Among the state variables S, for example, a trajectory of the tip end of a robot calculated based on a learning position included in the learning data may be used. The trajectory data S4 can as Calculating serial data for each prescribed cycle of relative coordinate values obtained when a trajectory within a prescribed time since the current time is viewed from the current position of the tip end of a robot.

The determination data acquiring section 108 for example, as one of the functions of the processor 101 or as in ROM 102 stored software for operating the processor 101 be configured. As determination data D the determination data acquiring section may 108 Moving speed determining data D1 which specify a fitness determination value with respect to a moving speed of the tip end of a robot when a teaching speed of each motor is set. The determination data D1 can be obtained from a moving speed of each of the motors of the robot obtained when the state observation section 106 the movement speed data S3 observed, be calculated. The determination data D are an index expressing a result obtained when a robot among the state variables S is controlled.

Related to the learning cycle of the session 110 are the ones in the session at the same time 110 entered state variables S those based on data from the previous learning cycle where the determination data D were recorded. While, as described above, the machine learning device 100 the controller 1 which advances machine learning becomes the acquisitions of the target speed data S2 , the motion speed data S3 and the trajectory data S4 and the implementation of controlling a robot according to one based on the training speed tuning amount data S1 coordinated learning speed and the determination of the determination data D repeatedly performed in an environment.

The session 110 for example, as one of the functions of the processor 101 or as in ROM 102 stored software for operating the processor 101 be configured. According to any learning algorithm called machine learning, the session learns 110 the teaching speed tuning amount data S1 with respect to a target speed of the tip end of a robot, a moving speed of each of the motors of the robot, and a trajectory near the tip end of the robot. The session 110 can learn based on a record with the state variables described above S and the determination data D Repeat. When the cycle of learning the teaching speed tuning amount data S1 with respect to a target speed of the tip end of a robot, a moving speed of each of the motors of the robot, and a trajectory near the tip end of the robot are repeatedly executed, the target speed data becomes S2 , the movement speed data S3 and the trajectory data S4 among the state variables S from the training data or, as described above, the state of each of the motors detected in the previous learning cycle, the training speed tuning amount data S1 correspond to a determination of the moving speed of each of the motors of the robot based on previous learning results, and the determination data D correspond to a fitness determination result with respect to the (coordinated) moving speed of the tip end of the robot in the current learning cycle in a state where the learning speed is based on the learning speed tuning amount data S1 was agreed.

By repeatedly executing such a learning cycle, the session may 110 automatically identify a feature that determines the correlation between a desired speed (target speed data S2 ) of the tip end of a robot, a moving speed (moving speed data S3 ) of each of the motors of the robot and a trajectory (trajectory data S4 ) in the vicinity of the tip end of the robot and a tuning amount of the moving speed of each of the motors of the robot with respect to the state. Although the correlation between the target speed data S2 , the movement speed data S3 and the trajectory data S4 and a tuning amount of a moving speed of each motor of a robot at the beginning of a learning algorithm is substantially unknown, the learning section identifies 110 gradually a feature indicating the correlation and interprets the correlation as learning progresses. If the correlation between the target speed data S2 , the movement speed data S3 and the trajectory data S4 and a tuning amount of a moving speed of each motor is interpreted to some reliable degree, those of the learning section 110 repeatedly output learning results for selecting the action (ie, decision making) used to determine to what extent a moving speed of each of the motors of the robot with respect to a current state (ie, a target speed of the tip end of the robot, a moving speed of each of the Motors of the robot and a trajectory near the top end of the robot) is agreed. That is, when a learning algorithm has advanced, the session may 110 the correlation between a target speed of the tip end of a robot, a moving speed of each of the motors of the robot and a trajectory near the tip end of the robot and the action of determining to what extent a moving speed of each of the motors of the robot with respect to the state is tuned to an optimal solution, gradually approaching.

As described above, the session learns 110 in the machine learning device 100 the controller 1 a tuning amount of a moving speed of each motor of a robot according to a machine learning algorithm using the state variables S from the state observation section 106 be observed, and the determination data D obtained from the determination data acquiring section 108 be recorded. The state variables S are composed of the teaching speed tuning amount data S1 , the target speed data S2 , the movement speed data S3 and the trajectory data S4 which are hardly affected by a disturbance. In addition, the determination data D is detected by the detection of one in the controller 1 stored teach-in speed and one of control 1 detected movement speed of the servomotor 50 clearly calculated. Accordingly, the machine learning device 100 the controller 1 using learning outcomes of the session 110 automatically and accurately calculate a tuning amount of a moving speed of each motor of a robot according to a target speed of the tip end of the robot, a moving speed of each motor of the robot, and a moving path near the tip end of the robot without relying on calculation or estimation.

When it is possible to automatically calculate a tuning amount of a moving speed of each motor of a robot without relying on calculation or estimation, an appropriate value of the tuning amount of the moving speed of each of the motors of the robot can be obtained only by understanding a target speed (the target speed data S2 ) of the tip end of the robot, a moving speed (the moving speed data S3 ) of each of the motors of the robot and a trajectory (the trajectory data S4 ) can be quickly determined near the tip end of the robot. Accordingly, the moving speed of each motor of a robot can be efficiently tuned.

As a first modified example of the machine learning device 100 the controller 1 the determination data acquiring section may 108 as determination data D , besides movement speed determination data D1 indicative of a fitness determination value of a moving speed of the tip end of the robot, tip end position determination data D2 which indicate a fitness determination result of the position of the tip end of a robot or the like.

According to the above modified example, the machine learning device 100 further taking into account a degree of deviation of a teaching position in learning a tuning amount of a moving speed of each motor of a robot with respect to a target speed of the tip end of the robot, a moving speed of each motor of the robot and a moving track in the vicinity of the tip end of the robot.

As a second modified example of the machine learning device 100 the controller 1 can the session 110 using the determined state variables S and determination data D for each of the plurality of robots performing the same work, learning a tuning amount of the moving speed of each motor of a plurality of robots. According to the configuration, it is possible to have a set of a data set representing the state variables acquired in a certain period of time S and determination data D includes, increase. Therefore, the speed and reliability of learning a moving speed adjustment amount of each motor of a robot with a set of more diverse data as inputs can be improved.

In the machine learning device 100 with the above configuration is one of the session 110 executed learning algorithm is not particularly limited. For example, a learning algorithm known as machine learning can be used. 3 As an embodiment of FIG 1 shown control 1 a configuration that completes the session 110 which performs the reinforcing learning as an example of a learning algorithm.

The reinforcing learning is a method in which, while the current state (ie, input) of an environment in which a learning exists exists, a prescribed action (ie, an output) is performed in the current state and the cycle of giving one Reward the action repeatedly performed by trial and error to learn action (adjusting a movement speed of each motor a robot, in the case of the machine learning apparatus of the present application) to maximize the sum of the rewards as an optimal solution.

In the machine learning device 100 the in 3 shown control 1 includes the session 110 a reward calculation section 112 and a value function updating section 114 , The reward calculation section 112 calculates a reward corresponding to a fitness determination result (corresponding to the determination data used in the next learning cycle D in which the state variables S detected) of the operating state of the tip end of a robot when a learning speed of each motor based on the state variables S is agreed. The value function update section 114 updated using the calculated reward R a function Q which expresses a value of a tuning amount of a moving speed of each of the motors of the robot. The session 110 learns a tuning amount of a moving speed of each motor of a robot with respect to a target speed of the tip end of the robot, a moving speed of each of the motors of the robot, and a moving track near the tip end of the robot such that the value function updating section 114 the function Q repeatedly updated.

An example of a learning algorithm for reinforcement learning that starts from the session 110 is performed is described. The algorithm in this example is known as Q-learning and expresses a method in which a state s an action subject and possibly the action subject in the state s Action taken a be accepted as independent variables and a function Q (s, a) expressing an action value when the action a in the state s is selected, is learned. The selection of action a, where the value function Q in the state s becomes the largest, leads to an optimal solution. By starting the Q-learning in a state where the correlation between the state s and the action a is unknown, and repeatedly executing the selection of various actions a by trying in any state s, the value function becomes Q repeatedly updated to approximate an optimal solution. When an environment (ie, the state s) changes while the action a is selected in the state s, becomes a reward (ie the weighting of the action a ) r according to the change and the learning is geared to selecting an action a, by which a higher reward r is obtained. So can the value function Q be approximated to an optimal solution in a relatively short period of time.

In general, the update formula of the value function Q may be expressed as the following formula (1). In the formula (1), s _t and a _t express a state and an action at time t, respectively, and the state changes to s _{t + 1} with the action a _t . r _{t + 1} expresses a reward obtained when the state changes from s _t to s _{t + 1} . The term maxQ expresses Q in a case where an action a is executed by which the maximum value Q is reached at time t + 1 (assumed at time t). α and γ express a learning coefficient and a discounting factor, respectively, and are arbitrarily set to fall within 0 <α ≦ 1 and 0 <γ ≦ 1, respectively. $$Q\left(s_t.a_t\right)\leftarrow Q\left(s_t.a_t\right)+\alpha \left(r_{t+1}+\gamma \underset{a}{\mathrm{Max}}Q\left(s_t.a\right)-Q\left(s_t.a_t\right)\right)$$

When the session 110 performing the Q-learning corresponds to that of the state observation section 106 observed state variables S and the state data acquiring section 108 recorded determination data D the state s in this updating formula, the action of determining a tuning amount of a moving speed of each motor of a robot with respect to a current state (ie, a target speed of the tip end of the robot, a moving speed of each of the motors of the robot, and a trajectory near the tip end of the robot) Robot) corresponds to the action a in the update formula, and that by the reward calculation section 112 calculated reward R corresponds to the reward r in the update formula. Accordingly, the value function updating section updates 114 repeats the function Q which expresses a value of a tuning amount of a moving speed of each motor of a robot with respect to a current state through the Q learning, using the reward R ,

For example, when the robot is controlled according to a moving speed of each motor that is determined based on a tuning amount determined after the determination of the tuning amount of the moving speed of each of the motors of the robot, the reward calculation section may determine 112 calculated reward R be positive (plus) when a fitness determination result of the operating state of a robot is determined to be "suitable" (for example, a case where the difference between a moving speed and a target speed of the tip end of the robot falls within an allowable range), a case where the Difference between the position of the tip end of the robot and a teaching position within an allowable range or the like), or may be negative (minus) if the fitness determination result of the operating state of the robot is determined to be "inappropriate" (for example, a case where the difference between the moving speed and the target speed the tip end of the robot goes beyond the allowable range, a case where the difference between the position of the tip end of the robot and the teaching position exceeds the allowable range, or the like).

The absolute values of the positive and negative rewards R can help in determining the rewards R be the same or different. In addition, as a determination condition, a plurality of in the determination data D combined values to make a determination.

In addition, a fitness determination result of the operation of a robot may include not only "suitable" and "inappropriate" results, but also a plurality of result levels. For example, assuming a maximum value within an allowable range of the difference between a moving speed and a target speed of the tip end of a robot as G _max , the reward R = 5 is awarded if the difference G between the moving speed and the target speed of the tip end of the robot is within 0 ≤ G <G _max / 5, the reward R = 2 is awarded when the difference G falls within G _max / 5 ≤ G <G _max / 2, and the reward R = 1 is awarded when the difference G within G _max / 2 ≤ G ≤ G _max . In addition, G _max can be set relatively larger in the initial phase of learning and set to decrease with increasing learning progress.

The value function update section 114 may have an action value table in which the state variables S , the determination data D and the rewards R in association with the through the function Q expressed action values (eg numerical values). In this case, the action of updating the function is Q with the value function update section 114 synonymous with the action of updating the action value table with the value function updating section 114 , At the beginning of Q learning, the correlation between the current state of an environment and a tuning amount of a moving speed of each motor is unknown. Therefore, in the action value table, various kinds of the state variables become S , the determination data D and the rewards R in assignment to values (function Q ) of random action values. It is noted that the reward calculation section 112 the rewards R according to the determination data D can calculate immediately if the determination data D known and the values of the calculated rewards R are written to the action value table.

If Q learning using the reward R According to a fitness determination result of the operating state of a robot, the learning is aimed at the action of obtaining a higher reward R select. Then values (function Q ) of action values for an action performed in a current state is rewritten to match the action value table according to the state of an environment (ie, state variables S and the determination data D ), which changes to update when the selected action is executed in the current state. Repeating the update will cause the values (the function Q ) the action values displayed in the action value table are rewritten larger if an action is more appropriate. Thus, the relationship between a current state (a target speed of the tip end of a robot, a moving speed of each of the motors of the robot and a trajectory near the tip end of the robot) in an unknown environment and a corresponding action (matching the moving speed of each of the Motors of the robot) gradually apparent. That is, by updating the action value table, the ratio between a target speed of the tip end of a robot, a moving speed of each of the motors of the robot and a trajectory near the tip end of the robot and a timing of adjusting the moving speed of each of the motors of the robot gradually become one approximated optimal solution.

The process of the above Q learning (ie, the embodiment of a machine learning method) by the learning section 110 becomes with reference to 4 described in more detail.

First, the value function update section selects 114 at step SA01 with reference to an action value table at this time, arbitrarily selects a tuning amount of a moving speed of each motor of a robot as an action performed in a current state indicated by the state variables S indicated by the state observation section 106 to be watched. Then the value function update section imports 114 the state variable S in the state observation section 106 in step SA02 watch current state and imports the determination data D in the determination data acquiring section 108 in step SA03 recorded current state. In step SA04 then determines the value function update section 114 based on the determination data D Whether the tuning amount of the moving speed of each of the motors of the robot is suitable or not. If the reconciliation amount is appropriate, the value function update section applies 114 in step SA05 one by the reward calculation section 112 calculated positive reward Up the function's update formula Q at. Next updated in step SA06 the value function update section 114 the action value table using the state variable S and the determination data D in the current state, the reward R and a value (updated function Q ) of an action value. When in step SA04 it is determined that the adjustment amount of the moving speed of each of the motors of the robot is inappropriate, the value function updating section uses 114 in step SA07 on the other hand, one of the reward calculation section 112 calculated negative reward R on the function's update formula Q at. In step SA06 updates the value function update section 114 then the action value table using the state variable S and the determination data D in the current state, the reward R and the value (updated function Q ) of the action value. The session 110 updates the action value table again by repeating the above processing of the steps SA01 to SA07 and advancing the learning of the amount of adjustment of the moving speed of each individual motor of the robot. It is noted that the processing of calculating the rewards R and the processing of updating the value function in the steps SA04 to SA07 for all of those in the investigation data D contained data.

To promote the gain learning described above, a neural network may be used instead of, for example, Q learning. 5A schematically shows a model of a neuron model. 5B schematically shows the model of a neural network with three layers, in which the in 5A neurons are combined. The neural network can be configured, for example, following a neuron model by a computing unit, a storage unit or the like.

This in 5A The illustrated neuron outputs a result y with respect to a plurality of inputs x (here, inputs x ₁ to x _3, for example). The inputs x ₁ to x ₃ are respectively multiplied by corresponding weights w (w ₁ to w ₃ ). The neuron thus outputs the result y expressed by the following formula 2. It is noted that in the following formula 2, an input x, a result y, and a weight w are all vectors. In addition, θ expresses a bias, and f _k expresses an activation function. $$y=f_k\left({\displaystyle {\Sigma }_{i-1}^n{x}_i{w}_i}-\theta \right)$$

In the neural network with the three in 5B shown layers are a variety of inputs x (here the inputs x1 to x3 as an example) from the left side of the neural network and the results y (here the results y1 to y3 as an example) from the right side of the neural network. In the in 5B example shown are the inputs x1 . x2 and x3 with corresponding weights (in total as w1 expressed) and in three neurons N11 . N12 respectively N13 entered.

In 5B become the respective outputs of the neurons N11 to N13 in total as z1 expressed. Expenditure z1 may be considered as feature vectors obtained by extracting feature sizes of the input vectors. In the in 5B the example shown, the respective feature vectors z1 with corresponding weights (in total as w2 expressed) and correspondingly in two neurons N21 to N22 entered. The feature vectors z1 push the features between the weights w1 and the weights w2 out.

In 5B become the respective outputs of the neurons N21 and N22 in total as z2 expressed. Expenditure z2 may be considered as feature vectors obtained by extracting feature amounts of the feature vectors z1 to be obtained. In the in 5B the example shown, the respective feature vectors z2 with corresponding weights (in total as w3 expressed) and in three neurons N31 . N32 respectively N33 entered. The feature vectors z2 push the features between the weights W2 and W3 out. Finally, give the neurons N31 to N33 the corresponding results y1 to y3 out.

It is noted that it is possible to employ the so-called deep learning in which a neural network constituting three or more layers is used.

In the machine learning device 100 the controller 1 the session leads 110 the calculation of the state variables S and the determination data D as inputs x based on a multi-layer structure according to the above neural network, so that the learning section 110 can output a tuning amount (result y) of a moving speed of each motor of a robot. Besides, the session uses 110 in the machine learning device 100 the controller 1 a neural network as a value function in the reinforcing learning and performs the calculation of the state variables S and action a as inputs x based on a multi-layer structure according to the above neural network, so that the session 110 a value (result y ) of the action in the state. It is noted that the action mode of the neural network includes a learning mode and a value predicting mode. For example, it is possible to weight using a learning data set in the learning mode w to learn and with the learned weight w in the value prediction mode, determine an action value. It is noted that detection, classification, subtraction or the like can be performed in the value prediction mode.

• einen Schritt des Beobachtens, durch die CPU eines Computers, von Einlerngeschwindigkeits-Abstimmbetragsdaten S1, Sollgeschwindigkeitsdaten S2, Bewegungsgeschwindigkeitsdaten S3 und Bewegungsbahndaten S4 als Zustandsvariablen S, die den aktuellen Zustand einer Umwelt ausdrücken, in der der Roboter gesteuert wird;
• einen Schritt des Erfassens von Bestimmungsdaten D, die ein Eignungsbestimmungsergebnis des Betriebszustandes des Roboters gemäß der abgestimmten Bewegungsgeschwindigkeit eines jeden der Motoren anzeigen; und
• einen Schritt des Lernens der Sollgeschwindigkeitsdaten S2, der Bewegungsgeschwindigkeitsdaten S3 und der Bewegungsbahndaten S4 und des Abstimmbetrags der Bewegungsgeschwindigkeit eines jeden der Motoren des Roboters in Zuordnung zueinander unter Verwendung der Zustandsvariablen S und der Bestimmungsdaten D.

The configuration of the above controller 1 may be implemented as a machine learning process (or software) performed by the processor 101 to be discribed. The machine learning method is a method of learning a tuning amount of the moving speed of each individual motor of the robot. The machine learning method includes:

• a step of observing, by the CPU of a computer, teaching-in tune amount data S1 , Target speed data S2 , Movement speed data S3 and trajectory data S4 as state variables S expressing the current state of an environment in which the robot is controlled;
• a step of acquiring determination data D indicating a fitness determination result of the operating state of the robot according to the coordinated moving speed of each of the motors; and
• a step of learning the target speed data S2 , the motion speed data S3 and the trajectory data S4 and the rate of adjustment of the speed of movement of each of the motors of the robot in association with each other using the state variables S and the destination data D.

6 shows a controller 2 according to a second embodiment.

The control 2 includes a machine learning device 120 and a state data acquiring section 3 that as state data S0 Einlerngeschwindigkeits-Abstimmbetragsdaten S1 , Target speed data S2 , Movement speed data S3 and trajectory data S4 that of a state observation section 106 observed state variables S detected. The state data acquiring section 3 can the status data S0 from every section of the controller 2 , various sensors of a robot, appropriate data inputs by a worker or the like.

The machine learning device 120 the controller 2 includes software (such as a learning algorithm) and hardware (such as a processor) 101 ) for spontaneously learning a tuning amount of a moving speed of each motor of a robot through machine learning, software (such as a calculation algorithm), and hardware (such as a processor 101 ) for outputting the learned tuning amount of the moving speed of each of the motors of the robot as a command for the control 2 , The machine learning device 120 the controller 2 can be configured so that a common processor executes all the software, such as a learning algorithm and a calculation algorithm.

A decision-making section 122 for example, as one of the functions of the processor 101 or as in ROM 102 stored software for operating the processor 101 be configured. Based on a learning outcome of the session 110 generates and gives the decision-making section 122 a set point C that includes a command for determining a trim amount of a moving speed of each motor of a robot with respect to a target speed of the tip end of the robot, the moving speed of each of the motors of the robot, and a moving path near the tip end of the robot. When the decision-making section 122 setpoint C to the controller 2 the condition of an environment changes accordingly.

The state observation section 106 In a next learning cycle, it observes state variables S, which follow the output of the setpoint C to an environment through the decision-making section 122 have changed. The session 110 for example, updates a value function Q (that is, an action value table) using the changed state variables S to learn a tuning amount of a moving speed of each motor of a robot. It is noted that the state observation section 106 the teaching speed tuning amount data S1 from a RAM 103 the machine learning device 120 as described in the first embodiment, instead of looking at it by the state data acquisition section 3 acquired status data S0 capture.

The decision-making section 122 For requesting a vote of a movement speed of each motor of a robot to the controller 2 a setpoint C which is calculated based on a learning outcome. By repeatedly performing the learning cycle, the machine learning device brings 120 learning a tuning amount of a moving speed of each motor of a robot and gradually improving the reliability of the tuning amount of the moving speed of each motor of the robot, by the machine learning device 120 itself is determined.

The machine learning device 120 the controller 2 with the above configuration produces the same effect as the machine learning apparatus described above 100 , In particular, the machine learning device 120 with the output of the decision-making section 122 change the state of an environment. On the other hand, the machine learning device 100 request an external device for a function corresponding to the decision making section to learn the learning section of the session 110 to play in an environment.

7 shows a system 170 , the robot 160 according to one embodiment.

The system 170 includes a variety of robots 160 and 160 ' that perform at least the same operation and a wired / wireless network 172 that the robots 160 and 160 ' connects with each other. At least one of the multitude of robots 160 is as a robot 160 with the above control 2 configured. Besides, the system can 170 robot 160 ' that have the control 2 do not include. The robots 160 and 160 ' have a mechanism needed to perform an operation for the same purpose.

In the system 170 with the above configuration, those can control 2 comprehensive robot 160 among the multitude of robots 160 and 160 ' automatically and accurately calculate a tuning amount of a moving speed of each motor of a robot with respect to a target speed of the tip end of the robot, the moving speed of each of the motors of the robot, and a moving track near the tip end of the robot without relying on a calculation or estimation Use of the learning outcomes of the session 110 to leave. In addition, the controller 2 of at least one of the robots 160 based on the one for each of the other robots 160 and 160 ' received state variables S and determination data D a tuning amount of the movement speed of each of the motors of the robot, all robots 160 and 160 ' is common, learning, so the learning outcomes between all the robots 160 and 160 ' to be shared. Accordingly, the system allows 170 , the speed and reliability of learning a tuning amount of a moving speed of each of the motors of the robot with a wider range of data sets (with state variables S and determination data D ) as inputs.

8th shows a system 170 ' with a variety of robots 160 ' according to a further embodiment.

The system 170 ' includes the machine learning device 120 (or 100 ), the variety of robots 160 ' with the same machine configuration and a wired / wireless network 172 that the robots 160 ' and the machine learning device 120 (or 100 ) connects to each other.

In the system 170 ' With the above configuration, the machine learning device 120 (or 100) a tuning amount of a moving speed of each motor of a robot with respect to a target speed of the tip end of the robot, all the robots 160 ' in common, the moving speed of each of the motors of the robot and a moving path in the vicinity of the tip end of the robot based on state variables S and determination data D that are common to each of the plurality of robots 160 ' and automatically and accurately learn the amount of adjustment of the moving speed of each of the motors of the robot with respect to the target speed of the tip end of the robot, the moving speed of each of the motors of the robot and the moving path near the tip end of the robot using the learning results calculate without relying on calculation or estimation.

In the system 170 ' can the machine learning device 120 (or 100 ) have a configuration in a cloud server or the like in the network 172 is available. According to the configuration, a desired number of robots 160 ' with the machine learning device 120 (or 100 ), regardless of the existing locations and the times of the plurality of robots 160 ' ,

Workers working in the systems 170 and 170 ' can make a determination to determine whether the degree of achievement of learning a A tuning amount of a moving speed of each motor of a robot with the machine learning device 120 (or 100 ) (ie, the reliability of the amount of movement speed of each of the motors of the robot) at an appropriate time after the start of the learning by the machine learning device 120 (or 100 ) has reached a required level.

The embodiments of the present invention are described above. However, the present invention is not limited to the examples of the above-mentioned embodiments and may be performed in various modes by adding appropriate modifications.

For example, one of the machine learning devices 100 and 120 executed learning algorithm, a through the machine learning device 120 executed calculation algorithm and one by the controllers 1 and 2 executed control algorithm is not limited to the above algorithms, but it can be used various algorithms.

In addition, the above embodiments describe a configuration in which the controller 1 (or 2 ) and the machine learning device 100 (or 120 ) have a different CPU. The machine learning device 100 (or 120 ) can however by the CPU 11 the controller 1 (or 2 ) and one in the ROM 12 stored system program can be realized.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• JP 6285402 [0002, 0003]

Claims (8)

A controller that adjusts a moving speed of each motor of a robot that performs a coating with a sealing material, the controller comprising: a machine learning device that learns a tuning amount of the moving speed of each of the motors of the robot, wherein the machine learning device has: a state observation section which, as state variables expressing a current state of an environment, learning speed adjustment amount data indicating the tuning amount of the moving speed of each of the motors of the robot, target speed data indicating a target speed of a tip end of the robot, moving speed data indicating the moving speed of a robot indicating each of the motors of the robot, and observing trajectory data indicating a trajectory near the tip end of the robot, a determination data acquiring section that acquires determination data indicating a fitness determination result of the moving speed of the tip end of the robot, and a learning section that learns the target speed data, the movement speed data and the trajectory data in association with the adjustment amount of the moving speed of each of the motors of the robot using the state variables and the determination data. Control after Claim 1 wherein the determination data includes, in addition to the fitness determination result of the moving speed of the tip end of the robot, a fitness determination result of a position of the tip end of the robot. Control after Claim 1 or 2 wherein the learning section comprises: a reward calculation section that calculates a reward associated with the fitness determination result, and a value function update section that updates a function using the reward that includes a value of the adjustment amount of the movement speed of each of the motors of the robot with respect to the target speed of the tip end of the robot, the moving speed of each of the motors of the robot, and the trajectory near the tip end of the robot are expressed. Control according to one of the Claims 1 to 3 wherein the learning section performs a calculation of the state variables and the determination data on the basis of a multi-layered structure. Control according to one of the Claims 1 to 4 , further comprising: a decision determining section that outputs a target value based on the amount of adjustment of the moving speed of each of the motors of the robot based on a learning result of the learning section. Control according to one of the Claims 1 to 5 wherein the learning portion learns the timing of adjusting the moving speed of each of the motors of the robot in each of a plurality of robots using the state variables and destination data obtained for each of the plurality of robots. Control according to one of the Claims 1 to 6 wherein the machine learning device is present in a cloud server. A machine learning apparatus that learns a tuning amount of a moving speed of each motor of a robot that performs coating with a sealing material, the machine learning apparatus comprising: a state observation section indicative of state variables expressing a current state of environment, training speed adjustment amount data indicating the tuning amount of the moving speed of each of the motors of the robot, target speed data indicating a target speed of a tip end of the robot, moving speed data representing the moving speed of each indicate the motors of the robot and observe trajectory data indicating a trajectory near the tip end of the robot; a determination data acquiring section that acquires determination data indicating a fitness determination result of the moving speed of the tip end of the robot; and a learning section that learns the target speed data, the movement speed data and the trajectory data in association with the adjustment amount of the moving speed of each of the motors of the robot using the state variables and the determination data.

Images