JP6490127B2 - Machine learning device, servo control device, servo control system, and machine learning method - Google Patents

Description

  The present invention relates to a machine learning device that learns about a coefficient in speed feedforward control for a servo control device using speed feedforward control, a servo control device and a servo control system including the machine learning device, and a machine learning method About.

  As a servo control device using speed feedforward control, for example, there is a servo control device described in Patent Document 1. The servo control device described in Patent Document 1 differentiates the position command to obtain the position feedforward control amount, adds the feedforward control amount to the control amount obtained by the position loop control to obtain a speed command, This is a servo control device in which a feedforward control amount of speed obtained by differentiating the feedforward amount is added to a value obtained by speed loop control to obtain a current command.

Japanese Patent Laid-Open No. 3-15911

  In the servo control device, a position deviation may occur when the speed command value changes due to the influence of nonlinear characteristics such as machine friction, backlash, and lost motion. In such a case, by making the speed feedforward higher, the position deviation can be reduced and the followability to the position command can be improved, but the adjustment of the speed feedforward coefficient (parameter) becomes difficult.

  In the servo control device using the speed feedforward control, the present invention provides a higher order speed when the speed feedforward is made higher in order to reduce the position deviation and improve the followability to the position command. It is an object of the present invention to provide a machine learning device that performs reinforcement learning that can avoid complicated adjustment of a feedforward coefficient, a servo control device including the machine learning device, a servo control system, and a machine learning method.

(1) A machine learning device (for example, a machine learning device 200 described later) according to the present invention is a speed feedforward calculation unit (for example, a speed feedforward calculation unit 110 described later) that creates a speed feedforward value based on a position command. A machine learning device that performs machine learning on a servo control device (for example, a servo control device 100 described later),
By causing the servo control device to execute a predetermined machining program, state information including at least a servo state including a positional deviation and a combination of coefficients of a transfer function of the speed feedforward calculation means is obtained from the servo control device. Status information acquisition means (for example, status information acquisition unit 201 described later) to acquire;
Behavior information output means (for example, behavior information output unit 203 described later) that outputs behavior information including adjustment information of the combination of the coefficients included in the state information to the servo control device;
Reward output means (for example, reward output unit 2021 described later) for outputting a reward value in reinforcement learning based on the position deviation included in the state information;
Value function update means (for example, a value function update unit 2022 described later) that updates the action value function based on the value of the reward output by the reward output means, the state information, and the action information;
Is a machine learning device.

  (2) In the servo control device of (1), the reward output means may output the reward value based on the absolute value of the position deviation.

  (3) In the servo control device according to (1) or (2), the reward output unit may calculate a reward value based on at least the position deviation and a value including the differential value of the position deviation.

(4) In the servo control device according to (1) or (2), the state information acquisition unit further includes the position deviation within a predetermined range after a speed command value included in the servo state is changed. Observe the time T until
The reward output means may calculate a reward value based on a value including at least the position deviation and the length of the time T.

(5) In the servo control device according to (1) or (2), the state information acquisition unit further includes:
Obtaining a torque command from the servo control device;
The reward output means may calculate a reward value based on at least a value including the position deviation and a differential value of the torque command.

(6) In the servo control device according to (1) or (2), the state information acquisition unit further acquires a torque command from the servo control device,
The reward output unit may calculate a reward value based on at least the position deviation and whether or not the torque command has reached a torque command allowable value.

(7) In the servo control device according to (1) or (2), the state information acquisition unit further acquires a speed deviation from the servo control device,
The reward output means may calculate a reward value based on a value including at least the position deviation and the speed deviation.

  (8) In the servo control device according to any one of (1) to (7), a combination of coefficients of the transfer function of the speed feedforward calculation unit is generated based on the value function updated by the value function update unit. Then, an optimization behavior information output unit (for example, an optimization behavior information output unit 205 described later) may be provided.

  (9) A servo control system according to the present invention includes a machine learning device according to any one of (1) to (8) above, and a servo control having a speed feedforward calculation unit that creates a speed feedforward value based on a position command. And a servo control system.

  (10) A servo control device according to the present invention includes the machine learning device according to any one of (1) to (8) above, and a speed feedforward calculation unit that creates a speed feedforward value based on a position command. Servo control device.

(11) A machine learning method according to the present invention is a machine learning method of a machine learning apparatus that includes a speed feedforward calculation unit that creates a speed feedforward value based on a position command and performs machine learning on a servo controller. There,
By causing the servo control device to execute a predetermined machining program, state information including at least a servo state including a positional deviation and a combination of coefficients of a transfer function of the speed feedforward calculation means is obtained from the servo control device. Acquired,
Action information including adjustment information of the combination of the coefficients included in the state information is output to the servo control device;
Updating a behavior value function based on the value of reward in reinforcement learning based on the positional deviation included in the state information, the state information, and the behavior information;
It is a machine learning method.

  According to the present invention, in the servo control device using the speed feedforward control, when the speed feedforward is made higher in order to reduce the position deviation and improve the followability to the position command, the higher order Reinforcement learning can be performed to avoid complicated adjustment of the speed feedforward coefficient.

1 is a block diagram illustrating a servo control system according to a first embodiment of the present invention. It is a block diagram which shows the structural example by which the servo control apparatus and the machine learning apparatus 200 were connected by the network. 3 is a block diagram illustrating an example of a control target 300. FIG. It is a figure for demonstrating operation | movement of the servomotor in case a process shape is circular. It is explanatory drawing which shows the locus | trajectory error which arises by coasting when the process shape is circular and the rotation direction of the servomotor which moves a table to a Y-axis direction is going to reverse at position A1. It is a figure for demonstrating operation | movement of the servomotor in case a process shape is a square. It is a figure for demonstrating operation | movement of a servomotor in case a process shape is a square with a corner | angular R. FIG. It is a block diagram showing machine learning device 200 of a 1st embodiment. 6 is a flowchart for explaining the operation of the machine learning device 200. 5 is a flowchart for explaining the operation of an optimized behavior information output unit 205 of the machine learning device 200. It is a block diagram which shows the servo control apparatus of the 2nd Embodiment of this invention. It is a characteristic view which shows the waveform of the position deviation which can be selected using the evaluation function weighted and added in the modification. It is a characteristic view which shows the waveform of the position deviation which can be selected using the weighted and added evaluation function in another modification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a block diagram showing a servo control system according to a first embodiment of the invention. As shown in FIG. 1, the servo control system 10 includes n servo control devices 100-1 to 100-n, n machine learning devices 200-1 to 200-n, and a network 400. Note that n is an arbitrary natural number.

Here, the servo control device 100-1 and the machine learning device 200-1 are in a one-to-one set and are connected to be communicable. The servo control devices 100-2 to 100-n and the machine learning devices 200-2 to 100-n are also connected in the same manner as the servo control device 100-1 and the machine learning device 200-1. In FIG. 1, n sets of the servo control devices 100-1 to 100-n and the machine learning devices 200-1 to 200-n are connected via the network 400, but the servo control device 100-1 ˜100-n and n sets of machine learning devices 200-1 to 200-n may be such that each set of servo control device and machine learning device is directly connected via a connection interface. The n sets of the servo control devices 100-1 to 100-n and the machine learning devices 200-1 to 200-n may be installed in the same factory, for example, or may be installed in different factories. May be.
The network 400 is, for example, a LAN (Local Area Network) constructed in a factory, the Internet, a public telephone network, or a combination thereof. There are no particular limitations on the specific communication method in the network 400, whether it is a wired connection or a wireless connection, or the like.
FIG. 2 is a block diagram showing a set of a servo control device and a machine learning device of the servo control system according to the first embodiment of the present invention, and a control target. The servo control device 100 and the machine learning device 200 in FIG. 2 correspond to, for example, the servo control device 100-1 and the machine learning device 200-1 shown in FIG.
The control object 300 is, for example, a servo motor, a machine tool including a servo motor, a robot, an industrial machine, or the like. The servo control device 100 may be provided as a part of a machine tool, a robot, an industrial machine, or the like.

First, the servo control device 100 will be described.
The servo control device 100 includes a position command creation unit 101, a subtractor 102, a position control unit 103, an adder 104, a subtractor 105, a speed control unit 106, an adder 107, an integrator 108, a position feedforward calculation unit 109, and A speed feedforward calculation unit 110 is provided.
The position command creation unit 101 creates a position command value and outputs the created position command value to the subtracter 102 and the position feedforward calculation unit 109. The subtractor 102 obtains a difference between the position command value and the detected position that has been subjected to position feedback, and outputs the difference as a position deviation to the position control unit 103 and transmits it to the machine learning device 200.

The position command creating unit 101 creates a position command value based on a program that operates the control target 300. The control object 300 is, for example, a machine tool including a servo motor. When a table on which a workpiece (workpiece) is mounted is moved in the X-axis direction and the Y-axis direction, the control target 300 is processed in the X-axis direction and the Y-axis direction. In contrast, a servo control device shown in FIG. 2 is provided. When the table is moved in directions of three or more axes, a servo control device is provided for each axis direction.
The position command creating unit 101 creates a position command value by setting a feed speed so as to obtain a machining shape specified by the machining program.

  The position control unit 103 outputs a value obtained by multiplying the position deviation by the position gain Kp to the adder 104 as a speed command value. The position feedforward calculation unit 109 outputs a value obtained by differentiating the position command value and multiplied by the feedforward coefficient to the adder 104 and the speed feedforward calculation unit 110.

  The adder 104 adds the speed command value and the output value of the position feedforward calculation unit 109, and outputs the result to the subtractor 105 as a speed command value subjected to feedforward control. The subtractor 105 obtains the difference between the output of the adder 104 and the speed detection value that is speed-feedback, and outputs the difference to the speed control unit 106 as a speed deviation.

  The speed control unit 106 adds the value obtained by multiplying the speed deviation by the integral gain K1v and the value obtained by multiplying the speed deviation by the proportional gain K2v, and outputs the result to the adder 107 as a torque command value.

The speed feedforward calculation unit 110 performs a speed feedforward calculation process indicated by a transfer function Gf (S) expressed by Equation 1 (shown as Equation 1 below) and outputs the result to the adder 107.

  The adder 107 adds the torque command value and the output value of the speed feedforward calculation unit 110, and outputs the result to the control target 300 as a torque command value subjected to feedforward control.

The control object 300 outputs a speed detection value, and the speed detection value is input to the subtractor 105 as speed feedback. The speed detection value is integrated by the integrator 108 to become a position detection value, and the position detection value is input to the subtractor 102 as position feedback.
As described above, the servo control device 100 is configured.

Next, the control target 300 controlled by the servo control device 100 will be described.
FIG. 3 is a block diagram illustrating a machine tool including a servo motor, which is an example of the control target 300.
The servo control device 100 moves the table 304 with the servo motor 302 via the coupling mechanism 303 to process a workpiece (workpiece) mounted on the table 304. The coupling mechanism 303 includes a coupling 3031 coupled to the servo motor 302 and a ball screw 3033 fixed to the coupling 3031, and a nut 3032 is screwed to the ball screw 3033. The nut 3032 screwed to the ball screw 3033 moves in the axial direction of the ball screw 3033 by the rotational drive of the servo motor 302.

  The rotation angle position of the servo motor 302 is detected by a rotary encoder 301 that is associated with the servo motor 302 and serves as a position detection unit, and the detected signal is used as speed feedback. The detected signal is integrated and used as position feedback. The output of the linear scale 305 that is attached to the end of the ball screw 3033 and detects the moving distance of the ball screw 3033 may be used as position feedback. Further, position feedback may be generated using an acceleration sensor.

<Machine learning device 200>
The machine learning device 200 learns the coefficient of the transfer function of the speed feedforward calculation unit 110 by executing a preset machining program (hereinafter also referred to as “learning machining program”). Here, the machining shape designated by the machining program at the time of learning is, for example, a circle, a square, a square with a square R (a square with quarter arc), or the like.

  4A and 4B are diagrams for explaining the operation of the servo motor when the machining shape is circular. FIG. 4C is a diagram for explaining the operation of the servo motor when the machining shape is a square. FIG. 4D is a diagram for explaining the operation of the servo motor when the machining shape is a square with a corner R. 4A to 4D, the table moves so that the workpiece (workpiece) is processed clockwise.

When the machining shape is circular, as shown in FIG. 4A, at the position A1, the servo motor that moves the table in the Y-axis direction reverses the rotation direction, and the table moves so that the table is linearly reversed in the Y-axis direction. At the position A2, the servo motor that moves the table in the X-axis direction reverses the rotation direction, and the table moves so as to reverse the line in the X-axis direction.
FIG. 4B is an explanatory diagram showing a trajectory error caused by coasting when the machining shape is circular and the rotation direction of the servo motor that moves the table in the Y-axis direction is about to be reversed at position A1.
As shown in FIG. 4B, when the rotation direction of the servo motor that moves the table in the Y-axis direction is about to reverse at position A1, a reverse delay occurs, the trajectory error expands in the radial direction, and an enlarged display of the trajectory error is displayed. When done, the trajectory error appears as a protrusion.

  When the machining shape is a square, as shown in FIG. 4C, the servo motor that moves the table in the Y-axis direction at position B shifts from the stop to the rotation operation, and the servo motor that moves the table in the X-axis direction is From rotation to stop, the table moves from linear motion in the X-axis direction to linear motion in the Y-axis direction.

When the machining shape is a square with a corner R, as shown in FIG. 4D, the servo motor that moves the table in the Y-axis direction at the position C1 moves from the stop to the rotation operation, and the table moves from the linear operation in the X-axis direction. Move to arc motion. At the position C2, the servo motor that moves the table in the X-axis direction moves from rotation to stop, and the table moves from the arc motion to the linear motion in the Y-axis direction.
Evaluate the coasting (running by inertia) that occurs when the rotation direction is reversed or stopped from the rotation state according to the machining shape specified by the machining program at the time of learning described above, and the influence on the position deviation Can be examined.

  By executing the machining program at the time of learning, the position command creation unit 101 of the servo control device 100 sequentially sets the position command value so as to obtain a machining shape of a circle, a square, and a square with a square R. Output. In addition, the feed speed is changed for each machining shape of a circle, a square, and a square with a square R (a square with quarter arc) so that the influence on a plurality of feed speeds can be learned. The feed rate may be changed during the movement of the figure of the machining shape, for example, when the table is moved to a square machining shape, when the corner is passed.

Prior to description of each functional block included in the machine learning device 200, first, a basic mechanism of reinforcement learning will be described. An agent (corresponding to the machine learning device 200 in the present embodiment) observes the state of the environment, selects a certain action, and the environment changes based on the action. As the environment changes, some reward is given and the agent learns to select a better action (decision).
While supervised learning shows a complete correct answer, the reward in reinforcement learning is often a fractional value based on some change in the environment. For this reason, the agent learns to select an action to maximize the sum of rewards in the future.

  In this way, in reinforcement learning, by learning an action, an appropriate action is learned based on the interaction that the action gives to the environment, that is, a learning method for maximizing a reward obtained in the future is learned. This indicates that, in the present embodiment, for example, an action that affects the future, such as selecting action information for reducing the positional deviation, can be acquired.

Here, although any learning method can be used as reinforcement learning, in the following description, a method of learning a value Q (s, a) for selecting an action a under a state s of an environment. A case where Q learning (Q-learning) is used will be described as an example.
The purpose of Q learning is to select the action a having the highest value Q (s, a) as the optimum action from the actions a that can be taken in a certain state s.

  However, when Q learning is first started, the correct value Q (s, a) is not known at all for the combination of the state s and the action a. Therefore, the agent selects various actions a under a certain state s, and selects a better action based on the reward given to the action a at that time, whereby the correct value Q (s , A) is learned.

In addition, since it is desired to maximize the total amount of rewards obtained in the future, the goal is to finally satisfy Q (s, a) = E [Σ (γ ^t ) r _t ]. Here E [] denotes the expected value, t is the time, parameter γ is called the discount rate to be described later, is r _t compensation at time t, sigma is the sum by the time t. The expected value in this equation is the expected value when the state changes according to the optimum behavior. However, since it is unclear what the optimal action is in the Q learning process, reinforcement learning is performed while searching by performing various actions. Such an update formula of the value Q (s, a) can be expressed by, for example, the following Formula 2 (shown as Formula 2 below).

In Equation 2 above, s _t represents the state of the environment at time t, a _t represents the action at time t. By the action _{a t,} the state changes to _{s t + 1.} rt _{+ 1} represents the reward obtained by the change in the state. The term with max is a value obtained by multiplying the Q value when the action a having the highest Q value known at that time is selected under the state s _{t + 1} by γ. Here, γ is a parameter of 0 <γ ≦ 1, and is called a discount rate. In addition, α is a learning coefficient and is in a range of 0 <α ≦ 1.

Equation 2 described above, the results of the trial _{a t,} based on the reward _{r t + 1} which has been returned, behavior in state _{s t a t} value _{Q (s t, a} t) represents a method for updating.
This update equation, behavior in state _{s t a t} value _{Q (s} t, _{a t)} than, action _{a t} by next state _s value of the best behavior in _{t + 1 max} a _Q of _{(s t +} 1, a) If the direction is larger, Q (s _t , a _t ) is increased. On the other hand, if the direction is smaller, Q (s _t , a _t ) is decreased. That is, the value of a certain action in a certain state is brought close to the value of the best action in the next state. However, the difference varies depending on the discount rate γ and the reward r _{t + 1} , but basically, the value of the best action in a certain state is propagated to the value of the action in the previous state. It is a mechanism to go.

  Here, in Q learning, there is a method in which learning is performed by creating a table of Q (s, a) for all state-action pairs (s, a). However, there are too many states to obtain the value of Q (s, a) for all the state-action pairs, and it may take a long time for Q learning to converge.

  Therefore, a known technique called DQN (Deep Q-Network) may be used. Specifically, the value function Q is configured using an appropriate neural network, and the value of Q (s, a) is approximated by adjusting the parameter of the neural network and approximating the value function Q with an appropriate neural network. A value may be calculated. By using DQN, it is possible to shorten the time required for Q learning to converge. The DQN is described in detail in the following non-patent literature, for example.

<Non-patent literature>
"Human-level control through deep reinforcement learning", Volodymyr Mnih1 [online], [Search January 17, 2017], Internet <URL: http://files.davidqiu.com/research/nature14236.pdf>

The machine learning device 200 performs the Q learning described above. Specifically, the machine learning device 200 stores the values of the coefficients a _i and b _j (i, j ≧ 0) of the transfer function of the speed feedforward calculation unit 110 in the servo control device 100 and the machining program at the time of learning. A servo state such as a command and feedback including position deviation information of the servo control device 100 acquired by executing the state s, and each coefficient a _i , b _j of the speed feedforward calculation unit 110 related to the state s The value Q for selecting the adjustment as the action a is learned.

The machine learning device 200 includes position deviation information of the servo control device 100 obtained by executing a machining program at the time of learning based on the coefficients a _i and b _j of the transfer function of the speed feedforward calculation unit 110. The state information s including the servo state such as the command and feedback is observed to determine the action a. The machine learning device 200 returns a reward every time the action a is performed. For example, the machine learning device 200 searches for an optimal action a by trial and error so that the total of rewards in the future becomes maximum. By doing so, the machine learning device 200 is obtained by executing the machining program at the time of learning based on the coefficients a _i and b _j of the transfer function of the speed feedforward calculation unit 110. It is possible to select an optimum action a (that is, optimum coefficients a _i and b _{j of} the speed feedforward calculation unit 110) for a state s including a servo state such as a command including position deviation information and feedback. Become.

That is, based on the value function Q learned by the machine learning device 200, among the actions a applied to the coefficients a _i and b _j of the transfer function of the speed feedforward calculation unit 110 according to a certain state s. By selecting the action a that maximizes the value of Q, the action a that minimizes the positional deviation acquired by executing the processing program at the time of learning (that is, the speed feedforward calculation unit 110) Coefficients a _i , b _j ) can be selected.

FIG. 5 is a block diagram illustrating the machine learning device 200 according to the first embodiment of this invention.
In order to perform the above-described reinforcement learning, as illustrated in FIG. 5, the machine learning device 200 includes a state information acquisition unit 201, a learning unit 202, a behavior information output unit 203, a value function storage unit 204, and optimized behavior information output. The unit 205 is provided. The learning unit 202 includes a reward output unit 2021, a value function update unit 2022, and a behavior information generation unit 2023.

The state information acquisition unit 201 is acquired by executing the machining program at the time of learning based on the coefficients a _i and b _j of the transfer function of the speed feedforward calculation unit 110 in the servo control device 100. The state s including the servo state such as the command including the position deviation information and the feedback is acquired from the servo control device 100. This state information s corresponds to the environmental state s in Q learning.
The state information acquisition unit 201 outputs the acquired state information s to the learning unit 202.
It should be noted that the coefficients a _i and b _j of the velocity feedforward calculation unit 110 at the time when Q learning is first started are generated by the user in advance. In this embodiment, the initial setting values of the coefficients a _i and b _j of the velocity feedforward calculation unit 110 created by the user are adjusted to the optimum values by reinforcement learning. The coefficients a _i and b _j of the speed feedforward calculation unit 110 are, for example, as initial setting values, a ₀ = 1, a ₁ = 0, b ₀ = 0, b ₁ = inertial value of the control target. Further, dimensions m and n of the coefficients a _i and b _j are set in advance.
That is, 0 ≦ i ≦ m b _j for a _{i and} 0 ≦ j ≦ n.

  The learning unit 202 is a part that learns a value Q (s, a) when a certain action a is selected under a certain environmental state s. Specifically, the learning unit 202 includes a reward output unit 2021, a value function update unit 2022, and a behavior information generation unit 2023.

The reward output unit 2021 is a part that calculates a reward when the action a is selected under a certain state s. Here, a set of position deviations (position deviation set) which is a state variable in the state s is PD (s), action information a (coefficients a _i and b _{j of the} speed feedforward calculation unit (i and j are 0 and positive). The position deviation set, which is a state variable related to the state information s ′ that has changed from the state s by the correction) of (indicating an integer) is indicated by PD (s ′). The value of the position deviation in the state s is a value calculated based on a preset evaluation function f (PD (s)).
As the evaluation function f, for example,
Function that calculates the integrated value of absolute value of position deviation 絶 対 | e | dt
Function that calculates the integrated value by weighting the absolute value of position deviation with time 時間 t | e | dt
A function for calculating an integrated value of the absolute value of the position deviation to the power of 2n (n is a natural number);
∫e ²ⁿ dt (n is a natural number)
Function for calculating the maximum absolute value of position deviation Max {| e |}
Etc. can be applied.

  At this time, the position deviation value f (PD (s ′)) of the servo control device 100 operated based on the corrected speed feedforward calculation unit 110 related to the state information s ′ corrected by the action information a is the action When the position deviation value f (PD (s)) of the servo control device 100 operated based on the speed feedforward calculation unit 110 before correction related to the state information s before correction by the information a becomes larger. The reward output unit 2021 sets the reward value to a negative value.

On the other hand, the position deviation value f (PD (s ′)) of the servo control device 100 operated based on the corrected speed feedforward calculation unit 110 related to the state information s ′ corrected by the action information a is the action When the value becomes smaller than the position deviation value f (PD (s)) of the servo control device 100 operated based on the speed feedforward calculation unit 110 before correction related to the state information s before correction by the information a. The reward value is a positive value.
Note that the position deviation value f (PD (s ′)) of the servo control device 100 operated based on the corrected speed feedforward calculation unit 110 related to the state information s ′ corrected by the behavior information a is the behavior information. When it is equal to the position deviation value f (PD (s)) of the servo control device 100 operated based on the speed feedforward calculation unit 110 before correction related to the state information s before correction by a, a reward output unit 2021 sets the value of the reward to zero.

  Further, the negative value when the position deviation value f (PD (s ′)) in the state s ′ after the execution of the action a becomes larger than the position deviation value f (PD (s)) in the previous state s. As the value, a negative value may be increased according to the ratio. In other words, it is preferable that the negative value increases in accordance with the degree of increase in the position deviation value. On the other hand, when the position deviation value f (PD (s ′)) in the state s ′ after the action “a” is executed becomes smaller than the position deviation value f (PD (s)) in the previous state s. As the value of, a positive value may be increased according to the ratio. That is, it is preferable that the positive value is increased according to the degree of decrease in the position deviation value.

The value function updating unit 2022 performs Q-learning based on the state s, the action a, the state s ′ when the action a is applied to the state s, and the reward value calculated as described above. As a result, the value function Q stored in the value function storage unit 204 is updated.
The update of the value function Q may be performed by online learning, batch learning, or mini batch learning.
The online learning is a learning method in which the value function Q is immediately updated every time the state s transitions to a new state s ′ by applying a certain action a to the current state s. In batch learning, learning data is collected by applying a certain action a to the current state s, and repeating the transition of the state s to a new state s ′. In this learning method, the value function Q is updated using the learning data. Further, mini-batch learning is a learning method in which the value function Q is updated each time learning data is accumulated to some extent, which is intermediate between online learning and batch learning.

The behavior information generation unit 2023 selects the behavior a in the Q learning process for the current state s. The behavior information generation unit 2023 performs an operation (corresponding to the behavior a in Q learning) for correcting the coefficients a _i and b _j of the speed feedforward calculation unit of the servo control device 100 in the process of Q learning. The behavior information a is generated, and the generated behavior information a is output to the behavior information output unit 203. More specifically, the behavior information generation unit 2023 includes, for example, each coefficient a _i and b _{j of the} speed feedforward calculation unit included in the behavior a with respect to each coefficient of the speed feedforward calculation unit included in the state s. Are added or subtracted incrementally (for example, about 0.01).

Then, the behavior information generation unit 2023 applies the increase or decrease of each coefficient a _i , b _j of the speed feedforward calculation unit 110, transitions to the state s ′, and positive reward (positive reward) Is returned, as the next action a ′, the position deviation value such as incrementally adding to or subtracting from each coefficient a _i , b _j of the velocity feedforward calculation unit as in the previous action is used. You may make it take the policy of selecting action a 'which becomes smaller.

Conversely, if a negative reward (negative reward) is returned, the behavior information generation unit 2023 may, for example, use the coefficients a _i and b _{j of the} speed feedforward calculation unit as the next behavior a ′. On the other hand, a measure may be taken to select an action a ′ whose positional deviation is smaller than the previous value, such as incrementally subtracting or adding to the previous action.

  In addition, the behavior information generation unit 2023 randomly selects a behavior a ′ having the highest value Q (s, a) among the currently estimated values of the behavior a, or randomly acts with a certain small probability ε. It is also possible to take a measure for selecting the action a ′ by a known method such as the ε-greedy method in which a ′ is selected and the action a ′ having the highest value Q (s, a) is selected.

The behavior information output unit 203 is a part that transmits the behavior information a output from the learning unit 202 to the servo control device 100. As described above, the servo control apparatus 100 finely corrects the current state s, that is, the coefficients a _i and b _j of the currently set speed feedforward calculation unit 110, based on this behavior information. State s ′ (that is, each coefficient of the velocity feedforward calculation unit 110 that has been corrected).

  The value function storage unit 204 is a storage device that stores the value function Q. The value function Q may be stored as a table (hereinafter referred to as an action value table) for each state s and action a, for example. The value function Q stored in the value function storage unit 204 is updated by the value function update unit 2022. Further, the value function Q stored in the value function storage unit 204 may be shared with other machine learning devices 200. If the value function Q is shared by a plurality of machine learning devices 200, it is possible to perform reinforcement learning in a distributed manner in each machine learning device 200, so that the efficiency of reinforcement learning can be improved. Become.

Based on the value function Q updated by the value function updating unit 2022 performing Q learning, the optimization behavior information output unit 205 sends an operation that maximizes the value Q (s, a) to the speed feedforward calculation unit 110. Action information a (hereinafter referred to as “optimized action information”) to be performed is generated.
More specifically, the optimization behavior information output unit 205 acquires the value function Q stored in the value function storage unit 204. The value function Q is updated by the value function updating unit 2022 performing Q learning as described above. Then, the optimized behavior information output unit 205 generates behavior information based on the value function Q, and outputs the generated behavior information to the servo control device 100 (speed feedforward calculation unit 110). The optimized behavior information includes information for correcting the coefficients a _i and b _j of the speed feedforward calculation unit 110, similarly to the behavior information output by the behavior information output unit 203 in the process of Q learning.

In the servo control device 100, the coefficients a _i and b _{j of} the speed feedforward calculation unit 110 are corrected based on this behavior information, and the speed feedforward is set to a higher order so as to reduce the position deviation value. can do.
As described above, by using the machine learning device 200 according to the present invention, parameter adjustment of the speed feedforward calculation unit 110 of the servo control device 100 can be simplified.

The function blocks included in the servo control device 100 and the machine learning device 200 have been described above.
In order to realize these functional blocks, each of the servo control device 100 and the machine learning device 200 includes an arithmetic processing device such as a CPU (Central Processing Unit). Each of the servo control device 100 and the machine learning device 200 includes an auxiliary storage device such as an HDD (Hard Disk Drive) that stores various control programs such as application software and an OS (Operating System), and an arithmetic processing device. Includes a main storage device such as a RAM (Random Access Memory) for storing data temporarily required for executing the program.

  In each of the servo control device 100 and the machine learning device 200, the arithmetic processing device reads the application software and the OS from the auxiliary storage device, and develops the read application software and the OS into the main storage device. And arithmetic processing based on the OS. In addition, based on the calculation result, various hardware included in each device is controlled. Thereby, the functional block of this embodiment is implement | achieved. That is, this embodiment can be realized by cooperation of hardware and software.

  Since the machine learning device 200 has a large amount of computation associated with machine learning, for example, a GPU (Graphics Processing Units) is installed in a personal computer, and the GPU is machine-driven by a technology called GPGPU (General-Purpose computing on Graphics Processing Units). If it is used for arithmetic processing associated with learning, high-speed processing can be performed. Furthermore, in order to perform higher-speed processing, a computer cluster is constructed by using a plurality of computers equipped with such GPUs, and parallel processing is performed by a plurality of computers included in the computer cluster. May be.

  Next, the operation of the machine learning device 200 during Q learning in the present embodiment will be described with reference to the flowchart of FIG.

In step S <b> 11, the state information acquisition unit 201 acquires state information s from the servo control device 100. The acquired state information is output to the value function update unit 2022 and the behavior information generation unit 2023. As described above, the state information s is information corresponding to the state in Q learning, and includes the coefficients a _i and b _j of the velocity feedforward calculation unit 110 at the time of step S11. In this way, a set PD (s) of position deviations corresponding to the predetermined feed speed and the machining shape of the circle when the coefficient is the initial value is acquired from the speed feedforward calculation unit 110.

As described above, the coefficients a _i and b _j of the velocity feedforward calculation unit 110 in the initial state s ₀ are, for example, a ₀ = 1, a ₁ = 0, b ₀ = 0, b ₁ = inertia to be controlled. Value.

The position deviation value PD (s ₀ ) in the state s ₀ from the subtracter 102 at the time when Q learning is first started is obtained by operating the servo control device 100 with the machining program at the time of learning. The position command generation unit 101 sequentially outputs position commands in a predetermined machining shape specified by the machining program, for example, a circle, a square, a square with a quarter arc, and a feed rate is changed. Output. For example, the position command value corresponding to the machining shape of the circle is output from the position command generator 101 at a predetermined feed speed, and the subtractor 102 calculates the difference between the position command value and the detected position output from the integrator 108 as a position deviation. It is output to the machine learning device 200 as PD (s ₀ ).

In step S <b> 12, the behavior information generation unit 2023 generates new behavior information a, and outputs the generated new behavior information a to the servo control device 100 via the behavior information output unit 203. The behavior information generation unit 2023 outputs new behavior information a based on the above-described policy. The servo control device 100 that has received the behavior information “a” uses the state s ′ in which the coefficients a _i and b _j of the speed feedforward calculation unit 110 according to the current state s are corrected based on the received behavior information. A machine tool including a motor is driven. As described above, this action information corresponds to action a in Q learning.

In step S _< b> 13, the state information acquisition unit 201 acquires the position deviation PD (s ′) in the new state s ′ from the subtractor 102 and the coefficients a _i and b _j from the speed feedforward calculation unit 110. In this way, the state information acquisition unit 201 obtains a set PD (s) of position deviations corresponding to a predetermined feed speed and a circular machining shape when the coefficients a _i and b _j in the state s ′ from the speed feedforward calculation unit 110. '). The acquired state information is output to the reward output unit 2021.

  In step S14, the reward output unit 2021 determines the magnitude relationship between the position deviation value f (PD (s ')) in the state s' and the position deviation value f (PD (s)) in the state s, and f If (PD (s ′))> f (PD (s)), the reward is set to a negative value in step S15. If f (PD (s ′)) <f (PD (s)), the reward is set to a positive value in step S16. If f (PD (s ′)) = f (PD (s)), the reward is set to zero in step S17. In addition, you may make it weight with respect to the negative value of a reward, and a positive value.

When any of step S15, step S16, and step S17 ends, the value function update unit 2022 stores the value function storage unit 204 in the value function storage unit 204 based on the reward value calculated in any one of the steps in step S18. The value function Q being updated is updated. And it returns to step S11 again and the value function Q converges to an appropriate value by repeating the process mentioned above. Note that the processing may be terminated on condition that the above-described processing is repeated a predetermined number of times or repeated for a predetermined time.
In addition, although step S18 has illustrated online update, it may replace with online update and may be replaced with batch update or mini batch update.

As described above, according to the operation described with reference to FIG. 6, in the present embodiment, by using the machine learning device 200, an appropriate value for adjusting the coefficients a _i and b _j of the higher-order velocity feedforward. A function can be obtained, and the optimization of the speed feedforward coefficients a _i and b _j can be simplified.
Next, with reference to the flowchart of FIG. 7, the operation | movement at the time of the production | generation of the optimization action information by the optimization action information output part 205 is demonstrated.
First, in step S <b> 21, the optimized behavior information output unit 205 acquires the value function Q stored in the value function storage unit 204. The value function Q is updated by the value function updating unit 2022 performing Q learning as described above.

  In step S <b> 22, the optimization behavior information output unit 205 generates optimization behavior information based on the value function Q, and sends the generated optimization behavior information to the speed feedforward calculation unit 110 of the servo control device 100. Output.

Further, according to the operation described with reference to FIG. 7, in the present embodiment, the optimization behavior information is generated based on the value function Q obtained by learning by the machine learning device 200, and the servo control device 100 Based on this optimized behavior information, the adjustment of the coefficients a _i and b _j of the currently set speed feedforward calculation unit 110 can be simplified and the position deviation value can be reduced. In addition, the position deviation value can be further reduced by initializing the velocity feedforward to a higher-dimensional one and learning by the machine learning device 200.

(Second Embodiment)
In the first embodiment, the reward output unit 2021 calculates a reward value based on a preset evaluation function f (PD (s)) with a positional deviation PD (s) in the state s as an input. The position deviation value f (PD) of the state s ′ calculated based on the evaluation function f with the position deviation value f (PD (s)) of the state s and the position deviation PD (s ′) in the state s ′ as inputs. (S ')) was compared.
However, elements other than the position deviation may be added in calculating the reward value.

FIG. 8 is a block diagram showing a servo control system 10A according to the second embodiment of the present invention.
The difference between the servo control system 10A of the second embodiment and the servo control system 10 of the first embodiment shown in FIG. 2 is that, in addition to the positional deviation that is the output of the subtractor 102, The position forward-controlled speed command that is the output of the adder 104, the difference between the speed command that is position-forward controlled and the speed feedback, and the position-forward controlled torque command that is the output of the adder 107 are output to the machine learning device 200. This is the point being entered. In FIG. 8, the position forward-controlled speed command that is the output of the adder 104, the difference between the speed command that is position-forward controlled and the speed feedback, and the position-forward-controlled torque command that is the output of the adder 107. However, it is possible to perform reinforcement learning using any one or a combination of these and a positional deviation.

Even if the position deviation becomes small, a shock may occur in the machine. Specifically, when the jerk is large (the change in acceleration is large), a shock occurs in the machine. In order to reduce the shock of the machine, it is desirable to have a small change in position deviation and a small change in torque command value. Therefore, in addition to the calculation of the reward based on the position deviation value f (PD (s)), the reward is calculated based on a change in the position deviation (differential value of the position deviation) and / or a change in the torque command value (differentiation of the torque command value). You may make it calculate.
Hereinafter, a set of differential values of the position deviation in the state s is referred to as PD ′ (s). A set of torque command values in the state s is described as TC (s), and a set of differential values of the torque command values in the state s is described as TC ′ (s).

<Derivative value of position deviation>
When considering the differential value of the position deviation in the calculation of the reward, the evaluation function g of the differential value of the position deviation is set in advance, the evaluation value g (PD ′ (s)) of the differential value of the position deviation of the state s, and the state By calculating the evaluation value g (PD ′ (s ′)) of the differential value of the positional deviation of s ′, the second reward based on the differential value of the positional deviation is calculated as in the case of the positional deviation. Can do.
As the evaluation function g, as with the evaluation function f, for example, a function for calculating the integrated value of the absolute value of the differential value of the position deviation, the integrated value by weighting the absolute value of the differential value of the position deviation with time. A function for calculating the absolute value of the absolute value of the differential value of the positional deviation, a function for calculating the maximum value of the absolute value of the differential value of the positional deviation, or the like can be applied.
By doing so, the evaluation value g (PD ′ (s ′)) of the differential value of the position deviation in the state s ′ corrected by the behavior information a is the position deviation in the state s before being corrected by the behavior information a. When the evaluation value g (PD ′ (s)) of the differential value becomes larger, the reward output unit 2021 sets the second reward value to a negative value.

On the other hand, the evaluation value g (PD ′ (s ′)) of the differential value of the position deviation in the state s ′ is smaller than the evaluation value g (PD ′ (s)) of the differential value of the position deviation in the previous state s. When it becomes, the reward output unit 2021 sets the value of the second reward to a positive value.
Further, when the evaluation value g (PD ′ (s ′)) of the differential value of the position deviation in the state s ′ is equal to the evaluation value g (PD ′ (s)) of the differential value of the position deviation in the previous state s. The value of the second reward is set to zero.

  When the reward calculated based on the evaluation value of the position deviation described in the first embodiment is referred to as a first reward, the reward output unit 2021 uses the first reward as a reward considering the differential value of the position deviation. You may make it weight and add between the value of a reward, and the value of a 2nd reward. The value function updating unit 2022 considers the state s, the action a, the state s ′ when the action a is applied to the state s, and the reward value that takes into account the differential value of the position deviation calculated as described above. The value function Q stored in the value function storage unit 204 is updated by performing Q learning based on the above.

<Modification>
In the example described above, the value of the first reward and the value of the second reward are weighted and added, but the evaluation function f regarding the absolute value of the position deviation and the evaluation regarding the absolute value of the differential value of the position deviation. The reward may be determined using the evaluation function added by weighting the function g. FIG. 9 is a characteristic diagram showing a waveform of a position deviation that can be selected by using the weighted and added evaluation function.
The position deviation is indicated by e,
Evaluation function f = ∫ | e | dt for position deviation
If the evaluation function g = ∫ | de / dt | dt for the differential value of the position deviation is
The evaluation function added by weighting both is
c × ∫ | e | dt + d × ∫ | de / dt | dt (where c and d are weighting factors)
It becomes.
When the reward value is determined based on the weighted evaluation function, as shown in FIG. 9, the position deviation derivative is compared with the waveform shown by the solid line in which the evaluation function value related to the position deviation derivative is large. It can be seen that a waveform indicated by a broken line having a small evaluation function value is selected.

<Derivative value of torque command value>
In calculating the reward, when considering the differential value of the torque command value TC (s) in the state s, an evaluation function h of the differential value TC ′ (s) of the torque command value is set in advance, and the differential value of the torque command value is calculated. Based on the value h (TC ′ (s)), the third reward based on the differential value of the torque command value can be calculated as in the case of the differential value of the position deviation.
The reward output unit 2021 may add the weights between the first reward value and the third reward value as the reward considering the differential value of the torque command value.
Also, when considering the differential value of the position deviation and the differential value of the torque command value, the reward output unit 2021, for example, sets the reward value, the first reward value, the second reward value, and the third reward value. You may make it weight and add between the value of reward.

<Modification>
As in the case of the position deviation, weighting is performed between the evaluation function related to the position deviation and the evaluation function related to the differential value of the torque command value, the weighted evaluation function is added, and the reward is calculated using the weighted evaluation function. You may decide.
In addition, when taking into account the differential value of the position deviation and the differential value of the torque command value, there is a relationship between the evaluation function related to the position deviation, the evaluation function related to the differential value of the position deviation, and the evaluation function related to the differential value of the torque command value. May be weighted, the weighted evaluation function may be added, and the reward may be determined using the weighted evaluation function.

<Speed command value>
It is preferable that the time T from when the speed command value changes until the position deviation falls within a certain range is as short as possible. Therefore, in addition to the calculation of the reward based on the position deviation, the reward considering the time T until the position deviation falls within a certain range after the speed command value changes can be calculated.
The state information acquisition unit 201 detects a change in the position forward-controlled speed command value that is output from the adder 104, and after the speed command value changes, the position deviation that is output from the subtractor 102 is within a predetermined range. The time T until it falls within is observed. Hereinafter, a set of time T from the change of the speed command value in the state s until the position deviation as the output of the subtracter 102 falls within a predetermined range is referred to as T (s).
In the calculation of the reward, when considering the time T from when the speed command value in the state s changes until the position deviation output from the subtracter 102 falls within a predetermined range, the evaluation function of the time T (s) p is set in advance, and the fourth reward based on the time T (s) is calculated based on the evaluation value p (T (s)) of the time T (s) as in the case of the differential value of the position deviation. Can do.
As a reward that considers the time T from when the speed command value in the state s changes until the positional deviation that is the output of the subtracter 102 falls within a predetermined range, the value of the first reward and the fourth reward You may make it add by weighting between the values of.
Further, when considering any combination of the differential value of the position deviation, the differential value of the torque command value, and the time T (s) described above, for example, the reward value is set as the first reward value. The second reward value, the third reward value, and the fourth reward value corresponding to the combination may be weighted and added.

<Modification>
In the example described above, the value of the first reward and the value of the fourth reward are weighted and added, but the evaluation function f regarding the absolute value of the position deviation and the evaluation function p regarding the time T (s) The reward may be calculated using the added evaluation function. FIG. 10 is a characteristic diagram showing a waveform of a position deviation that can be selected by using the weighted and added evaluation function.
The position deviation is indicated by e,
評 価 | e | dt
If the evaluation function p for the time T (s) is ∫t ₀ dt,
The weighted and added evaluation function is y × ∫ | e | dt + z × ∫t ₀ dt (y and z are weighting factors).
When the reward value is determined based on the weighted and added evaluation function, as shown in FIG. 10, it can be seen that the waveform W1 indicated by a broken line whose time T is smaller than the waveform W2 indicated by the solid line is selected. In FIG. 10, time T is shown as t ₀ (1) and t ₀ (2) in waveforms W1 and W2, respectively.
In addition, when considering any combination of the differential value of the position deviation, the differential value of the torque command value, and the time T (s) described above, the evaluation function related to the position deviation and the evaluation function related to the differential value of the position deviation Weighting may be performed between the evaluation function related to the differential value of the torque command value and the evaluation function related to the time T (s), the weighted evaluation function may be added, and the reward may be determined using the weighted evaluation function. .

<Torque command value>
There is an upper limit for the torque command value. Therefore, it is preferable to set the allowable value TCmax to, for example, an upper limit value or a value less than or equal to the upper limit value so that the allowable value TCmax is not exceeded. Therefore, in addition to the calculation of the reward based on the position deviation, the reward is calculated based on whether or not the torque command value has reached the allowable value TCmax.
Specifically, the state information acquisition unit 201 observes the position forward-controlled torque command value TC (s) that is the output of the adder 107 in the state s. When the state information acquisition unit 201 observes that even one torque command value observed in the state s exceeds the allowable value TCmax, the reward output unit 2021 determines the value of the first reward or the torque in the previous state. Regardless of the command value, the reward is a negative value.

If the torque command value observed in the state s does not exceed the allowable value TCmax, an evaluation function q of the torque command value TC (s) is set in advance, and the evaluation value q (TC ( s)), the fifth reward based on the torque command value TC (s) may be calculated in the same manner as the differential value of the position deviation.
In that case, the reward output unit 2021 may weight and add between the first reward value and the fifth reward value.
Further, when considering any combination of the differential value of the position deviation, the differential value of the torque command value, the time T (s) described above, and the torque command value, for example, the reward value is changed to the first reward value. The value and the second reward value, the third reward value, the fourth reward value, and the fifth reward value corresponding to the combination may be weighted and added. .

<Modification>
As in the case of the position deviation, weighting is performed between the evaluation function relating to the position deviation and the evaluation function q of the torque command value TC (s), the weighted evaluation function is added, and the weighted evaluation function is used. A reward may be determined.
Further, when any one of the differential value of the position deviation, the differential value of the torque command value, the time T (s), and the torque command value TC (s) is considered in combination, the evaluation function and the position deviation related to the position deviation are considered. Are weighted between the evaluation function related to the differential value of the torque, the evaluation function related to the differential value of the torque command value, the evaluation function related to the time T (s), and the evaluation function of the torque command value TC (s). The reward may be determined using a weighted evaluation function.

<Speed deviation>
In addition to position deviation, deviation includes speed deviation. Even if the position deviation is small, it is not preferable that the speed deviation is too large. It is preferable to use not only the position deviation but also the speed deviation, increase the weight of the position deviation reward value, and find a speed feedforward coefficient that reduces the position deviation and the speed deviation. In this case, it is preferable to calculate the sixth reward based on the speed deviation in addition to the calculation based on the position deviation. Here, a set of speed deviations (speed deviation set), which is a state variable in the state s, is referred to as VD (s).
When calculating the speed deviation VD (s) in the state s in the calculation of the reward, an evaluation function u of the speed deviation VD (s) is set in advance, and the evaluation value u (VD (s)) of the speed deviation VD (s) The sixth reward based on the speed deviation VD (s) can be calculated based on the same as in the case of the differential value of the position deviation.

As a reward considering the speed deviation VD (s) in the state s, the first reward value and the sixth reward value may be weighted and added.
In addition, when considering any combination of the differential value of the position deviation, the differential value of the torque command value, the time T (s), the torque command value, and the speed deviation VD (s), for example, the reward value is Between the value of the first reward and the value of the second reward, the value of the third reward, the value of the fourth reward, the value of the fifth reward, and the value of the sixth reward It is also possible to add by weighting.

<Modification>
As in the case of the position deviation, weighting is performed between the evaluation function related to the position deviation and the evaluation function of the speed deviation VD (s), the weighted evaluation function is added, and the reward is calculated using the weighted evaluation function. You may decide.
Further, in the case of considering any combination of the differential value of the position deviation, the differential value of the torque command value, the time T (s), the torque command value TC (s), and the speed deviation, an evaluation function related to the position deviation. And an evaluation function related to the differential value of the position deviation, an evaluation function related to the differential value of the torque command value, an evaluation function related to the time T (s), an evaluation function of the torque command value TC (s), and an evaluation of the speed deviation VD (s) A weight may be weighted between the functions, the weighted evaluation function may be added, and the reward may be determined using the weighted evaluation function.

  Note that the evaluation functions g (PD ′ (s)), h (TC ′ (s)), p (T (s)), q (TC (s)), u (VD () mentioned in the second embodiment are used. As for the evaluation function f (PD (s)), for example, a function for calculating an integrated value of absolute values, a function for calculating an integrated value by weighting the absolute values with time, A function for calculating the integrated value of 2n, a function for calculating the maximum absolute value, or the like can be applied.

  Each component included in the servo control unit and the machine learning device of the servo control device described above can be realized by hardware, software, or a combination thereof. In addition, a servo control method performed by the cooperation of each component included in the servo control device can also be realized by hardware, software, or a combination thereof. Here, “realized by software” means realized by a computer reading and executing a program.

  The program may be stored using various types of non-transitory computer readable media and supplied to the computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD- R, CD-R / W, and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

  Although the above-described embodiment is a preferred embodiment of the present invention, the scope of the present invention is not limited only to the above-described embodiment, and various modifications are made without departing from the gist of the present invention. Can be implemented.

<Modification in which the servo control device includes a machine learning device>
In the above-described embodiment, the machine learning device 200 is configured by a device separate from the servo control device 100. However, part or all of the functions of the machine learning device 200 may be realized by the servo control device 100. Good.

<Freedom of system configuration>
In the embodiment described above, the machine learning device 200 and the servo control device 100 are communicably connected as a one-to-one set. For example, one machine learning device 200 includes a plurality of servo control devices 100 and a network 400. May be communicably connected to each other, and machine learning of each servo control device 100 may be performed.
In that case, each function of the machine learning device 200 may be distributed to a plurality of servers as appropriate. Moreover, you may implement | achieve each function of the machine learning apparatus 200 using a virtual server function etc. on a cloud.
Further, when there are a plurality of machine learning devices 200-1 to 200-n respectively corresponding to a plurality of servo control devices 100-1 to 100-n of the same type, the same specification, or the same series, each machine learning You may make it comprise so that the learning result in apparatus 200-1-200-n may be shared. By doing so, it becomes possible to construct a more optimal model.

DESCRIPTION OF SYMBOLS 10 Servo control system 100 Servo control apparatus 101 Position command preparation part 102 Subtractor 103 Position control part 104 Adder 105 Subtractor 106 Speed control part 107 Adder 108 Integrator 109 Position feedforward calculation part 110 Speed feedforward calculation part 200 Machine Learning device 201 State information acquisition unit 202 Learning unit 203 Behavior information output unit 204 Value function storage unit 205 Optimization behavior information output unit 300 Control target 400 Network

Claims (11)

A machine learning device for performing machine learning on a servo control device, comprising speed feedforward calculation means for creating a speed feedforward value based on a position command,
By causing the servo control device to execute a predetermined machining program, state information including at least a servo state including a positional deviation and a combination of coefficients of a transfer function of the speed feedforward calculation means is obtained from the servo control device. State information acquisition means for acquiring;
Behavior information output means for outputting behavior information including adjustment information of the combination of the coefficients included in the state information to the servo control device;
Reward output means for outputting a reward value in reinforcement learning based on the position deviation included in the state information;
Value function updating means for updating the value function based on the value of the reward output by the reward output means, the state information, and the behavior information;
A machine learning device comprising:   The machine learning device according to claim 1, wherein the reward output unit outputs the reward value based on an absolute value of the position deviation.   The machine learning device according to claim 1, wherein the reward output unit calculates a reward value based on at least the position deviation and a value including the differential value of the position deviation. The state information acquisition means further observes a time T from when the speed command value included in the servo state changes until the position deviation falls within a predetermined range,
The machine learning device according to claim 1, wherein the reward output unit calculates a reward value based on a value including at least the position deviation and the length of the time T. The state information acquisition means further acquires a torque command from the servo control device,
The machine learning device according to claim 1, wherein the reward output unit calculates a reward value based on a value including at least the position deviation and a differential value of the torque command. The state information acquisition means further acquires a torque command from the servo control device,
The machine learning device according to claim 1, wherein the reward output means calculates a reward value based on at least the position deviation and whether the torque command reaches a torque command allowable value. The state information acquisition means further acquires a speed deviation from the servo control device,
The reward output means, based on a value including at least the positional deviation and the speed deviation, the machine learning device according to claim 1 or 2 to calculate the value of compensation. Based on the value function is updated by the value function update unit, according to claim from the speed feed claim 1 having the optimized action information output means for generating and outputting a combination of the coefficients of the transfer function of the forward calculation means 8. The machine learning device according to any one of 7 above.   A servo control system comprising: the machine learning device according to any one of claims 1 to 8; and a servo control device having speed feedforward calculation means for creating a speed feedforward value based on a position command. .   A servo control device comprising: the machine learning device according to any one of claims 1 to 8; and a speed feedforward calculation unit that creates a speed feedforward value based on a position command. A machine learning method of a machine learning device that performs machine learning on a servo control device, comprising a speed feedforward calculation means for creating a speed feedforward value based on a position command,
By causing the servo control device to execute a predetermined machining program, state information including at least a servo state including a positional deviation and a combination of coefficients of a transfer function of the speed feedforward calculation means is obtained from the servo control device. Acquired,
Action information including adjustment information of the combination of the coefficients included in the state information is output to the servo control device;
A machine learning method of updating a value function based on a value of reward in reinforcement learning based on the position deviation included in the state information, the state information, and the behavior information.

Images