JP2020098538A - Control device, control method and control program - Google Patents

Abstract

To provide a control device, a control method, and a control program for controlling a device to operate normally even when an abnormality is detected in the device.SOLUTION: A control device 10 includes: an acquisition unit 11a that acquires time-series data indicating a state of a device 20; a calculation unit 13a that calculates a first degree of abnormality of the device 20 based on the time-series data; a learning unit 15 including a reinforcement learning model 15a that selects the time-series data as a state and a control amount of the device 20 as an action so as to maximize a reward that increases when the first degree of abnormality is low; and a control unit 17a that controls the device 20 based on the output of the reinforcement learning model 15a.SELECTED DRAWING: Figure 1

Description

The present invention relates to a control device, a control method, and a control program.

2. Description of the Related Art Conventionally, when controlling an apparatus such as a machine tool or a robot, it is sometimes detected that an abnormality has occurred in the apparatus, the apparatus is stopped, and inspection or repair is performed.

In Patent Document 1 below, data is acquired from a plurality of sensors, and learning data is generated by adding or deleting data to or from the learning data based on the similarity between the data and whether or not there is an abnormality in the data.・Updated, generated and updated learning data is modeled by the subspace method, and newly acquired observation data and individual data included in the learning data are modeled by the subspace method including the local subspace method. An anomaly detection method for detecting anomalies in observation data based on a distance relationship with space is described.

JP, 2010-191556, A

By using the technique described in Patent Document 1, it is possible to detect an abnormality in the device. However, even if such a technique is used, it is necessary to stop the device and perform inspection or repair when an abnormality is detected. Therefore, for example, it becomes necessary to stop the production line in which the device is installed.

Even when an abnormality of the device is detected, not only when the device needs immediate repair, but also when properly controlled, it may be possible to continue the use for a while. Therefore, when an abnormality is detected in the device, it is sometimes desired to continue using the device until the timing when the device can be stopped does not cause any problem.

Therefore, the present invention provides a control device, a control method, and a control program for controlling the device to operate normally even when an abnormality is detected in the device.

A control device according to an aspect of the present invention includes an acquisition unit that acquires time-series data indicating a state of the device, a calculation unit that calculates a first abnormality degree of the device based on the time-series data, and a time-series data. As a state, a learning unit including a reinforcement learning model that selects a control amount of the device as an action so as to maximize a reward that increases when the first abnormality degree is low, and a device based on the output of the reinforcement learning model. And a control unit for controlling.

According to this aspect, by controlling the device so as to maximize the reward that increases when the abnormality degree is low, the device operates normally even when an abnormality is detected in the device. Can be controlled.

In the above aspect, the learning unit learns the reinforcement learning model using the measured time series data when the first abnormality degree is equal to or higher than the threshold, and the control unit learns the reinforcement learning model by the learning unit. In parallel, the device may be controlled based on the output of the reinforcement learning model.

According to this aspect, it is possible to perform the learning of the reinforcement learning model while continuing the control of the device and control the device in which the abnormality is detected to gradually operate normally.

In the above aspect, the learning unit may perform pre-learning of the reinforcement learning model using the physical model of the device.

According to this aspect, the control device can control the device without learning the reinforcement learning model using the actually measured time series data.

In the above aspect, the calculating unit may determine the first abnormality degree based on the number of times of space division required to isolate one data included in the time-series data from other data in the space in which the time-series data is distributed. May be calculated.

According to this aspect, the outlier can be efficiently detected, and the abnormality of the device can be detected while suppressing the calculation load.

In the above aspect, the calculation unit calculates the second abnormality degree based on the time-series data and the control amount, and the reinforcement learning model sets the time-series data and the second abnormality degree as states, and when the first abnormality degree is low. A controlled variable may be selected as an action to maximize the higher reward.

According to this aspect, it is possible to detect, as a change in the second abnormality degree, that the abnormality is corrected and controlled so that the device operates normally.

In the above aspect, an output unit may be further provided that outputs information indicating that the device is controlled so as to reduce the first abnormality degree based on the second abnormality degree.

According to this aspect, it is possible to inform the user that the apparatus is correcting and controlling the abnormality so that the apparatus operates normally.

In the above aspect, the reinforcement learning model stores a table in which evaluation values for states and actions are summarized, with the values obtained by discretizing states as one of rows or columns and the values obtained by discretizing actions as the other of rows or columns. The behavior may be selected so that the evaluation value is maximized when the state is measured.

According to this aspect, even when an abnormality is detected in the device, the device can be controlled to operate normally with a relatively low calculation load.

In the above aspect, the reinforcement learning model may randomly select an action with a predetermined probability.

According to this aspect, the control amount can be searched so that not only the locally optimal control but also the overall optimal control of the device can be performed.

A control method according to another aspect of the present invention is to obtain time-series data indicating the state of the device, calculate an abnormality degree of the device based on the time-series data, and set the time-series data as a state, Including a reinforcement learning model that selects the control amount of the device as an action so as to maximize the reward that increases when the degree of abnormality is low, and controlling the device based on the output of the reinforcement learning model. Including.

According to this aspect, by controlling the device so as to maximize the reward that increases when the abnormality degree is low, the device operates normally even when an abnormality is detected in the device. Can be controlled.

A control program according to another aspect of the present invention includes an arithmetic unit included in a control device, an acquisition unit that acquires time-series data indicating a state of the device, and a calculation unit that calculates an abnormality degree of the device based on the time-series data. , A learning unit including a reinforcement learning model that selects the control amount of the device as an action so as to maximize the reward that becomes high when the degree of abnormality is low, based on the output of the reinforcement learning model. , As a control unit for controlling the device.

According to this aspect, by controlling the device so as to maximize the reward that increases when the abnormality degree is low, the device operates normally even when an abnormality is detected in the device. Can be controlled.

According to the present invention, it is possible to provide a control device, a control method, and a control program that control a device so that the device operates normally even when an abnormality is detected in the device.

It is a figure which shows the outline|summary of the apparatus controlled by the control apparatus which concerns on embodiment of this invention. It is a figure which shows the structure of the control apparatus which concerns on this embodiment. It is a figure which shows an example of the evaluation value table referred by the control apparatus which concerns on this embodiment. It is a flow chart of control processing performed by a control device concerning this embodiment. 6 is a flowchart of a learning process executed by the control device according to the present embodiment.

Embodiments of the present invention will be described with reference to the accompanying drawings. In addition, in each of the drawings, those denoted by the same reference numerals have the same or similar configurations.

FIG. 1 is a diagram showing an outline of a device controlled by a control device 10 according to an embodiment of the present invention. In the present embodiment, the device controlled by the control device 10 is the ball screw 20. The ball screw 20 is an example of a device controlled by the control device 10. The device controlled by the control device 10 may be any machine tool or robot.

The control device 10 controls the torque of the motor 21 that drives the ball screw 20. The ball screw 20 is a device that rotates a motor 21 to linearly move the table 22. In this example, the control device 10 acquires time series data obtained by measuring the torque T of the motor 21 of the ball screw 20 and the speed v of the table 22. The torque T and the speed v are examples of time series data indicating the state of the ball screw 20.

FIG. 2 is a diagram showing a configuration of the control device 10 according to the present embodiment. The control device 10 includes a controller communication unit 11, a time series data storage unit 12, an abnormality monitoring unit 13, an input/output unit 14, a learning unit 15, a setting information input unit 16, a control program execution unit 17, and a control program storage unit 18. ..

The controller communication unit 11 is composed of a communication interface and transmits/receives data to/from a device to be controlled. In the present embodiment, the controller communication unit 11 transmits/receives data to/from the ball screw 20. The controller communication unit 11 includes an acquisition unit 11a. The acquisition unit 11a acquires time series data indicating the state of the device. In the case of the present embodiment, the acquisition unit 11a acquires time series data indicating the torque T of the motor 21 of the ball screw 20 and time series data indicating the speed v of the table 22 of the ball screw 20.

The time-series data storage unit 12 is composed of a storage unit and stores time-series data indicating the state of the device. The time-series data storage unit 12 may be configured by any storage medium, for example, a semiconductor storage element.

The abnormality monitoring unit 13 is composed of a calculation unit and monitors whether or not an abnormality has occurred in the device. The abnormality monitoring unit 13 includes a calculation unit 13a, and the calculation unit 13a calculates the first abnormality degree of the control target device based on the time-series data. In the case of the present embodiment, the calculation unit 13a calculates the first abnormality degree of the ball screw 20 based on the time series data indicating the speed v of the table 22 of the ball screw 20. The calculation unit 13a calculates the first abnormality degree based on the number of space divisions required to isolate one data included in the time series data from other data in the space where the time series data is distributed. You can The calculation unit 13a uses Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou, "Isolation Forest", [online], Internet <URL:https://cs.nju.edu.cn/zhouzh/zhouzh.files/ Using the technology disclosed in publication/icdm08b.pdf>, the first degree of abnormality is calculated based on the number of spatial divisions required to isolate one data included in time series data from other data. It may be calculated. By using such a method, an outlier can be efficiently detected, and an abnormality of the device can be detected while suppressing the calculation load.

The input/output unit 14 includes an input unit including a keyboard and a touch panel, and an output unit including a liquid crystal display device and a buzzer. The input/output unit 14 receives an input regarding the setting of the control device 10. The input/output unit 14 also outputs an alarm when the abnormality monitoring unit 13 detects an abnormality in the device.

The learning unit 15 is configured by a calculation unit and performs a learning process of the reinforcement learning model 15a. The reinforcement learning model 15a selects the control amount of the device as the action so as to maximize the reward that increases when the first abnormality degree is low, with the time series data as the state. In the case of the present embodiment, the reinforcement learning model 15a selects the control amount of the torque T of the ball screw 20 so that the table 22 of the ball screw 20 operates at the normal speed v.

The setting information input unit 16 includes a keyboard and a touch panel, and includes setting information regarding learning of the reinforcement learning model 15a by the learning unit 15 and threshold setting information regarding the first abnormality degree for outputting an alarm by the abnormality monitoring unit 13. Accept input.

The control program execution unit 17 is composed of a calculation unit and executes a program for controlling the device by the control device 10. The control program execution unit 17 includes a control unit 17a, and the control unit 17a controls the device based on the output of the reinforcement learning model 15a. In the case of the present embodiment, the control unit 17a controls the torque T of the motor 21 of the ball screw 20 based on the output of the reinforcement learning model 15a.

The control program storage unit 18 is configured by a storage unit and stores a program for controlling the device by the control device 10.

According to the control device 10 according to the present embodiment, by controlling the device so as to maximize the reward that increases when the abnormality degree is low, the device operates normally even when an abnormality is detected in the device. Can be controlled to perform various actions.

The learning unit 15 learns the reinforcement learning model 15a using the actually measured time series data when the first degree of abnormality is equal to or greater than the threshold, and the control unit 17a causes the learning unit 15 to learn the reinforcement learning model 15a. In parallel with this, the device may be controlled based on the output of the reinforcement learning model 15a. For example, when the first abnormality degree is a numerical value of 0 to 1, the threshold value is set to 0.5, and when the first abnormality degree is 0.5 or more, the learning unit 15 uses the actually measured time series data. Then, the reinforcement learning model 15a may be learned. As a result, the reinforcement learning model 15a is learned while continuing the control of the device, and the device in which the abnormality is detected can be controlled to gradually operate normally.

The learning unit 15 may perform the pre-learning of the reinforcement learning model 15a using the physical model of the device. The physical model of the device may be input by the setting information input unit 16. By performing the prior learning of the reinforcement learning model 15a using the physical model of the device, the control device 10 can control the device without learning the reinforcement learning model 15a using the actually measured time series data. Like

The calculation unit 13a calculates the second abnormality degree based on the time series data and the control amount, and the reinforcement learning model 15a becomes high when the first abnormality degree is low with the time series data and the second abnormality degree as states. The controlled variable may be selected as an action to maximize the reward. In the case of the present embodiment, the calculation unit 13a calculates the second abnormality degree based on the time series data indicating the speed v of the table 22 and the time series data of the torque T of the motor 21, and the reinforcement learning model 15a uses the table 22 , The torque T of the motor 21 is selected as an action so as to maximize the reward that increases when the first abnormality degree is low. You can Here, the calculation of the second abnormality degree is a space division required for separating one data included in the time series data and the control amount from other data in the space where the time series data and the control amount are distributed. It may be calculated based on the number of times. The second abnormality degree calculated in this way becomes a large value not only when the speed v of the table 22 includes an outlier, but also when the torque T of the motor 21 includes an outlier. Therefore, it is possible to detect that the abnormality is corrected and controlled so that the apparatus operates normally as a change in the second abnormality degree.

The input/output unit 14 may output information indicating that the device is controlled so as to reduce the first abnormality degree based on the second abnormality degree. The input/output unit 14 displays, for example, a text on the liquid crystal display device indicating that the device is controlled to reduce the first abnormality degree, or controls the device to reduce the first abnormality degree. A buzzer may output a sound indicating that there is a sound. In this way, the user can be informed that the device is correcting and controlling the abnormality so that the device operates normally.

The calculation unit 13a of the control device 10 separates one data included in the time-series data from other data in the space in which the time-series data is distributed, using the maximum absolute value of the speed in the table 22 as the time-series data. The first degree of abnormality may be calculated based on the number of space divisions required for this purpose. When the ball screw 20 is operating normally, the maximum absolute value of the speed of the table 22 is almost the same. However, when the parts of the ball screw 20 are abnormally operated due to wear or the like, the maximum absolute value of the speed of the table 22 becomes an outlier. Such an outlier is detected by the first degree of abnormality.

When the control device 10 is operating normally, the maximum absolute value of the speed in the table 22 remains almost constant. However, when the parts of the ball screw 20 are abnormally operated due to wear or the like, the maximum absolute value of the speed of the table 22 may gradually increase. The control device 10 may determine normal when the maximum absolute value of the speed of the table 22 is less than the threshold value, and may determine the above when the maximum absolute value of the speed of the table 22 is equal to or more than the threshold value.

FIG. 3 is a diagram showing an example of the evaluation value table T1 referred to by the control device 10 according to the present embodiment. The evaluation value table T1 is an example of a table in which evaluation values for states and behaviors are summarized with the values obtained by discretizing states as one of rows and columns and the values obtained by discretizing actions as the other of rows and columns. The reinforcement learning model 15a selects an action so as to maximize the evaluation value when the state is measured.

The evaluation value table T1 of this example is a table in which the values obtained by discretizing states are set as rows, and the values obtained by discretizing actions are shown as columns, and evaluation values for states and actions are summarized. Specifically, the first row of the evaluation value table T1 describes "Vel[-inf, -10]" and "Score[0, 0.3]", and the speed (Vel) of the table 22 is -10 or less and This indicates a state in which the first degree of abnormality (Score) is 0 to 0.3. Similarly, the second row of the evaluation value table T1 describes "Vel[-10, 0]" and "Score[0, 0.3]", and the speed (Vel) of the table 22 is -10 to 0 and the first value. This indicates a state in which the degree of abnormality (Score) is 0 to 0.3. Further, "Vel[10, inf]" and "Score[0.6, 1.0]" are written in the last line of the evaluation value table T1, and the speed (Vel) of the table 22 is 10 or more and the first abnormality degree (Score). Represents a state of 0.6 to 1.0. In this example, the speed of the table 22 and the first abnormality degree are discretized so as to take 6 different values, and the ball screw 20 is treated as a total of 36 different states. Therefore, in this example, the final row is the 36th row.

On the other hand, "Trq[0, 10]" is written in the first column of the evaluation value table T1 and represents the action of setting the torque (Trq) of the motor 21 to 0-10. Similarly, "Trq [10, 20]" is written in the second column of the evaluation value table T1 and represents the action of setting the torque (Trq) of the motor 21 to 10-20. Further, "Trq[50, 60]" is described in the last column of the evaluation value table T1 and represents the action of setting the torque (Trq) of the motor 21 to 50-60. In this example, the torque of the motor 21 is discretized so as to take six different values, and is treated as selecting any one of the six actions by the reinforcement learning model 15a. Therefore, in this example, the final row is the sixth row.

The reinforcement learning model 15a selects an action so as to maximize the evaluation value when the state is measured. For example, when "Vel[-inf, -10]" and "Score[0, 0.3]" are measured as the states, the action having the maximum evaluation value is selected from the first row of the evaluation value table T1. .. The evaluation value in the first row of the evaluation value table T1 is 0.62 when “Trq[0, 10]” is selected as the action, and 0.83 when “Trq[10, 20]” is selected as the action. And 0.4 when selecting "Trq [50, 60]" as the action. Therefore, when there is no action whose evaluation value is larger than 0.83, the reinforcement learning model 15a selects "Trq[10, 20]" as the action. In this case, the control device 10 may control the ball screw 20 so that the average value of 10 to 20 is adopted and the torque of the motor 21 is set to 15.

The learning unit 15 of the control device 10 uses Q(s _t , a _t ) _new ←Q(s _t , a _t ) _old +α[(r _t+1 +γ max _a Q(s _t+1 , a _t+1 )). )-Q(s _t , a _t ) _old ], the evaluation value table T1 may be updated. Here, Q is an evaluation value, s _t is the state at time _t, a t is the behavior in time t, α is the learning rate, r _t is the reward at time t, γ is discount Is the rate.

The learning unit 15 may update the reward by setting the reward when the first abnormality degree is equal to or less than the threshold value to be positive and the reward when the first abnormality degree is greater than the threshold value to be negative. More specifically, the learning unit 15 sets the reward when the first abnormality degree is equal to or less than the threshold value to 1, and the reward when the first abnormality degree is higher than the threshold value, -1×([first abnormality degree]- The reward may be updated as 0.5)×100. Here, the first degree of abnormality has a value of 0 to 1.

As described above, according to the control device 10 of the present embodiment, even when an abnormality is detected in the device, it is possible to control the device to operate normally with a relatively low calculation load.

Further, the reinforcement learning model 15a may randomly select an action with a predetermined probability. When the predetermined probability is represented by ε, the reinforcement learning model 15a may select an action according to the evaluation value table T1 with a probability of 1-ε and randomly select an action with a probability of ε. As a result, the control amount can be searched so that not only the locally optimal control but also the overall optimal control of the device can be performed.

FIG. 4 is a flowchart of control processing executed by the control device 10 according to the present embodiment. First, the control device 10 creates a physical model of the device (S10). The control device 10 may accept an input of a physical model of the device from the outside.

The control device 10 executes the pre-learning of the reinforcement learning model by the physical model (S11). After that, the control device 10 sets the pre-learned reinforcement learning model 15a in the control device 10 (S12).

The control device 10 acquires time-series data indicating the state of the device (S13), and determines whether the first abnormality degree of the measured time-series data is greater than or equal to a threshold value (S14). When the first abnormality degree is equal to or higher than the threshold value (S14: YES), the control device 10 online-learns and updates the reinforcement learning model 15a while calculating the output with the reinforcement learning model 15a learned in advance (S15). Further, the control device 10 outputs information indicating that the device is controlled so as to reduce the first abnormality degree (S16).

On the other hand, when the first degree of abnormality is not equal to or more than the threshold value (S14: NO), the control device 10 calculates the output by the reinforcement learning model 15a learned in advance (S17).

After that, the control device 10 controls the device based on the output of the reinforcement learning model 15a (S18). When the control is not ended (S19: NO), the control device 10 acquires new time series data (S13) and continues the control of the device based on the output of the reinforcement learning model 15a. On the other hand, when the control is to be ended (S19: YES), an appropriate action for ending the control of the device is performed, and then the operation is ended.

FIG. 5 is a flowchart of the learning process executed by the control device 10 according to this embodiment. In the figure, details of the pre-learning process (S11) of the reinforcement learning model of FIG. 4 are shown.

First, the control device 10 initializes the evaluation value table (S111). After that, the control device 10 acquires time-series data representing the state simulated by the physical model (S112).

The control device 10 calculates the first abnormality degree and the second abnormality degree based on the time-series data (S113), and determines whether to use the evaluation value (S114). Whether or not to use the evaluation value may be randomly selected with a predetermined probability.

When the evaluation value is used (S114: YES), the control device 10 sets the time series data and the second abnormality degree as the state, and selects the control amount having the maximum evaluation value as the action (S115). On the other hand, when the evaluation value is not used (S114: NO), the control device 10 randomly selects the control amount (S116).

After that, the control device 10 sets the reward to be high when the first abnormality degree is low (S117).

When the data acquisition is not completed (S118: NO), the time series data representing the state simulated by the physical model is acquired (S112), and the process of selecting the control amount is continued. On the other hand, when the data acquisition is to be ended (S118: YES), the control device 10 determines whether or not all the episodes of reinforcement learning have ended (S119). When all episodes have not ended (S119: NO), the control device 10 acquires time-series data representing the state simulated by the physical model (S112) and continues the process of selecting the control amount. On the other hand, when all the episodes have ended (S119: YES), the learning process ends.

The embodiments described above are for facilitating the understanding of the present invention and are not for limiting the interpretation of the present invention. Each element included in the embodiment and its arrangement, material, condition, shape, size and the like are not limited to the exemplified ones but can be changed as appropriate. Further, the configurations shown in different embodiments can be partially replaced or combined.

[Appendix]
An acquisition unit (11a) for acquiring time-series data indicating the state of the device (20),
A calculation unit (13a) for calculating a first abnormality degree of the device (20) based on the time-series data;
A learning unit including a reinforcement learning model (15a) that selects the control amount of the device (20) as an action so as to maximize the reward that increases when the first abnormality degree is low with the time series data as a state. (15),
A controller (17a) for controlling the device (20) based on the output of the reinforcement learning model (15a);
A control device (10) provided with.

10... Control device, 11... Controller communication part, 11a... Acquisition part, 12... Time series data storage part, 13... Abnormality monitoring part, 13a... Calculation part, 14... Input/output part, 15... Learning part, 15a... Reinforcement learning Model, 16... Setting information input section, 17... Control program execution section, 17a... Control section, 18... Control program storage section, 20... Ball screw, 21... Motor, 22... Table

Claims (10)

An acquisition unit that acquires time series data indicating the state of the device,
A calculating unit that calculates a first abnormality degree of the device based on the time-series data;
A learning unit including a reinforcement learning model that selects the control amount of the device as an action so as to maximize a reward that increases when the first abnormality degree is low, with the time series data as a state,
A control unit for controlling the device based on the output of the reinforcement learning model;
A control device including. The learning unit uses the actually measured time series data to learn the reinforcement learning model when the first abnormality degree is equal to or higher than a threshold value,
The control unit controls the device based on the output of the reinforcement learning model in parallel with the learning of the reinforcement learning model by the learning unit,
The control device according to claim 1. The learning unit performs pre-learning of the reinforcement learning model using a physical model of the device,
The control device according to claim 1. The calculator calculates the first abnormality degree based on the number of times of space division required to isolate one data included in the time-series data from other data in a space in which the time-series data is distributed. To calculate,
The control device according to any one of claims 1 to 3. The calculation unit calculates a second abnormality degree based on the time series data and the control amount,
The reinforcement learning model selects the control amount as an action so that the time series data and the second abnormality degree are in a state, and a reward that becomes high when the first abnormality degree is low is maximized.
The control device according to any one of claims 1 to 4. And an output unit configured to output information indicating that the device is controlled to reduce the first abnormality degree based on the second abnormality degree.
The control device according to claim 5. The reinforcement learning model is
A value obtained by discretizing the state is one of a row and a column, a value obtained by discretizing the action is the other of the row or the column, and a table summarizing evaluation values for the state and the action is stored,
Selecting the behavior to maximize the evaluation value when the condition is measured,
The control device according to any one of claims 1 to 6. The reinforcement learning model is
Randomly select the action with a predetermined probability,
The control device according to claim 7. Acquiring time series data indicating the state of the device,
Calculating an abnormality degree of the device based on the time-series data,
Including the reinforcement learning model that selects the control amount of the device as an action so as to maximize the reward that increases when the abnormality degree is low, with the time series data as a state,
Controlling the device based on the output of the reinforcement learning model;
Control method including. The arithmetic unit included in the control device,
An acquisition unit that acquires time series data indicating the state of the device,
A calculation unit that calculates an abnormality degree of the device based on the time-series data,
With the time-series data as a state, a learning unit including a reinforcement learning model that selects a control amount of the device as an action so as to maximize a reward that increases when the degree of abnormality is low, and an output of the reinforcement learning model. A control unit for controlling the device based on
Control program to function as.

Images