JP2019067238A - Control device, control method and control program - Google Patents

Abstract

To simply and accurately execute an optimum control for an actual environment.SOLUTION: A control device 10 collects data acquired by a sensor 21 installed in a control target facility 20; determines a control content using a model for determining a control content of the control target facility 20 while receiving the collected data as an input; and controls the control target facility 20 on the basis of the control content. Then, the control device 10 learns about a model so that a value of data obtained by the sensor 21 after the control is performed is given higher reward as the value is closer to a predetermined value.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a control device, a control method, and a control program.

  Conventionally, in various environments such as a factory, a plant, a building, and a data center, control of control devices, air conditioning, and various other devices is performed. In the conventional method, in general, a human determines thresholds and control contents and performs rule making to create an optimum temperature and a stable control state.

Unexamined-Japanese-Patent No. 2017-142654

  However, in the conventional method, there has been a problem that the optimum control for the real environment can not be simply and accurately executed. For example, when a human decides a threshold or control content and performs rule making and creating an optimum temperature and a stable control state, an expert manually determines the rule, so it takes time and is easy to execute. I could not.

  In addition, it is possible to learn a threshold value, a control content, etc. using reinforcement learning, and to control an apparatus automatically. In designing such a reward for reinforcement learning, a simple reward design is performed, so optimal control can not be performed accurately.

  In order to solve the problems described above and to achieve the object, the control device of the present invention comprises: collection means for collecting data acquired by a sensor installed in a control target facility; and data collected by the collection means Control means for determining control contents using a model for determining control contents of the control target equipment as input, and control means for controlling the control target equipment based on the control contents, and the control means for the model And a learning means for learning such that a higher reward is given as the value of the data obtained by the sensor after the control is performed is closer to a predetermined value.

  Further, the control method of the present invention is a control method executed by a control device, and includes a collection step of collecting data acquired by a sensor installed in a control target facility, and data collected by the collection step. As an input, the control content is determined using a model for determining control content of the control target equipment, and a control step of controlling the control target equipment based on the control content, and the control step for the model And a learning step of learning such that the value of the data obtained by the sensor after the control is performed according to the value of the data is higher as it is closer to a predetermined value.

  Further, the control program of the present invention determines the control content of the control target facility by using the collection step of collecting data acquired by the sensor installed in the control target facility and the data collected by the collection step. Control content is determined using the model to be controlled, and based on the control content, a control step of controlling the equipment to be controlled and a sensor obtained by the sensor after the control step is performed on the model And causing the computer to execute a learning step of learning so that a higher reward is given as the value of the stored data is closer to the predetermined value.

  According to the present invention, it is possible to easily and accurately execute the optimum control for the real environment.

FIG. 1 is a block diagram showing an example of the configuration of a control system according to the first embodiment. FIG. 2 is a diagram illustrating an example of a real environment in which the control device according to the first embodiment performs optimum control. FIG. 3 is a block diagram showing a configuration example of a control device according to the first embodiment. FIG. 4 is a diagram showing an example of data stored in the sensor data storage unit. FIG. 5 is a diagram for explaining the flow of the optimum control process in the control device according to the first embodiment. FIG. 6 is a diagram for explaining parallel processing of optimal control learning in the control device according to the first embodiment. FIG. 7 is a diagram for explaining reward provision in the control device according to the first embodiment. FIG. 8 is a flowchart showing an example of the flow of processing in the control device according to the first embodiment. FIG. 9 is a diagram showing an outline of processing of a control device according to the second embodiment. FIG. 10 is a diagram illustrating a computer that executes a control program.

  Hereinafter, embodiments of a control device, a control method, and a control program according to the present application will be described in detail based on the drawings. The control device, the control method, and the control program according to the present application are not limited by the embodiment.

First Embodiment
In the following embodiments, the configuration of the control system 100 according to the first embodiment, the configuration of the control device 10, and the flow of processing of the control device 10 will be described in order, and finally the effects of the first embodiment will be described. .

[Control system configuration]
FIG. 1 is a block diagram showing an example of the configuration of a control system according to the first embodiment. The control system 100 according to the first embodiment includes the control device 10 and a plurality of control target facilities 20A to 20C in a real environment, and the control device 10 and the control target facilities 20A to 20C mutually communicate via the network 30. It is connected. The configuration shown in FIG. 1 is merely an example, and the specific configuration and the number of devices are not particularly limited. Moreover, when demonstrating control object installation 20A-20C especially without distinction, it describes with the control object installation 20 suitably.

  The control device 10 collects data acquired by the sensors 21 installed in the control target facilities 20A to 20C. Then, the control device 10 receives the collected sensor data as an input, determines the control content using a model such as a neural network model for determining the control content of the control target equipment 20A to 20C, and based on the control content Control the control target equipment 20A to 20C.

  Subsequently, the control device 10 performs reinforcement learning so that the value of data obtained by the sensor 21 after the control is performed on the model is given a higher reward as the value is closer to a predetermined value. For example, the control device 10 gradually gives higher rewards as it approaches the average value of the predetermined upper limit value and the predetermined lower limit value set in advance.

  Each of the control target facilities 20A to 20C is provided with a plurality of sensors 21. The control target equipment is, for example, an in-plant apparatus, a reactor, a building air conditioner, a rack in a data center, and the like. Here, each of the control target equipments 20A to 20C is assumed to be distant in distance. For example, as illustrated in FIG. 2, the control target facility 20 is a tank installed in a plant, and each of the six tanks is provided with a temperature sensor (not shown). There is. FIG. 2 is a diagram illustrating an example of a real environment in which the control device according to the first embodiment performs optimum control.

  In this example, each sensor 21 acquires data on the temperature of the tank and transmits it to the control device 10. Then, since the control device 10 acquires data of the real environment and performs reinforcement learning, it also includes external factors on the real environment which can not be obtained in the virtual environment, etc., and the randomness is further enhanced, and in various situations. It is possible to carry out learning.

  Further, the control device 10 determines the control content for each of the six tanks according to the collected data of the sensor 21, for example, and adjusts the temperature of each of the six tanks with air cooling, cold water or the like. The temperature adjustment is automatically adjusted so that it always stays at the optimum value.

[Configuration of control unit]
Next, the configuration of the control device 10 will be described with reference to FIG. FIG. 3 is a block diagram showing a configuration example of a control device according to the first embodiment. As shown in FIG. 3, the control device 10 includes a communication processing unit 11, a control unit 12, and a storage unit 13. The process of each part which control device 10 has below is explained.

  The communication processing unit 11 controls communication regarding various information. For example, the communication processing unit 11 transmits and receives sensor data to and from the control target facility 20.

  The storage unit 13 stores data and programs necessary for various processes performed by the control unit 12, and particularly includes a sensor data storage unit 13a as closely related to the present invention. For example, the storage unit 13 is a semiconductor memory device such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

  The sensor data storage unit 13a temporarily stores data of the sensors 21 at the same time collected from the sensors 21 of the control target facilities 20 by the collecting unit 12a described later. For example, as illustrated in FIG. 4, the sensor data storage unit 13 a uses the data of the sensor 21 acquired by each sensor 21 at time 12:00, the status 1 to 3 of the real environment A, and the status 1 of the real environment B. The states 1 to 3 of the real environment C are stored. Here, the “state” is various data such as temperature, pressure, sound, vibration and the like acquired by the sensor 21. FIG. 4 is a diagram showing an example of data stored in the sensor data storage unit.

  Describing the specific example, the states 1 to 3 of the real environment A may be, for example, the values of the temperatures of the respective temperature sensors installed at different places in the plant, and the state 1 is the temperature The data may be data of different types of sensors 21 such as the value, the state 2 being a pressure value, and the state 3 being a vibration value. In addition, below, the case where each state is a value of the temperature of each temperature sensor installed in each different place is demonstrated as an example.

  For example, in the example of FIG. 4, the sensor data storage unit 13a stores “40” degrees as state 1 of real environment A at time 12:00, “31” degrees as state 2, and “17” degrees as state 3. It stores "70" degrees as state 1 of real environment B, "80" degrees as state 2 and "66" degrees as state 3 and "50" degrees as state 1 of real environment C and "45" as state 2 Degree, "56" degree is stored as state 3.

  The control unit 12 has a program that defines various processing procedures and the like, and an internal memory for storing required data, and executes various processing by these, and particularly as closely related to the present invention, It has a collection unit 12a, a control unit 12b, and a learning unit 12c. Here, the control unit 12 is an electronic circuit such as a central processing unit (CPU) or a micro processing unit (MPU) or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  The collection unit 12a collects data acquired by the sensor 21 installed in the control target facility 20. Specifically, the collection unit 12a collects data of each sensor 21 installed in each of a plurality of control target facilities 20, and buffers data of the same time in the sensor data storage unit 13a.

  For example, the collection unit 12a periodically (eg, every one minute) receives data from the sensor 21 installed in the control target facility 20 such as a factory or a plant, and buffers the data in the sensor data storage unit 13a. Here, the data acquired by the sensor 21 is, for example, various data such as temperature, pressure, sound, vibration, etc. of the plant which is the control target equipment, the apparatus in the plant, and the reaction furnace.

  The control unit 12b receives the data collected by the collection unit 12a, determines the control content using a model for determining the control content of the control target equipment 20, and based on the control content, the control target equipment 20 Control. The control unit 12b inputs the data of each sensor 21 collected by the collection unit 12a into each model to determine the control content, and controls each control target facility 20 based on each control content.

  For example, the control unit 12b stores the data of each sensor collected by the collection unit 12a in the sensor data storage unit 13a, inputs data of the same time simultaneously to each model, and determines the control content, respectively. Each control target equipment 20 is controlled based on the content.

  The learning unit 12c learns that the value of the data obtained by the sensor 21 after being controlled by the control unit 12b for the model is given a higher reward as the value is closer to a predetermined value. For example, the learning unit 12c is higher as the value of the data obtained by the sensor 21 after the control by the control unit 12b is closer to the average value between the predetermined upper limit and the predetermined lower limit set in advance. Give a reward. That is, as the learning unit 12c goes to the center of stability, it gives higher rewards in stages.

  Here, the flow of the optimum control process in the control device 10 according to the first embodiment will be described with reference to FIG. FIG. 5 is a diagram for explaining the flow of the optimum control process in the control device according to the first embodiment. As illustrated in FIG. 5, the control device 10 is installed in each of the real environments A to C as the states 1 to 3 of the real environment A, the states 1 to 3 of the real environment B, and the states 1 to 3 of the real environment C. The data of the sensor 21 is collected. Then, the control device 10 executes optimal control learning in parallel in each of the real environments A to C, and adopts an optimal model among the models applied to each of the real environments A to C.

  Here, the parallel processing of the optimal control learning will be specifically described using FIG. FIG. 6 is a diagram for explaining parallel processing of optimal control learning in the control device according to the first embodiment. As shown in FIG. 6, the control device 10 inputs data of the sensors 21 of the real environments A to C into the respective models, and acquires control contents for the control target equipment 20 as the outputs of the respective models. And control device 10 performs control of each real environment AC based on each control contents.

  And the control apparatus 10 acquires the control result of each real environment AC. Specifically, the control device 10 acquires the value of the data obtained by the sensor 21 of each of the real environments A to C after the control is performed. Subsequently, for each model, the control device 10 determines that the value of the data obtained by the sensor 21 after the control is performed is closer to the average value of the predetermined upper limit value and the predetermined lower limit value set in advance. Learn to get high rewards.

  Here, reward giving will be specifically described using the example of FIG. 7. FIG. 7 is a diagram for explaining reward provision in the control device according to the first embodiment. As shown in FIG. 7, the control device 10 appropriately determines, for each model, the value of the data obtained by the sensor 21 after the control is performed, as the upper limit (for example, the upper limit temperature) of the appropriate value to be determined. It is learned that a higher reward is given closer to the average value with the lower limit (for example, the lower limit temperature) of the value, and a lower reward is given as the distance is farther from the average value.

For example, the control device 10 calculates the value “x” of the data obtained by the sensor 21 after the control is performed, the predetermined upper limit “θ ₁ ” and the predetermined lower limit “θ ₂ ” which are set in advance. "-A (x- (theta) ₁ ) (x- (theta) ₂ )" is calculated as a method of calculating the reward to give using these. Here, “a” is a variable that can be arbitrarily changed.

  After performing such learning, the control device 10 adopts the optimal model among the models of the real environments A to C, and updates all the models to the optimal model. Note that the timing of updating may be performed each time learning is performed, or may be any timing. In addition, how to determine the optimal model may be determined automatically from predetermined conditions or may be determined manually.

[Processing procedure of control device]
Next, an example of a processing procedure by the control device 10 according to the first embodiment will be described using FIG. FIG. 8 is a flowchart showing an example of the flow of processing in the control device according to the first embodiment.

  As illustrated in FIG. 8, when collecting data of the sensor 21 in the control target facility 20 (Yes at Step S101), the collecting unit 12a stores the collected data in the sensor data storage unit 13a (Step S102).

  Then, the control unit 12b determines whether data at the same time in all real environments has been collected (step S103). As a result, when the control unit 12b does not collect data of the same time in all real environments (No at step S103), the control unit 12b returns to step S101 and continues the steps until collecting data of the same time in all real environments. The processing of S101 to step S103 is repeated.

  When the control unit 12b collects data at the same time in all real environments (Yes at step S103), the control unit 12b inputs the data of each sensor 21 collected by the collection unit 12a to each model (step S104). And control part 12b determines control contents, respectively (Step S105), and controls each controlled object equipment 20, respectively based on each control contents (Step S106).

  Subsequently, the learning unit 12c gives a higher reward to the model as the value of the data obtained by the sensor 21 after the control by the control unit 12b is closer to the predetermined value (step S107). For example, the learning unit 12c is higher as the value of the data obtained by the sensor 21 after the control by the control unit 12b is closer to the average value between the predetermined upper limit and the predetermined lower limit set in advance. Give a reward. Then, the control device 10 adopts an optimal model among the models applied to each of the real environments A to C (step S108).

[Effect of First Embodiment]
The control device 10 according to the first embodiment collects data acquired by the sensor 21 installed in the control target facility 20, and uses the collected data as an input to determine the control content of the control target facility 20. The control content is determined using a model, and the control target equipment 20 is controlled based on the control content. Then, the control device 10 learns about the model so that the value of the data obtained by the sensor 21 after the control is performed is given higher reward as the value is closer to a predetermined value. For this reason, in the control device 10, it is possible to easily and accurately execute the optimum control for the real environment. That is, in the control device 10, for example, by applying a model in which the reward becomes higher as the way of giving reinforcement learning goes to the center of stability, the optimal solution of reinforcement learning is obtained by increasing the value of more desirable reward. It is possible to increase the probability of being generated.

  In addition, the control device 10 collects data of each sensor 21 installed in each of a plurality of control target facilities 20, respectively, and inputs data of each collected sensor 21 to each model to determine control contents. And each control target equipment 20 based on each control content. Then, for each model, the control device 10 learns that the value of the data of each sensor 21 after the control is performed is given a higher reward as the value is closer to a predetermined value. Therefore, in the real environment, it also includes external factors on the real environment which can not be obtained in the virtual environment. In the control device 10, by learning the real environment in parallel, randomness is further enhanced, and learning in various situations can be performed.

  Further, the control device 10 stores the collected data of each sensor in the sensor data storage unit 13a, simultaneously inputs data of the same time simultaneously to each model to determine the control content, and based on each control content, Each control target equipment 20 is controlled. In the control device 10, when performing parallel processing in a real environment, it is possible to buffer temporal differences that occur and to learn simultaneously in a similar environment. For this reason, it becomes possible to absorb the time difference which arises from causes, such as distance of real environments being separated, using buffering, and to create the state of simultaneous execution.

Second Embodiment
In the first embodiment described above, the control device 10 buffers data of each sensor in the sensor data storage unit 13a, inputs data of the same time simultaneously into each model, determines the control content, and Although the case where each control object installation was controlled was explained based on control contents, it is not limited to this. For example, the control device simultaneously collects data of each sensor installed in each of a plurality of control target facilities, and simultaneously inputs the collected data of each sensor to the model to determine the control content, and the control content The control target equipment may be simultaneously controlled based on

  Therefore, in the following, the control device according to the second embodiment simultaneously collects data of each sensor installed in each of a plurality of control target facilities, and simultaneously inputs the collected data of each sensor to the model. The case where control content is determined respectively and each control object installation is simultaneously controlled based on the control content is explained. Descriptions of configurations and processes similar to those of the control device 10 according to the first embodiment will be omitted.

  The collection unit 12a of the control device according to the second embodiment simultaneously collects data of each sensor 21 installed in each of a plurality of control target facilities 20. Further, the control unit 12b of the control device according to the second embodiment simultaneously inputs the collected data of each sensor 21 to the model to determine the control content, and based on the control content, each control target The equipment 20 is simultaneously controlled.

  Here, an outline of processing of the control device according to the second embodiment will be described with reference to FIG. FIG. 9 is a diagram showing an outline of processing of a control device according to the second embodiment. As illustrated in FIG. 9, the control device according to the second embodiment includes the real environment A as states 1 to 3, the real environment B as states 1 to 3, and the real environment C as states 1 to 3 respectively. The data of the sensors 21 installed at ~ C are collected at the same time. Here, it is assumed that the respective real environments A to C are close in distance.

  And the control apparatus which concerns on 2nd Embodiment simultaneously inputs the data of the sensor 21 of each real environment AC to a model, and determines the control content with respect to the control object installation 20. FIG. And the control apparatus which concerns on 2nd Embodiment performs control of the control object installation 20 in each environment AC simultaneously based on the determined control content.

[Effect of Second Embodiment]
The control device according to the second embodiment simultaneously collects data of each sensor 21 installed in each of a plurality of control target facilities, and simultaneously inputs the collected data of each sensor 21 into a model to control content. Each of the control target facilities 20 is simultaneously controlled based on the control content. That is, in the control device according to the second embodiment, for example, it is assumed that there is some mutual influence in a real environment placed closer, and learning is performed simultaneously including them, which is more efficient. It is possible to make a structured learning.

[System configuration etc.]
Further, each component of each device illustrated is functionally conceptual, and does not necessarily have to be physically configured as illustrated. That is, the specific form of the distribution and integration of each device is not limited to the illustrated one, and all or a part thereof may be functionally or physically dispersed in any unit depending on various loads, usage conditions, etc. It can be integrated and configured. Furthermore, all or any part of each processing function performed in each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as wired logic hardware.

  Also, among the processes described in the present embodiment, all or part of the process described as being automatically performed can be manually performed, or the process described as being manually performed. All or part of can be performed automatically by a known method. In addition to the above, the processing procedures, control procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be arbitrarily changed unless otherwise specified.

[program]
In addition, it is also possible to create a program in which the processing executed by the control device described in the above embodiment is described in a language that can be executed by a computer. For example, it is also possible to create a control program in which the processing to be executed by the control device 10 according to the embodiment is described in a computer executable language. In this case, when the computer executes the control program, the same effect as that of the above embodiment can be obtained. Furthermore, the control program may be recorded in a computer readable recording medium, and the control program recorded in the recording medium may be read and executed by a computer to realize the same processing as that of the above embodiment.

  FIG. 10 is a diagram illustrating a computer that executes a control program. As illustrated in FIG. 10, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These units are connected by a bus 1080.

  The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012 as illustrated in FIG. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to the hard disk drive 1090 as illustrated in FIG. The disk drive interface 1040 is connected to the disk drive 1100 as illustrated in FIG. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120 as illustrated in FIG. The video adapter 1060 is connected to, for example, a display 1130 as illustrated in FIG.

  Here, as illustrated in FIG. 10, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the control program described above is stored in, for example, the hard disk drive 1090 as a program module in which an instruction to be executed by the computer 1000 is described.

  In addition, various data described in the above embodiment are stored as program data in, for example, the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and program data 1094 stored in the memory 1010 and the hard disk drive 1090 into the RAM 1012 as necessary, and executes various processing procedures.

  The program module 1093 and the program data 1094 related to the control program are not limited to being stored in the hard disk drive 1090, and may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Good. Alternatively, the program module 1093 and the program data 1094 related to the control program are stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.), and via the network interface 1070 It may be read by the CPU 1020.

  The above embodiments and the modifications thereof are included in the invention described in the claims and the equivalents thereof as well as included in the technology disclosed in the present application.

DESCRIPTION OF SYMBOLS 10 control apparatus 11 communication processing part 12 control part 12a collection part 12b control part 12c learning part 13 memory | storage part 13a sensor data storage part 20, 20A-20C control object installation 21 sensor 100 control system

Claims (7)

Collection means for collecting data acquired by a sensor installed in the control target facility;
Control means for determining control content using a model for determining control content of the control target facility using data collected by the collection means as input and controlling means for controlling the control target facility based on the control content When,
And a learning means for learning that the higher the value of the data obtained by the sensor after the control by the control means is given, the higher the reward is given to the model. Control device. The collecting means collects data of each sensor installed in each of a plurality of control target equipments,
The control means inputs the data of each sensor collected by the collection means into each model to determine the control content, and controls each control target facility based on the control content.
The learning means is characterized in that, for each model, the value of data of each sensor after being controlled by the control means learns such that a higher reward is given as the value of each sensor is closer to a predetermined value. The control device according to claim 1.   The control means stores the data of each sensor collected by the collection means in the storage unit, inputs data of the same time simultaneously to each model to determine the control contents respectively, and based on each control contents The control device according to claim 2, wherein each control target facility is controlled. The collection means simultaneously collects data of each sensor installed in each of a plurality of control target facilities,
The control means simultaneously inputs into the model the data of each sensor collected by the collection means to determine the control contents respectively, and simultaneously controls the respective control target facilities based on the control contents. The control device according to claim 1, wherein   The learning means has a higher reward as the value of the data obtained by the sensor after the control by the control means is closer to an average value of a predetermined upper limit and a predetermined lower limit. The control device according to claim 1, wherein the control device is provided. A control method implemented by the controller,
A collection step of collecting data acquired by a sensor installed in the control target facility;
A control step of determining control content using a model for determining control content of the control target facility using data collected by the collection step as an input, and controlling the control target facility based on the control content When,
And a learning step of learning that the value of the data obtained by the sensor after the control step is performed on the model is given higher reward as the value is closer to a predetermined value. Characteristic control method. A collection step of collecting data acquired by a sensor installed in the control target facility;
A control step of determining control content using a model for determining control content of the control target equipment using the data collected in the collection step as an input, and controlling the control target equipment based on the control content When,
Performing a learning step of learning that the value of the data obtained by the sensor after the control in the control step is performed on the model is higher as the value is closer to a predetermined value. A control program characterized by

Images