JP7484868B2 - Operation system, operation method, and operation program, as well as evaluation model generation device, evaluation model generation method, and evaluation model generation program - Google Patents

Description

The present invention relates to an operation system, an operation method, and an operation program, as well as an evaluation model generation device, an evaluation model generation method, and an evaluation model generation program.

Patent Document 1 states that "in response to input of measurement data, a learning process for a first model is executed to output recommended control parameters indicating a first type of control content recommended for increasing a reward value determined by a preset reward function."
[Prior Art Literature]
[Patent Documents]
[Patent Document 1] JP2021-086283
[Patent Document 2] JP2020-027556A
[Patent Document 3] JP2019-020885A

In a first aspect of the present invention, an operation system is provided. The operation system may include an evaluation model generation device that generates, by machine learning, an evaluation model that outputs an index that evaluates a state of the equipment for a target goal based on an operation goal of the equipment and the state of the equipment. The operation system may include an operation model generation device that generates, by reinforcement learning using the output of the evaluation model as at least a part of a reward, an operation model that outputs an action according to the state of the equipment. The operation system may include a control device that applies an operation amount based on the action output by the operation model according to the state of the equipment to a control target of the equipment.

The evaluation model generation device may update the evaluation model based on the state of the equipment when the control object is controlled using the operation model.

The operation model generation device may update the operation model by reinforcement learning using the output of the updated evaluation model as at least a part of the reward.

The control device may control the control object using the updated operation model.

The evaluation model generation device may include an operation goal acquisition unit that acquires the operation goal. The evaluation model generation device may include a state data acquisition unit that acquires state data indicating a state of the equipment. The evaluation model generation device may include a correlation data generation unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data, based on the operation goal. The evaluation model generation device may include a labeling unit that labels the correlation data using a labeling model. The evaluation model generation device may include an evaluation model generation unit that generates the evaluation model using the labeled correlation data.

The evaluation model generation device may further include an evaluation model determination unit that determines the validity of the evaluation model.

The evaluation model generation device may further include an evaluation model output unit that outputs the evaluation model when the evaluation model is determined to be valid.

The evaluation model generation device may further include a labeling model update unit that updates the labeling model when the evaluation model is determined to be valid.

The evaluation model generating device may further include a teacher label acquiring unit that acquires teacher labels for at least a portion of the correlation data. The labeling model updating unit may generate an updated labeling model separately from the initial labeling model generated based on the teacher labels.

In a second aspect of the present invention, an operation method is provided. The operation method may include generating, by machine learning, an evaluation model that outputs an index that evaluates a state of the equipment for a target target based on an operation target of the equipment and a state of the equipment. The operation method may include generating, by reinforcement learning using an output of the evaluation model as at least a part of a reward, an operation model that outputs an action according to the state of the equipment. The operation method may include providing, to a control target of the equipment, an operation amount based on the action output by the operation model according to the state of the equipment.

In a third aspect of the present invention, an operation program is provided. The operation program may be executed by a computer. The operation program may cause the computer to function as an evaluation model generation device that generates, by machine learning, an evaluation model that outputs an index that evaluates a state of the equipment for a target goal based on an operational goal of the equipment and the state of the equipment. The operation program may cause the computer to function as an operation model generation device that generates, by reinforcement learning using the output of the evaluation model as at least a part of a reward, an operation model that outputs an action according to the state of the equipment. The operation program may cause the computer to function as a control device that gives an operation amount based on the action output by the operation model according to the state of the equipment to a control target in the equipment.

In a fourth aspect of the present invention, an evaluation model generation device is provided. The evaluation model generation device may include an operation target acquisition unit that acquires an operation target in the facility. The evaluation model generation device may include a state data acquisition unit that acquires state data indicating a state in the facility. The evaluation model generation device may include a correlation data generation unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data, based on the operation target. The evaluation model generation device may include a labeling unit that labels the correlation data using a labeling model. The evaluation model generation device may include an evaluation model generation unit that generates an evaluation model that outputs an index that evaluates the state of the facility for a target target based on the operation target in the facility and the state of the facility, using the labeled correlation data.

In a fifth aspect of the present invention, there is provided an evaluation model generation method. The evaluation model generation method may include acquiring an operation target for a facility. The evaluation model generation method may include acquiring status data indicating a status of the facility. The evaluation model generation method may include generating correlation data indicating at least one of a correlation between at least one physical quantity included in the status data and time and a correlation between at least two physical quantities included in the status data, based on the operation target. The evaluation model generation method may include labeling the correlation data using a labeling model. The evaluation model generation method may include generating an evaluation model that outputs an index that evaluates a status of the facility for a target target based on the operation target for the facility and the status of the facility, using the labeled correlation data.

In a sixth aspect of the present invention, an evaluation model generation program is provided. The evaluation model generation program may be executed by a computer. The evaluation model generation program may cause the computer to function as an operation target acquisition unit that acquires an operation target in the facility. The evaluation model generation program may cause the computer to function as a status data acquisition unit that acquires status data indicating a status in the facility. The evaluation model generation program may cause the computer to function as a correlation data generation unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the status data and time and a correlation between at least two physical quantities included in the status data, based on the operation target. The evaluation model generation program may cause the computer to function as a labeling unit that labels the correlation data using a labeling model. The evaluation model generation program may cause the computer to function as an evaluation model generation unit that generates an evaluation model that outputs an index that evaluates the status of the facility for a target target based on the operation target in the facility and the status of the facility, using the labeled correlation data.

Note that the above summary of the invention does not list all of the features of the present invention. Also, subcombinations of these features may also be inventions.

An example of a block diagram of an operation system 100 according to this embodiment is shown together with a facility 10 and a headquarters 20. 1 shows an example of a block diagram of an evaluation model generating device 200 in an operation system 100 according to this embodiment. 13 shows an example of training data that the labeling unit 220 inputs to the learning device when generating an initial labeling model. An example of a design example of a learning module used by the labeling unit 220 will be described below. 13 shows another example of a design of a learning module used by the labeling unit 220. 13 shows an example of correlation data to which labels have not been assigned by the labeling unit 220. 2 shows an example of an I/O list in the facility 10. 1 shows an example of a segment diagram for the facility 10. 13 shows an example of labeling data output by the labeling data output unit 222. 13 shows an example of a processing flow in a labeling function unit 210 of the evaluation model generating device 200. 2 shows an example of a block diagram of an evaluation model generating unit 250. 1 shows an example of an output of the evaluation model. 13 shows another example of an output of the evaluation model. 13 shows an example of a processing flow in the machine learning function unit 230 of the evaluation model generating device 200. 1 shows an example of a block diagram of an operation model generating device 300 in an operation system 100 according to this embodiment. 1 shows an example of an operation model generated by the operation model generating device 300. 13 shows an example of a behavior decision table. 1 shows an example of a process flow in an operation model generating device 300. 13 shows an example of a reinforcement learning flow in the operation model generation unit 316. 1 shows an example of a block diagram of a control device 400 in an operation system 100 according to this embodiment. 1 shows an example of a process flow in the control device 400. An example of a computer 9900 is shown in which aspects of the present invention may be embodied in whole or in part.

The present invention will be described below through embodiments of the invention, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention.

Figure 1 shows an example of a block diagram of an operating system 100 according to this embodiment, together with equipment 10 and headquarters 20. Note that these blocks are functionally separated functional blocks and do not necessarily correspond to the actual device configuration. In other words, just because something is shown as one block in this diagram does not necessarily mean that it is made up of one device. Also, just because something is shown as separate blocks in this diagram does not necessarily mean that they are made up of separate devices. The same applies to the block diagrams that follow.

The equipment 10 is a facility or device equipped with an actuator or other control target. For example, the equipment 10 may be a plant, or a composite device that combines multiple devices. Examples of plants include industrial plants such as chemical and bio plants, as well as plants that manage and control wellheads and surrounding areas of gas and oil fields, plants that manage and control hydroelectric, thermal, and nuclear power generation, plants that manage and control environmental power generation such as solar and wind power, and plants that manage and control water supply and sewage systems and dams.

The headquarters 20 is the central institution for operating the facility 10, and may be, for example, the head office of the business operator. For example, such a headquarters 20 may house the management team in charge of the business operator's management. The management team specifies operational goals to the operation system 100. Here, the operational goals are goals set for operating the facility 10, and may include, for example, target items and values.

In the operation system 100 according to this embodiment, an evaluation model that outputs an index that evaluates the state of the equipment 10 is generated by machine learning, and an operation model is generated by reinforcement learning in which the output of the evaluation model is used as at least a part of the reward. Then, in the operation system 100 according to this embodiment, the operation model generated in this manner is used to control the controlled object in the equipment 10.

The operation system 100 includes an evaluation model generation device 200, an operation model generation device 300, and a control device 400.

The evaluation model generating device 200 uses machine learning to generate an evaluation model that outputs an index that evaluates the state of the facility 10 for a target goal based on the operational goal of the facility 10 and the state of the facility 10. The evaluation model generating device 200 supplies the generated evaluation model to the operation model generating device 300.

The operation model generating device 300 generates an operation model that outputs an action according to the state of the facility 10 by reinforcement learning using the output of the evaluation model generated by the evaluation model generating device 200 as at least a part of the reward. The operation model generating device 300 supplies the generated operation model to the control device 400.

The control device 400 applies to the controlled object in the facility 10 an operation amount based on the behavior output by the operation model generated by the operation model generating device 300 according to the state of the facility 10. In other words, the control device 400 functions as an AI (Artificial Intelligence) controller using the operation model generated by reinforcement learning.

At this time, the evaluation model generating device 200 updates the evaluation model based on the state of the equipment 10 when the controlled object is controlled using the operation model. In response to this, the operation model generating device 300 updates the operation model by reinforcement learning in which the output of the updated evaluation model is at least a part of the reward. Then, the control device 400 controls the controlled object using the updated operation model.

In this way, in the operation system 100 according to this embodiment, the AI automatically finds bottlenecks (potential faults) in the operation and generates indicators for improvement as an evaluation model. Then, the AI performs trial and error based on the given indicators to generate an operation model that indicates a better operation method. Then, the AI controller uses the operation model to AI-control the controlled object. As a result, the operation system 100 according to this embodiment provides an environment in which the equipment 10 can be autonomously controlled using AI technology. Then, the operation system 100 according to this embodiment updates the evaluation model and the operation model based on the state of the equipment under such AI control, and uses the updated operation model to AI-control the controlled object. As a result, the operation system 100 according to this embodiment can autonomously run a loop to improve the operation of the equipment 10. This will be described in detail.

Figure 2 shows an example of a block diagram of the evaluation model generating device 200 in the operation system 100 according to this embodiment. The evaluation model generating device 200 may be a computer such as a PC (personal computer), a tablet computer, a smartphone, a workstation, a server computer, or a general-purpose computer, or may be a computer system to which multiple computers are connected. Such a computer system is also a computer in the broad sense. The evaluation model generating device 200 may be implemented by one or more virtual computer environments that can be executed within a computer. Alternatively, the evaluation model generating device 200 may be a dedicated computer designed for generating an evaluation model, or may be dedicated hardware realized by a dedicated circuit. In addition, if the evaluation model generating device 200 is connectable to the Internet, it may be realized by cloud computing.

The evaluation model generating device 200 includes a labeling function unit 210 and a machine learning function unit 230. Note that, in this figure, a case where the labeling function unit 210 and the machine learning function unit 230 are configured as an integrated device is shown as an example, but this is not limited thereto. The labeling function unit 210 and the machine learning function unit 230 may be configured as separate devices.

The labeling function unit 210 includes an operation target acquisition unit 212, a state data acquisition unit 214, a correlation data generation unit 216, a teacher label acquisition unit 218, a labeling unit 220, a labeling data output unit 222, and a labeling model update unit 224. That is, the evaluation model generation device 200 includes an operation target acquisition unit 212, a state data acquisition unit 214, a correlation data generation unit 216, a teacher label acquisition unit 218, a labeling unit 220, a labeling data output unit 222, and a labeling model update unit 224.

The operation target acquisition unit 212 acquires the operation targets for the facility 10. For example, the operation target acquisition unit 212 acquires the operation targets from the headquarters 20 via a network. However, this is not limited to this. The operation target acquisition unit 212 may acquire the operation targets from another device, may acquire them via various memory devices, or may acquire them via user input. The operation target acquisition unit 212 supplies the acquired operation targets to the correlation data generation unit 216.

The status data acquisition unit 214 acquires status data indicating the status of the equipment 10. For example, the status data acquisition unit 214 acquires various physical quantities measured by various sensors provided in the equipment 10 as status data in chronological order from the equipment 10 via a network. However, this is not limited to this. The status data acquisition unit 214 may acquire the status data from another device, may acquire the status data via various memory devices, or may acquire the status data via user input. The status data acquisition unit 214 supplies the acquired status data to the correlation data generation unit 216.

The correlation data generating unit 216 generates correlation data indicating at least one of the correlation between at least one physical quantity and time included in the state data acquired by the state data acquiring unit 214, and the correlation between at least two physical quantities included in the state data, based on the operation target acquired by the operation target acquiring unit 212. At this time, the correlation data generating unit 216 may generate correlation data including a graph image that graphs these correlations. The correlation data generating unit 216 supplies the generated correlation data to the labeling unit 220.

The teacher label acquisition unit 218 acquires teacher labels for at least a portion of the correlation data generated by the correlation data generation unit 216. For example, the teacher label acquisition unit 218 acquires teacher labels via user (expert, etc.) input. However, this is not limited to this. The teacher label acquisition unit 218 may acquire the teacher labels from another device, via a network, or via various memory devices. The teacher label acquisition unit 218 supplies the acquired teacher labels to the labeling unit 220.

The labeling unit 220 labels the correlation data generated by the correlation data generation unit 216 using a labeling model generated based on the teacher label acquired by the teacher label acquisition unit 218. For example, the labeling unit 220 generates an initial labeling model by inputting data to which the teacher label acquired by the teacher label acquisition unit 218 is attached to at least a part of the correlation data generated by the correlation data generation unit 216 as teacher data into a learning device. Then, the labeling unit 220 labels the unlabeled correlation data using the initial labeling model and an update labeling model described later. In the above description, the case where the correlation data in the teacher data is generated by the correlation data generation unit 216 is shown as an example, but is not limited to this. The correlation data in the teacher data may be generated by an expert or the like. In this case, the teacher label acquisition unit 218 may acquire the teacher data itself to which the teacher label is attached, instead of acquiring the teacher label from the expert or the like, and supply it to the labeling unit 220. The labeling unit 220 supplies the labeled correlation data to the labeling data output unit 222.

The labeling data output unit 222 generates labeling data based on the correlation data labeled by the labeling unit 220 and the sensor data (measured values of physical quantities). The labeling data output unit 222 outputs the generated labeling data to the machine learning function unit 230.

The labeling model update unit 224 obtains the judgment result of the evaluation model described below. Then, when it is judged that the evaluation model is appropriate, the labeling model update unit 224 updates the labeling model. At this time, the labeling model update unit 224 may generate an updated labeling model separately from the initial labeling model generated based on the teacher label.

The machine learning function unit 230 includes a labeling data acquisition unit 240, an evaluation model generation unit 250, an evaluation model determination unit 260, and an evaluation model output unit 270. That is, the evaluation model generation device 200 includes a labeling data acquisition unit 240, an evaluation model generation unit 250, an evaluation model determination unit 260, and an evaluation model output unit 270.

The labeling data acquisition unit 240 acquires the labeling data output by the labeling data output unit 222. The labeling data acquisition unit 240 supplies the acquired labeling data to the evaluation model generation unit 250.

The evaluation model generation unit 250 uses the labeling data acquired by the labeling data acquisition unit 240, i.e., the labeling data generated based on the correlation data labeled by the labeling unit 220 and the sensor data, to generate an evaluation model that outputs an index that evaluates the state of the equipment 10 for a target target based on the operation target of the equipment 10 and the state of the equipment 10. The evaluation model generation unit 250 supplies the generated evaluation model to the evaluation model determination unit 260 and the evaluation model output unit 270.

The evaluation model determination unit 260 determines the validity of the evaluation model generated by the evaluation model generation unit 250. If the evaluation model determination unit 260 determines that the evaluation model is valid, it notifies the labeling model update unit 224 and the evaluation model output unit 270 of this fact. In response to this, the labeling model update unit 224 updates the labeling model.

If the evaluation model is determined to be valid, the evaluation model output unit 270 outputs the evaluation model to the operation model generation device 300.

First, we will explain the details of the processing in the labeling function unit 210 of the evaluation model generation device 200 in detail using example data and flow charts.

Figure 3 shows an example of teacher data that the labeling unit 220 inputs to the learning device when generating an initial labeling model. As shown in this figure, the teacher data input to the learning device may include "operation goal", "target", "tag information 1", "tag information 2", "graph image", and "label". Here, "operation goal" may include "target item" indicating the target item and "target value" for the target item. Furthermore, "target" may include "category" of the target segment and "control loop" to be targeted. Furthermore, "tag information 1" may include "name", "type of physical quantity", "minimum value", "maximum value", and "unit" of tag 1. Furthermore, "tag information 2" may include "name", "type of physical quantity", "minimum value", "maximum value", and "unit" of tag 2.

The "graph image" may be an image that graphs the correlation between tag 1 and tag 2. For example, "xxx1.jpg" and "xxx2.jpg" may be time-series images that are graphed with the concentration of tag 1 on the vertical axis and the time of tag 2 on the horizontal axis. Similarly, "xxx3.jpg" and "xxx4.jpg" may be time-series images that are graphed with the acceleration of tag 1 on the vertical axis and the time of tag 2 on the horizontal axis. Furthermore, "xxx5.jpg" and "xxx6.jpg" may be distribution (scatter) images that are graphed with the temperature of tag 1 on the vertical axis and the concentration of tag 2 on the horizontal axis. In other words, the vertical axis in the "graph image" is defined by "tag information 1", and the horizontal axis in the "graph image" is defined by "tag information 2".

Furthermore, a "label" is a teacher label assigned by an expert or the like. The labeling unit 220 may generate an initial labeling model by inputting teacher data including such items into a learning device. Note that in this figure, an example is shown in which both an "OK" label and an "NG" label are used as labels, but this is not limited to this. Only an "OK" label may be used as a label, or only an "NG" label may be used as a label. In particular, in applications for finding bottlenecks in operations, it is preferable to use at least an "NG" label as a label.

Figure 4 shows an example of a design example of a learning device used by the labeling unit 220. The labeling unit 220 may use various learning algorithms as such a learning device, and as an example, Deep Learning may be used. The learning device may have a functional unit for classifying operation goals, a functional unit for classifying objects, a functional unit for classifying physical quantities (tag information), and a functional unit for classifying graph images. In this case, the learning device may be designed to have layers that respond to each item, such as a target classification layer that classifies operation goals, an object classification layer that classifies objects, a physical quantity classification layer that classifies physical quantities, and a graph classification layer that classifies graph images, as shown in this figure.

Figure 5 shows another example of the design of a learning machine used by the labeling unit 220. Instead of providing a layer in the learning machine that responds to each item as in Figure 4, the learning machine may be designed to have a classifiable model for each layer, such as a target classification model that classifies operational goals, an object classification model that classifies objects, a physical quantity classification model that classifies physical quantities, and a graph classification model that classifies graph images, as shown in this figure.

Figure 6 shows an example of unlabeled correlation data that is the target of labeling by the labeling unit 220. As shown in this figure, the unlabeled correlation data may include "operational goal," "target," "tag information 1," "tag information 2," and "graph image." Each of these items may be the same as each of the items in the teacher data shown in Figure 3, so a description of them will be omitted here.

Figure 7 shows an example of an I/O list in the facility 10. The I/O list is a list of information about each device installed in the facility 10. Such an I/O list is referred to as appropriate when generating correlation data and labeling data.

Figure 8 shows an example of a segment diagram for equipment 10. A segment diagram is a diagram showing the configuration of segments in equipment 10. As with I/O lists, such segment diagrams are referred to as appropriate when generating correlation data and labeling data.

Figure 9 shows an example of labeling data output by the labeling data output unit 222. As shown in this figure, the labeling data may include labeling data with an "OK" label and labeling data with an "NG" label. Each such labeling data includes sensor data corresponding to a sensor ID (tag name). In other words, the labeling data indicates what values of measurements from one or more sensors are labeled with an "OK" label and what values are labeled with an "NG" label.

The labeling function unit 210 generates and outputs labeling data, for example as shown in this figure. This will be explained in detail using a flow chart.

Figure 10 shows an example of a processing flow in the labeling function unit 210 of the evaluation model generating device 200. The labeling function unit 210 of the evaluation model generating device 200 may execute the labeling process, for example, according to the flow shown in this figure.

In step S1002, the evaluation model generating device 200 acquires the operation targets for the facility 10. For example, the operation target acquisition unit 212 acquires the operation targets from the headquarters 20 via a network. Such operation targets may include, for example, a target item indicating a target item and a target value for the target item. As an example, when the facility 10 is a plant, the operation target acquisition unit 212 may acquire a plant KPI (Key Performance Indicator) as an operation target. The operation target acquisition unit 212 supplies the acquired operation targets to the correlation data generating unit 216.

In step S1004, the evaluation model generating device 200 acquires state data indicating the state of the facility 10. For example, the state data acquiring unit 214 acquires various physical quantities measured by various sensors provided in the facility 10 as state data from the facility 10 in time series via a network. Such physical quantities may include, for example, temperature, concentration, acceleration, and pressure at various locations in the facility 10. The state data acquiring unit 214 supplies the acquired state data to the correlation data generating unit 216.

In step S1006, the evaluation model generating device 200 generates correlation data indicating at least one of the correlation between at least one physical quantity included in the state data and time, and the correlation between at least two physical quantities included in the state data, based on the operation target. For example, the correlation data generating unit 216 generates correlation data indicating at least one of the correlation between at least one physical quantity included in the state data acquired in step S1004 and time, and the correlation between at least two physical quantities included in the state data, based on the operation target acquired in step S1002. As an example, the correlation data generating unit 216 inputs the "operation target" item based on the operation target acquired in step S1002. In addition, the correlation data generating unit 216 comprehensively inputs the "target" item, "tag information 1", and "tag information 2" items by referring to the I/O list shown in FIG. 7 and the segment diagram shown in FIG. 8. Then, the correlation data generation unit 216 defines the horizontal axis by "tag information 1" and the vertical axis by "tag information 2" to graph the correlation between tags 1 and 2, and inputs the graphed image (e.g., a time series image or a distribution map image) into the "graph image" field. In this way, the correlation data generation unit 216 generates, for example, correlation data other than the "teacher label" field of the teacher data shown in FIG. 3, and unlabeled correlation data shown in FIG. 6. The correlation data generation unit 216 supplies the generated correlation data to the labeling unit 220.

In step S1008, the evaluation model generating device 200 determines whether or not a labeling model exists. For example, the evaluation model generating device 200 determines whether or not a labeling model for labeling unlabeled correlation data has already been generated. If it is determined that a labeling model does not exist (has not been generated) (No), the evaluation model generating device 200 proceeds to step S1010.

In step S1010, the evaluation model generating device 200 acquires teacher labels for at least a portion of the correlation data. For example, the teacher label acquisition unit 218 accepts input of teacher labels in response to the display of the correlation data generated in step S1006. In response to this, the expert or the like judges the correlation data based on the graph image, and assigns an "OK" label to correlation data that is judged to be "no problem", and an "NG" label to correlation data that is judged to be "problematic/suspicious". In other words, the expert or the like focuses on which data and labels where the problem is. The teacher label acquisition unit 218 acquires teacher labels via user input, for example, in this way. The teacher label acquisition unit 218 supplies the acquired teacher labels to the labeling unit 220.

In step S1012, the evaluation model generating device 200 generates an initial labeling model. For example, the labeling unit 220 generates teacher data such as that shown in FIG. 3 by attaching the teacher label acquired in step S1010 to at least a part of the correlation data generated in step S1006. The labeling unit 220 then generates an initial labeling model by inputting such teacher data to a learning device such as that shown in FIG. 4 or FIG. 5. The evaluation model generating device 200 then returns the process to step S1002 and continues the flow. From this point on, the evaluation model generating device 200 uses the labeling model generated in this way to label correlation data to which no labels have been assigned.

After the initial labeling model is generated in step S1012, in step S1008, the evaluation model generation device 200 determines that a labeling model exists (has already been generated), and proceeds to step S1014.

In step S1014, the evaluation model generating device 200 labels the correlation data generated in step S1006 using the labeling model generated based on the teacher label acquired in step S1010. At this time, the range of data used in the graph image may be linked based on the operation target, or may be narrowed down using the data used when generating the initial labeling model. For example, the labeling unit 220 inputs unlabeled correlation data to the initial labeling model generated in step S1012 and the labeling model for update. In response to this, the labeling model classifies the operation target, classifies the object, classifies the physical quantity, and classifies the graph image. That is, the labeling model identifies data with common or similar operation targets, identifies data with common or similar objects, and identifies data with common or similar physical quantities. The labeling model then compares the graph images thus identified, and assigns an "OK" label to unlabeled correlation data if the graph image is similar to a graph image labeled "OK", and an "NG" label if the graph image is similar to a graph image labeled "NG". In this case, the labeling model may turn the data into an image and determine the similarity in the form of a graph, or may use a recurrent neural network (RNN) or a long short term memory (LSTM) to determine whether the data waveforms are similar.

The labeling unit 220 may attach an "OK" label to correlation data that the initial labeling model classifies as "OK" and that the updated labeling model classifies as "OK". Similarly, the labeling unit 220 may attach an "NG" label to correlation data that the initial labeling model classifies as "NG" and that the updated labeling model classifies as "NG". That is, the labeling unit 220 may label the correlation data by the logical product of the classification result of the initial labeling model and the classification result of the updated labeling model. However, it is also conceivable that the initial labeling model and the updated labeling model show different classification results. In such a case, the labeling unit 220 may give priority to the classification result of the initial labeling model. Alternatively, the labeling unit 220 may give priority to the classification result of the updated labeling model. Alternatively, the labeling unit 220 may label the correlation data by performing a logical sum of the classification result of the initial labeling model and the classification result of the updated labeling model. The labeling unit 220 supplies the correlation data labeled in this manner to the labeling data output unit 222.

In step S1016, the evaluation model generating device 200 outputs the labeling data. For example, the labeling data output unit 222 generates labeling data such as that shown in FIG. 9 based on the correlation data and the sensor data labeled in step S1014. Then, the labeling data output unit 222 outputs the generated labeling data to the machine learning function unit 230.

In step S1018, the evaluation model generating device 200 determines whether a judgment result indicating that the evaluation model is valid has been obtained. If it is determined that a judgment result has not been obtained (No), the evaluation model generating device 200 ends the flow. On the other hand, if it is determined that a judgment result has been obtained (Yes), the evaluation model generating device 200 advances the process to step S1020.

In step S1020, the evaluation model generating device 200 updates the labeling model when it is determined that a judgment result indicating that the evaluation model is valid has been obtained. For example, the labeling model updating unit 224 generates an updated labeling model separately from the initial labeling model generated based on the teacher label, and updates the updated labeling model. In general, the teacher label assigned by an expert or the like is more likely to be accurate than the label assigned by the learning device. Therefore, the labeling model updating unit 224 does not update the initial labeling model, but updates the updated labeling model generated separately from the initial labeling model, thereby preventing the influence of the teacher label assigned by an expert or the like from gradually weakening. However, this is not limited to this, and does not exclude the case where the labeling model updating unit 224 updates the initial labeling model.

The labeling function unit 210 of the evaluation model generating device 200 may perform the labeling process in this manner, for example. Next, the details of the processing in the machine learning function unit 230 of the evaluation model generating device 200 will be explained in detail using example data and a flow.

FIG. 11 shows an example of a block diagram of the evaluation model generation unit 250. The evaluation model generation unit 250 has multiple learning units 252a, 252b, ..., learning unit 252n (collectively referred to as "learning units 252"), and the multiple learning units 252 perform learning in parallel. The learning unit 252a includes a preprocessing unit 254a and a machine learning unit 256a. Similarly, the learning unit 252b includes a preprocessing unit 254b and a machine learning unit 256b. Similarly, the learning unit 252c includes a preprocessing unit 254c and a machine learning unit 256c. Here, the preprocessing units 254a, 254b, ..., 254n are collectively referred to as "preprocessing units 254". Also, the machine learning units 256a, 256b, ..., 256n are collectively referred to as "machine learning units 256".

The preprocessing unit 254 preprocesses the labeling data. For example, the preprocessing unit 254 performs standardization, normalization, low-pass filtering, high-pass filtering, principal component analysis, and other processes on the labeling data. The preprocessing unit 254 supplies the preprocessed labeling data to the machine learning unit 256.

The machine learning unit 256 uses the labeling data preprocessed by the preprocessing unit 254 as learning data and generates an evaluation model using a machine learning algorithm.

In this way, the evaluation model generation unit 250 has multiple learning units 252, each of which includes a machine learning unit 256. As a result, the evaluation model generation unit 250 generates multiple evaluation models using the respective learning units 252. At this time, it is preferable that at least one of the processing contents of the preprocessing unit 254 and the algorithm of the machine learning unit 256 is different in the multiple learning units 252. As a result, the evaluation model generation unit 250 can generate multiple evaluation models that are different from each other. The evaluation model generated in this way may output a numerical value indicating whether the sensor data is close to an OK teacher or a NG teacher, for example.

Figure 12 shows an example of the output of an evaluation model. In this figure, as an example, the output of an evaluation model with the goal of improving the quality of the target quality in the operational objectives is shown. In this figure, the vertical axis represents the health index. As an example, such an evaluation model outputs a health index = 0 when it is predicted that the target will be the same value as the target value. The more the target is predicted to be better than the target value, the more the evaluation model outputs a value greater than 0, and the more the target is predicted to be worse than the target value, the more the evaluation model outputs a value smaller than 0.

In addition, the horizontal axis in this figure represents time. As an example, the first half of the horizontal axis in this figure shows a case where data from a period labeled "OK" is input into the evaluation model. In this case, the evaluation model can be said to be valid because the higher the rate at which values greater than or equal to 0 are output during that period, the higher the accuracy rate with the labeling. Similarly, the second half of the horizontal axis in this figure shows a case where data from a period labeled "NG" is input into the evaluation model. In this case, the higher the rate at which values less than 0 are output during that period, the higher the accuracy rate with the labeling.

Figure 13 shows another example of the output of the evaluation model. In this figure, as an example, the output of an evaluation model with the goal of extending catalyst use (cost cutting) as a target in the operational goal is shown. In this figure, the vertical axis represents the health index. Also, in this figure, the horizontal axis represents time. Generally, an event occurs in which the catalyst gradually decreases over time. Therefore, as shown by the arrow in this figure, the evaluation model can be said to be valid because the more monotonically decreasing the output is, the more accurately the event is captured. The machine learning function unit 230 of the evaluation model generation device 200 generates an evaluation model that can output such a result, for example.

Figure 14 shows an example of a processing flow in the machine learning function unit 230 of the evaluation model generating device 200. The machine learning function unit 230 of the evaluation model generating device 200 may execute the process of generating an evaluation model by machine learning, for example, according to the flow shown in this figure.

In step S1410, the evaluation model generating device 200 acquires labeling data. For example, the labeling data acquisition unit 240 acquires the labeling data output in step S1016 in the flow of FIG. 10. As an example, the labeling data acquisition unit 240 may acquire labeling data such as that shown in FIG. 9.

In step S1420, the evaluation model generating device 200 samples the judgment data. For example, the labeling data acquiring unit 240 samples a portion of the labeling data acquired in step S1410 as judgment data. As an example, the labeling data acquiring unit 240 may randomly sample judgment data from the acquired labeling data. Alternatively, the labeling data acquiring unit 240 may sample the first half of the acquired labeling data as machine learning data and the second half as judgment data. The labeling data acquiring unit 240 supplies the sampled judgment data to the evaluation model determining unit 260. In addition, the labeling data acquiring unit 240 supplies the remaining labeling data to the evaluation model generating unit 250 as machine learning data.

In step S1430, the evaluation model generating device 200 uses the labeling data supplied in step S1420, i.e., the labeling data generated based on the correlation data and sensor data labeled in step S1014 in the flow of FIG. 10, to generate an evaluation model that outputs an index that evaluates the state of the equipment 10 for a target target based on the operational target of the equipment 10 and the state of the equipment 10.

More specifically, in step S1432, the evaluation model generating device 200 preprocesses the labeling data. For example, the preprocessing units 254a, 254b, ..., 254n each perform standardization, normalization, low-pass filtering, high-pass filtering, and principal component analysis on the labeling data supplied in step S1420. At this time, the preprocessing units 254a, 254b, ..., 254n may each perform different processing. The preprocessing units 254a, 254b, ..., 254n each supply the preprocessed labeling data to the machine learning units 256a, 256b, ..., 256n.

In step S1434, the evaluation model generating device 200 performs machine learning. For example, the machine learning units 256a, 256b, ..., 256n each generate an evaluation model using a machine learning algorithm, using the labeling data preprocessed in step S1432 as learning data. At this time, the machine learning units 256a, 256b, ..., 256n may perform machine learning using different algorithms. Therefore, the learning units 252a, 252b, ..., 252n may generate multiple different evaluation models by performing processing in which at least one of the preprocessing content and the machine learning algorithm is different. As an example, such an evaluation model may be a model that outputs a result such as that shown in FIG. 12 or FIG. 13 in response to input data. The evaluation model generating unit 250 supplies the evaluation model generated in this manner to the evaluation model determining unit 260 and the evaluation model output unit 270.

In step S1440, the evaluation model generating device 200 judges the validity of the evaluation model. For example, the evaluation model judging unit 260 judges the validity of each of the multiple evaluation models by inputting the judgment data sampled in step S1420 into each of the multiple evaluation models generated by the multiple learning units 252 in the evaluation model generating unit 250.

In this case, as an example, if the generated evaluation model is an evaluation model that outputs a result as shown in FIG. 12, when the evaluation model outputs a value of 0 or more in response to input of judgment data labeled with an "OK" label (correct answer rate for the "OK" label) exceeds a predetermined threshold, the evaluation model determination unit 260 may determine that the evaluation model is valid. Alternatively, or in addition, if the generated evaluation model is an evaluation model that outputs a result as shown in FIG. 12, when the evaluation model outputs a value of less than 0 in response to input of judgment data labeled with an "NG" label (correct answer rate for the "NG" label) exceeds a predetermined threshold, the evaluation model determination unit 260 may determine that the evaluation model is valid. As another example, if the generated evaluation model is an evaluation model that outputs a result as shown in FIG. 13, when the evaluation model outputs a result having monotonically decreasing properties in response to input of judgment data, the evaluation model determination unit 260 may determine that the evaluation model is valid.

If it is determined that none of the multiple evaluation models generated are valid (No), the evaluation model generating device 200 returns the process to step S1410 and continues the flow. On the other hand, if it is determined that at least one of the multiple evaluation models generated is valid (Yes), the evaluation model generating device 200 advances the process to step S1450.

In step S1450, the evaluation model generating device 200 feeds back the judgment result. For example, the evaluation model judgment unit 260 notifies the labeling model update unit 224 that it has judged the evaluation model to be valid, together with information identifying the labeling data used to obtain the judgment, i.e., the labeling data acquired in step S1410. In response to this, the labeling model update unit 224 updates the labeling model. In addition, the evaluation model judgment unit 260 notifies the evaluation model output unit 270 that it has judged the evaluation model to be valid, together with information identifying the evaluation model judged to be valid.

In step S1460, the evaluation model generating device 200 outputs the evaluation model. For example, if the evaluation model output unit 270 is notified in step S1450 that the evaluation model is valid, the evaluation model output unit 270 outputs the evaluation model that is determined to be valid to the operation model generating device 300.

The machine learning function unit 230 of the evaluation model generating device 200 may execute the process of generating an evaluation model by machine learning, for example, in this manner. That is, the evaluation model generating device 200 executes the labeling process according to the flow of FIG. 10, and executes the process of generating an evaluation model according to the flow of FIG. 14. Then, the evaluation model generating device 200 outputs the generated evaluation model to the operation model generating device 300.

Figure 15 shows an example of a block diagram of the operation model generating device 300 in the operation system 100 according to this embodiment. Like the evaluation model generating device 200, the operation model generating device 300 may be a computer or a computer system to which multiple computers are connected. The operation model generating device 300 may also be implemented by a virtual computer environment in which one or more programs can be executed within a computer. Alternatively, the operation model generating device 300 may be a dedicated computer designed for generating an operation model, or may be dedicated hardware realized by a dedicated circuit. Furthermore, if the operation model generating device 300 is connectable to the Internet, it may be realized by cloud computing.

The operation model generating device 300 includes an evaluation model acquisition unit 312, a learning environment data acquisition unit 314, an operation model generating unit 316, a learning operation instruction unit 318, an operation model determination unit 320, and an operation model output unit 322.

The evaluation model acquisition unit 312 acquires the evaluation model output by the evaluation model output unit 238, for example, via a network. However, this is not limited to this. The evaluation model acquisition unit 312 may acquire the evaluation model via various memory devices or via user input. The evaluation model acquisition unit 312 supplies the acquired evaluation model to the operation model generation unit 316.

The learning environment data acquisition unit 314 acquires learning environment data indicating the state in the learning environment via a network. However, this is not limited to this. The learning environment data acquisition unit 314 may acquire the learning environment data via various memory devices or via user input. The learning environment data acquisition unit 314 supplies the acquired learning environment data to the operation model generation unit 316.

The operation model generation unit 316 uses the learning environment data acquired by the learning environment data acquisition unit 314 to generate an operation model that outputs behavior according to the state of the facility 10 through reinforcement learning using the output of the evaluation model acquired by the evaluation model acquisition unit 312 as at least a part of the reward. The operation model generation unit 316 supplies the generated operation model to the operation model determination unit 320 and the operation model output unit 322.

The learning operation instruction unit 318 applies the operation amount based on the behavior output by the operation model during reinforcement learning to the control object in the learning environment.

The operation model determination unit 320 determines the validity of the operation model generated by the operation model generation unit 316. If the operation model determination unit 320 determines that the operation model is valid, it notifies the operation model output unit 322 of that effect.

If the operation model is determined to be valid, the operation model output unit 322 outputs the operation model to the control device 400. The details of the processing in the operation model generating device 300 will be explained in detail using example data and flow charts.

Figure 16 shows an example of an operation model generated by the operation model generating device 300. The operation model is composed of a combination (s, a) of a state s indicating a set of sampled state data and an action a taken under each state, and a weight w calculated by the reward. Note that the output of the evaluation model generated by the evaluation model generating device 200 is used as at least a part of the reward for calculating such weights. In this figure, as an example, a case where state s = (TI001, TI002, TI003, FI001, FI002, VI001) is shown. In this figure, for example, when an action a = 1 is taken under a state s = (-2.47803, -2.48413, -0.07324, 29.71191, 24.2511, 70), the weight calculated by the reward is w = 144.1484. The next action is determined by such an operation model.

Figure 17 shows an example of an action decision table. The action decision table is composed of an input state s and possible actions a. In this figure, as an example, the input state s = (0.1, 0.2, 0.4, 0.3, 0.8, 0.2) is shown, and the possible actions are a = (-3, -1, 0, 1, 3). For example, by inputting such an action decision table into the operation model shown in Figure 16, the next action is decided. This will be explained in detail using a flow chart.

Figure 18 shows an example of a processing flow in the operation model generation device 300. The operation model generation device 300 may execute the generation process of the operation model, for example, according to the flow shown in this figure.

In step S1802, the operation model generation device 300 acquires an evaluation model. For example, the evaluation model acquisition unit 312 acquires the evaluation model output in step S1460 in the flow of FIG. 14 via a network. The evaluation model acquisition unit 312 supplies the acquired evaluation model to the operation model generation unit 316.

In step S1804, the operation model generation device 300 generates an operation model by reinforcement learning. For example, the operation model generation unit 316 generates an operation model that outputs an action according to the state of the equipment 10 by reinforcement learning using the output of the evaluation model acquired in step S1802 as at least a part of the reward. As an example, the operation model generation unit 316 generates an operation model as shown in FIG. 16. Details of this will be described later using a separate flow. The operation model generation unit 316 supplies the generated operation model to the operation model determination unit 320 and the operation model output unit 322.

In step S1806, the operation model generating device 300 judges the validity of the operation model. For example, the operation model judging unit 320 judges the validity of the operation model generated in step S1804. As an example, the operation model judging unit 320 prepares data (a) when a user operates the plant simulator, or past data (b) of the user's operation, as reference data based on the target setting, operation terminal, and observation point information set by the evaluation model generating device 200. Next, the operation model judging unit 320 operates the operation model on the plant simulator for the generated operation model (c). At this time, the operation model judging unit 320 may use an actual machine instead of the plant simulator. Then, the operation model judging unit 320 judges the validity of the operation model by comparing the result output by (c) with (a) or (b). That is, the operational model determination unit 320 determines the validity of the operational model by comparing the reference data when the user inputs operations into the plant simulator with the results of operations performed by AI. If the results of operations performed by AI are higher, the operational model determination unit 320 determines that the generated operational model is valid (good). If the operational model determination unit 320 determines that the operational model is valid, it notifies the operational model output unit 322 of that effect.

In step S1808, the operation model generating device 300 outputs the operation model. For example, if the operation model is determined to be valid in step S1806, the operation model output unit 322 outputs the operation model to the control device 400.

Figure 19 shows an example of a reinforcement learning flow in the operation model generation unit 316. The operation model generation unit 316 may execute the process in step S1804 in Figure 18, for example, according to the flow shown in this figure.

In step S1902, the operation model generating device 300 acquires learning environment data. For example, the learning environment data acquiring unit 314 acquires learning environment data indicating the state in the learning environment via a network. As such a learning environment, a simulator that simulates the behavior of the equipment 10 may be used, or the actual equipment 10 may be used. For example, if the equipment 10 is a plant, a plant simulator may be used as the learning environment, or the actual plant may be used. The learning environment data acquiring unit 314 supplies the acquired learning environment data to the operation model generating unit 316.

In step S1904, the operation model generating device 300 determines an action. For example, the operation model generating unit 316 randomly determines an action. In the above description, the case where the operation model generating unit 316 randomly determines an action is shown as an example, but is not limited to this. When the operation model generating unit 316 determines an action, a known AI algorithm such as FKDPP (Factorial Kernel Dynamic Policy Programming) may be used. When using such a kernel method, the operation model generating unit 316 generates a vector of state s from the sensor value obtained by the learning environment data. Next, the operation model generating unit 316 generates a combination of the state s and all possible actions a as an action determination table such as that shown in FIG. 17. Then, the operation model generating unit 316 inputs the action determination table into an operation model such as that shown in FIG. 16. In response to this, the operation model performs kernel calculations between each row of the behavior decision table and each sample data of the operation model excluding the weight column, and calculates the distance between each sample data. Then, the operation model sequentially adds the distances calculated for each sample data multiplied by the value of each weight column to calculate the reward expectation value for each action. The operation model selects the action that will maximize the reward expectation value calculated in this way. For example, the operation model generation unit 316 may determine the action by selecting the action that is determined to have the highest reward expectation value using the operation model being updated in this way. During learning, the operation model generation unit 316 may determine the action while appropriately selecting whether to determine the action randomly or to determine the action using the operation model. The operation model generation unit 316 supplies the determined action to the learning operation instruction unit 318.

In step S1906, the operation model generation device 300 instructs the learning environment to perform an operation. For example, the learning operation instruction unit 318 adds the action determined in step S1904 to the value of the control object in the learning environment (valve value, etc.) to obtain an operation amount, and gives the operation amount to the control object in the learning environment. This changes the state of the learning environment.

In step S1908, the operation model generation device 300 acquires learning environment data. For example, the learning environment data acquisition unit 314 acquires learning environment data indicating the state in the learning environment, similar to step S1902. That is, the learning environment data acquisition unit 314 acquires the state of the learning environment after it has changed in response to the operation amount based on the determined action being applied to the control object. The learning environment data acquisition unit 314 supplies the acquired learning environment data to the operation model generation unit 316.

In step S1910, the operation model generating device 300 calculates a reward value. For example, the operation model generating unit 316 calculates the reward value based at least in part on the output of the evaluation model. As an example, the operation model generating unit 316 may calculate the reward value by directly using the index output by the evaluation model in response to inputting the learning environment data acquired in step S1908 into the evaluation model acquired in step S1802 of FIG. 18, or may calculate the reward value as 1 when the evaluation model judges the data to be OK, and as 0 when the evaluation model judges the data to be NG.

In step S1912, the operation model generating device 300 determines whether the process of acquiring the state according to the action decision has exceeded a specified number of steps. Note that such a number of steps may be specified in advance by the user, or may be determined based on the learning period (e.g., 10 days, etc.). If it is determined that the above process has not exceeded the specified number of steps (No), the operation model generating device 300 returns the process to step S1904 and continues the flow. The operation model generating device 300 executes the process of acquiring the state according to the action decision for the specified number of steps.

If it is determined in step S1912 that the above-mentioned processing has exceeded the specified number of steps (Yes), the operation model generation device 300 advances the processing to step S1914. In step S1914, the operation model generation device 300 updates the operation model. For example, the operation model generation unit 316 overwrites the values of the weight column in the operation model shown in FIG. 16, and adds new sample data that has not been saved so far to the operation model.

In step S1916, the operation model generating device 300 determines whether the update process of the operation model has exceeded a specified number of repetitions. Note that such a number of repetitions may be specified in advance by the user, or may be determined according to the validity of the operation model. If it is determined that the above process has not exceeded the specified number of repetitions (No), the operation model generating device 300 returns the process to step S1902 and continues the flow.

If it is determined in step S1916 that the above-mentioned process has exceeded the specified number of repetitions (Yes), the operation model generation device 300 ends the flow. In this way, for example, the operation model generation device 300 can generate an operation model that outputs an action according to the state of the equipment 10 by reinforcement learning in which the output of the evaluation model is at least a part of the reward.

Figure 20 shows an example of a block diagram of the control device 400 in the operation system 100 according to this embodiment. The control device 400 may be, for example, a controller in a distributed control system (DCS) or a medium-scale instrumentation system, or may be a real-time OS controller, etc.

The control device 400 includes an operation model acquisition unit 412, an actual environment data acquisition unit 414, a control unit 416, and an actual operation instruction unit 418.

The operation model acquisition unit 412 acquires the operation model output by the operation model output unit 322, for example, via a network. However, this is not limited to this. The operation model acquisition unit 412 may acquire the operation model via various memory devices or via user input. The operation model acquisition unit 412 supplies the acquired operation model to the control unit 416.

The real-environment data acquisition unit 414 acquires real-environment data indicating the state of the real environment, i.e., the equipment 10. Such real-environment data may be data similar to the state data described above. The real-environment data acquisition unit 414 supplies the acquired real-environment data to the control unit 416.

The control unit 416 determines the amount of operation based on the behavior output by the operation model acquired by the operation model acquisition unit 412 according to the actual environment acquired by the actual environment data acquisition unit 414, i.e., the state of the equipment 10. The control unit 416 supplies the determined amount of operation to the actual operation instruction unit 418.

The actual operation instruction unit 418 applies the operation amount determined by the control unit 416 to the control target in the actual environment, i.e., the facility 10.

Figure 21 shows an example of a processing flow in the control device 400. The control device 400 may execute control processing of the control target, for example, according to the flow shown in this figure.

In step S2102, the control device 400 acquires an operation model. For example, the operation model acquisition unit 412 acquires the operation model output in step S1808 of FIG. 18 via a network. The operation model acquisition unit 412 supplies the acquired operation model to the control unit 416.

In step S2104, the control device 400 acquires real-world environment data. For example, the real-world data acquisition unit 414 acquires real-world environment data indicating the state in the real environment. Such real-world environment data may be data similar to the state data indicating the state in the equipment 10 described above. The real-world data acquisition unit 414 supplies the acquired real-world environment data to the control unit 416.

In step S2106, the control device 400 determines an action. For example, the control unit 416 determines the action by selecting the action that is determined to have the highest expected reward value using the operation model. The control unit 416 supplies the determined action to the actual operation instruction unit 418.

In step S2108, the control device 400 instructs the real environment to perform an operation. For example, the real operation instruction unit 418 applies to the control object in the facility 10 an operation amount obtained by adding the action determined in step S2106 to the value of the control object in the facility 10. This changes the state of the real environment.

In step S2110, the control device 400 determines whether to end AI control. If it is determined that AI control is to be ended (Yes), the control device 400 ends the flow. If it is determined that AI control is not to be ended (No), the control device 400 returns the process to step S2104 and continues the flow.

Conventionally, as in Patent Document 1, for example, an AI control technology that uses a reinforcement learning model to control a control target is known. However, in the AI control technology, the user needs to set the reward function for calculating the reward value in advance based on experience, intuition, etc. When human intervention is involved, it takes a huge amount of time and effort to bring the operation cycle to a solution, such as long-term work using multiple labor forces over a period of years. In addition, it is necessary to consider delays and interruptions due to labor shortages and personnel allocation errors, and the possibility of work in remote or dangerous areas. Furthermore, even if the experience and intuition of a skilled operator are used, it is not always possible to make quick and optimal decisions. In the case of long-term plant management, it is not easy to secure a successor who can inherit the same level of skills. In addition, individual skills are often one-sided, and there are limitations to sharing information between different departments and functions and comprehensively understanding and solving multiple problems.

In contrast, in the operation system 100 according to the present embodiment, the AI automatically finds bottlenecks (potential faults) in the operation and generates indicators for improvement as an evaluation model. Then, the AI performs trial and error based on the given indicators to generate an operation model that indicates a better operation method. Then, the AI controller uses the operation model to control the controlled object by AI. As a result, the operation system 100 according to the present embodiment provides an environment in which the equipment 10 can be autonomously controlled using AI technology. Then, the operation system 100 according to the present embodiment updates the evaluation model and the operation model based on the state of the equipment under such AI control, and uses the updated operation model to control the controlled object by AI. As a result, the operation system 100 according to the present embodiment can autonomously run a loop to improve the operation of the equipment 10. Therefore, according to the operation system 100 according to the present embodiment, the PDCA cycle of data collection and investigation that has been performed so far can be performed continuously and at high speed 24 hours a day, 365 days a year, without interruption, thereby making it possible to continue to increase the productivity and efficiency of plants, etc. semi-permanently. In addition, because decisions can be made objectively and comprehensively according to the situation, knowledge accumulated over a long period of time can be utilized in a multifaceted way without being limited by risks such as skill transfer when experienced operators retire.

Various embodiments of the present invention may be described with reference to flow charts and block diagrams, where the blocks may represent (1) stages of a process in which operations are performed or (2) sections of an apparatus responsible for performing the operations. Particular stages and sections may be implemented by dedicated circuitry, programmable circuitry provided with computer readable instructions stored on a computer readable medium, and/or a processor provided with computer readable instructions stored on a computer readable medium. Dedicated circuitry may include digital and/or analog hardware circuitry and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including logical AND, logical OR, logical XOR, logical NAND, logical NOR, and other logical operations, memory elements such as flip-flops, registers, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

A computer-readable medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that the computer-readable medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray (RTM) disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk®, JAVA®, C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor or programmable circuit of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, either locally or over a wide area network (WAN) such as a local area network (LAN), the Internet, etc., to execute the computer-readable instructions to create means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

22 shows an example of a computer 9900 in which aspects of the present invention may be embodied in whole or in part. A program installed on the computer 9900 may cause the computer 9900 to function as or perform operations associated with an apparatus according to an embodiment of the present invention or one or more sections of the apparatus, and/or to perform a process or steps of a process according to an embodiment of the present invention. Such a program may be executed by the CPU 9912 to cause the computer 9900 to perform certain operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 9900 according to this embodiment includes a CPU 9912, a RAM 9914, a graphics controller 9916, and a display device 9918, which are interconnected by a host controller 9910. The computer 9900 also includes input/output units such as a communication interface 9922, a hard disk drive 9924, a DVD drive 9926, and an IC card drive, which are connected to the host controller 9910 via an input/output controller 9920. The computer also includes legacy input/output units such as a ROM 9930 and a keyboard 9942, which are connected to the input/output controller 9920 via an input/output chip 9940.

The CPU 9912 operates according to the programs stored in the ROM 9930 and the RAM 9914, thereby controlling each unit. The graphics controller 9916 retrieves image data generated by the CPU 9912 into a frame buffer or the like provided in the RAM 9914 or into itself, and causes the image data to be displayed on the display device 9918.

The communication interface 9922 communicates with other electronic devices via a network. The hard disk drive 9924 stores programs and data used by the CPU 9912 in the computer 9900. The DVD drive 9926 reads programs or data from the DVD-ROM 9901 and provides the programs or data to the hard disk drive 9924 via the RAM 9914. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

The ROM 9930 stores therein a boot program, etc., executed by the computer 9900 upon activation, and/or a program that depends on the hardware of the computer 9900. The input/output chip 9940 may also connect various input/output units to the input/output controller 9920 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

The programs are provided by a computer-readable medium such as a DVD-ROM 9901 or an IC card. The programs are read from the computer-readable medium and installed in the hard disk drive 9924, RAM 9914, or ROM 9930, which are also examples of computer-readable media, and executed by the CPU 9912. The information processing described in these programs is read by the computer 9900, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the manipulation or processing of information according to the use of the computer 9900.

For example, when communication is performed between the computer 9900 and an external device, the CPU 9912 may execute a communication program loaded into the RAM 9914 and instruct the communication interface 9922 to perform communication processing based on the processing described in the communication program. Under the control of the CPU 9912, the communication interface 9922 reads transmission data stored in a transmission buffer processing area provided in the RAM 9914, the hard disk drive 9924, the DVD-ROM 9901, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes reception data received from the network to a reception buffer processing area or the like provided on the recording medium.

The CPU 9912 may also cause all or a necessary portion of a file or database stored on an external recording medium such as a hard disk drive 9924, a DVD drive 9926 (DVD-ROM 9901), an IC card, etc. to be read into the RAM 9914, and perform various types of processing on the data on the RAM 9914. The CPU 9912 then writes back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 9912 may perform various types of processing on the data read from the RAM 9914, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 9914. The CPU 9912 may also search for information in a file, database, etc. in the recording medium. For example, if multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 9912 may search for an entry that matches a condition in which an attribute value of the first attribute is specified from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-described program or software module may be stored on a computer-readable medium on the computer 9900 or in the vicinity of the computer 9900. Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, thereby providing the program to the computer 9900 via the network.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in this order.

10 Facility 20 Headquarters 100 Operation system 200 Evaluation model generating device 210 Labeling function unit 212 Operation target acquisition unit 214 Status data acquisition unit 216 Correlation data generating unit 218 Teacher label acquisition unit 220 Labeling unit 222 Labeling data output unit 224 Labeling model update unit 230 Machine learning function unit 240 Labeling data acquisition unit 250 Evaluation model generating unit 252 Learning unit 254 Preprocessing unit 256 Machine learning unit 260 Evaluation model determination unit 270 Evaluation model output unit 300 Operation model generating device 312 Evaluation model acquisition unit 314 Learning environment data acquisition unit 316 Operation model generating unit 318 Learning operation instruction unit 320 Operation model determination unit 322 Operation model output unit 400 Control device 412 Operation model acquisition unit 414 Real environment data acquisition unit 416 Control unit 418 Real operation instruction unit 9900 Computer 9901 DVD-ROM
9910 Host controller 9912 CPU
9914 RAM
9916 Graphics controller 9918 Display device 9920 Input/output controller 9922 Communication interface 9924 Hard disk drive 9926 DVD drive 9930 ROM
9940 Input/Output Chip 9942 Keyboard

Claims (14)

an evaluation model generation device that generates an evaluation model that outputs an index that evaluates a state of a facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
an operation model generation device that generates an operation model that outputs an action according to a state of the facility by reinforcement learning using an output of the evaluation model as at least a part of a reward;
a control device that applies a manipulation amount based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ,
the evaluation model generation device updates the evaluation model based on a state of the facility when the control object is controlled using the operation model; and
An operation system , wherein the operation model generation device updates the operation model through reinforcement learning using the output of the updated evaluation model as at least a part of a reward. The operation system according to claim 1 , wherein the control device controls the controlled object using the updated operation model. an evaluation model generation device that generates an evaluation model that outputs an index that evaluates a state of a facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
an operation model generation device that generates an operation model that outputs an action according to a state of the facility by reinforcement learning using an output of the evaluation model as at least a part of a reward;
a control device that applies a manipulation amount based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ,
The evaluation model generation device comprises:
An operation target acquisition unit that acquires the operation target;
A status data acquisition unit that acquires status data indicating a status of the equipment;
a correlation data generating unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data based on the operation target;
a labeling unit that labels the correlation data using a labeling model;
an evaluation model generation unit that generates the evaluation model using the labeled correlation data;
An operating system comprising : The evaluation model generation device comprises:
The operation system according to claim 3 , further comprising an evaluation model determination unit that determines the validity of the evaluation model. The evaluation model generation device comprises:
The operation system according to claim 4 , further comprising an evaluation model output unit that outputs the evaluation model when the evaluation model is determined to be valid. The evaluation model generation device comprises:
The operation system according to claim 4 or 5 , further comprising a labeling model update unit that updates the labeling model when the evaluation model is determined to be valid. The evaluation model generation device further includes a truth label acquisition unit that acquires truth labels for at least a portion of the correlation data,
The operation system according to claim 6 , wherein the labeling model update unit generates an updated labeling model separately from an initial labeling model generated based on the teacher label. generating an evaluation model that outputs an index that evaluates a state of the facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
generating an operation model that outputs an action according to a state of the equipment by reinforcement learning using an output of the evaluation model as at least a part of a reward;
and providing a manipulated variable based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ,
generating the evaluation model includes updating the evaluation model based on a state of the equipment when the control object is controlled using the operation model;
An operating method , wherein generating the operating model includes updating the operating model by reinforcement learning using an output of the updated evaluation model as at least a part of a reward. generating an evaluation model that outputs an index that evaluates a state of the facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
generating an operation model that outputs an action according to a state of the equipment by reinforcement learning using an output of the evaluation model as at least a part of a reward;
and providing a manipulated variable based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ,
Generating the valuation model includes:
obtaining said operational objectives;
acquiring status data indicative of a status of the facility;
generating correlation data indicating at least one of a correlation between at least one physical quantity included in the status data and time and a correlation between at least two physical quantities included in the status data based on the operation target;
labeling the correlation data using a labeling model; and
generating the evaluation model using the labeled correlation data; and
A method of operation having the steps : When executed by a computer, the computer is caused to
an evaluation model generation device that generates an evaluation model that outputs an index that evaluates a state of a facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
an operation model generation device that generates an operation model that outputs an action according to a state of the facility by reinforcement learning using an output of the evaluation model as at least a part of a reward;
a control device that applies a manipulated variable based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ;
the evaluation model generation device updates the evaluation model based on a state of the facility when the control object is controlled using the operation model; and
The operation model generation device updates the operation model through reinforcement learning using the output of the updated evaluation model as at least a part of a reward. When executed by a computer, the computer is caused to
an evaluation model generation device that generates an evaluation model that outputs an index that evaluates a state of a facility with respect to a target target based on an operation target of the facility and a state of the facility by machine learning;
an operation model generation device that generates an operation model that outputs an action according to a state of the facility by reinforcement learning using an output of the evaluation model as at least a part of a reward;
a control device that applies a manipulated variable based on the behavior output by the operation model in accordance with the state of the equipment to a control target in the equipment ;
The evaluation model generation device comprises:
An operation target acquisition unit that acquires the operation target;
A status data acquisition unit that acquires status data indicating a status of the equipment;
a correlation data generating unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data based on the operation target;
a labeling unit that labels the correlation data using a labeling model;
an evaluation model generation unit that generates the evaluation model using the labeled correlation data;
An operational program that includes : an operation target acquisition unit that acquires an operation target in the facility;
A status data acquisition unit that acquires status data indicating a status of the equipment;
a correlation data generating unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data based on the operation target;
a labeling unit that labels the correlation data using a labeling model;
and an evaluation model generation unit that generates an evaluation model that outputs an index that evaluates a state of the facility with respect to a target target based on an operation target of the facility and a state of the facility, using the labeled correlation data ;
The evaluation model generating device outputs the index used as at least a part of a reward in reinforcement learning of an operation model that outputs an action according to a state of the equipment. Obtaining operational goals for the facility;
acquiring status data indicative of a status of the facility;
generating correlation data indicating at least one of a correlation between at least one physical quantity included in the status data and time and a correlation between at least two physical quantities included in the status data based on the operation target;
labeling the correlation data using a labeling model; and
generating an evaluation model that outputs an index that evaluates a state of the facility with respect to a target target based on an operation target of the facility and a state of the facility, using the labeled correlation data ;
An evaluation model generation method, in which the evaluation model outputs the index used as at least a part of a reward in reinforcement learning of an operation model that outputs an action according to a state of the equipment. When executed by a computer, the computer is caused to
an operation target acquisition unit that acquires an operation target in the facility;
A status data acquisition unit that acquires status data indicating a status of the equipment;
a correlation data generating unit that generates correlation data indicating at least one of a correlation between at least one physical quantity included in the state data and time and a correlation between at least two physical quantities included in the state data based on the operation target;
a labeling unit that labels the correlation data using a labeling model;
using the labeled correlation data, the evaluation model generating unit generates an evaluation model that outputs an index that evaluates a state of the facility with respect to a target target based on an operation target of the facility and a state of the facility ;
The evaluation model generates an evaluation model that outputs the index used as at least a part of a reward in reinforcement learning of an operation model that outputs an action according to a state of the equipment.

Images