JP7187961B2 - Reinforcement learning program, reinforcement learning method, and reinforcement learning device - Google Patents

Description

The present invention relates to a reinforcement learning program, a reinforcement learning method, and a reinforcement learning device.

Conventionally, a power generation system including one or more power generators using natural energy may be controlled by reinforcement learning. Reinforcement learning uses, for example, a table that associates an effective value indicating the effectiveness of a command value for a generator with each possible state value of the generator. The table is, for example, a Q table. In reinforcement learning, for example, learning for estimating effective values for command values is repeatedly performed, and the table is updated.

Japanese translation of PCT publication No. 2016-517104

However, the conventional technology may lead to an increase in the amount of processing in reinforcement learning. For example, as the number of regions that the state value of the generator can take increases, the number of times learning for estimating the effective value of the command value increases, resulting in an increase in the amount of processing in reinforcement learning.

In one aspect, an object of the present invention is to reduce the amount of processing in reinforcement learning.

According to one embodiment, learning is performed using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of areas that the state value related to the generator can take, and the power demand is If it is equal to or less than a predetermined threshold , referring to the state value including the observed output voltage of the generator, based on a characteristic function of the state value of the generator, two or more consecutive regions among the plurality of regions and generate validity information indicating the effectiveness of each command value for the generator in the area where the A reinforcement learning program, a reinforcement learning method, and a reinforcement learning device that perform learning using effectiveness information for each area are proposed.

According to one aspect, it is possible to reduce the amount of processing in reinforcement learning.

FIG. 1 is an explanatory diagram of an example of a reinforcement learning method according to an embodiment. FIG. 2 is an explanatory diagram showing an example of the power generation system 200. As shown in FIG. FIG. 3 is a block diagram showing a hardware configuration example of the reinforcement learning device 100. As shown in FIG. FIG. 4 is a block diagram showing a functional configuration example of the reinforcement learning device 100. As shown in FIG. FIG. 5 is an explanatory diagram showing a specific configuration example of a power generation system 200 including a wind power generator. FIG. 6 is an explanatory diagram showing normal intervals and roughly divided intervals that the state values of the wind power generator under stall control can take. FIG. 7 is an explanatory diagram showing normal intervals and roughly divided intervals that the state values of the pitch-controlled wind power generator can take. FIG. 8 is an explanatory diagram showing an example of realizing the normal table 801 and the rough division table 802. As shown in FIG. FIG. 9 is an explanatory diagram showing an example of the contents stored in the normal table 801. As shown in FIG. FIG. 10 is an explanatory diagram showing an example of creating a characteristic function. FIG. 11 is an explanatory diagram showing an example of switching the table to be used to the rough division table 802. As shown in FIG. FIG. 12 is an explanatory diagram showing an example of switching the table to be used to the normal table 801. As shown in FIG. FIG. 13 is an explanatory diagram of an example of updating valid values. FIG. 14 is an explanatory diagram showing a specific configuration example of a power generation system 200 including a thermal power generator. FIG. 15 is an explanatory diagram showing an example of the contents of the normal table 801 regarding thermal power generators. FIG. 16 is a flow chart showing an example of the overall processing procedure. FIG. 17 is a flowchart illustrating an example of a switching determination processing procedure; FIG. 18 is a flowchart illustrating an example of a value setting procedure. FIG. 19 is a flow chart showing an example of a characteristic function creation processing procedure for a stall control wind power generator. FIG. 20 is a flowchart showing an example of a characteristic function creation processing procedure for a pitch-controlled wind power generator.

Hereinafter, embodiments of a reinforcement learning program, a reinforcement learning method, and a reinforcement learning apparatus according to the present invention will be described in detail with reference to the drawings.

(One Example of Reinforcement Learning Method According to Embodiment)
FIG. 1 is an explanatory diagram of an example of a reinforcement learning method according to an embodiment. The reinforcement learning device 100 is a computer that applies reinforcement learning to a power generation system including one or more generators and controls the power generation system including one or more generators. The power generator is, for example, a wind power generator or a thermal power generator.

In reinforcement learning, for example, an effective value indicating the effectiveness of each combination of command values for one or more generators in each of a plurality of regions that can be taken by a combination of state values for one or more generators is associated. A table is used that represents The table is, for example, a Q table. In reinforcement learning, for example, the state value of a generator is observed, the command value for the generator is determined, the determined command value is input to the generator, and the validity of the input command value in the region containing the observed state value is determined. Learning for estimating effective values that indicate sex is repeatedly performed, and the table is updated. Reinforcement learning is implemented by, for example, Q-learning, SARSA, or the like.

Here, it is conceivable that an increase in the amount of processing in reinforcement learning may be caused. For example, as the number of regions that can be taken by combinations of state values relating to one or more generators increases, the number of times learning is performed increases, leading to an increase in the amount of processing in reinforcement learning. Specifically, if the state values are finely divided and the regions are set, the number of regions increases. Also, specifically, when the number of generators increases, the number of regions increases. Therefore, it is desired to reduce the amount of processing in reinforcement learning.

On the other hand, it is conceivable to reduce the number of regions and reduce the amount of processing in reinforcement learning. For example, it is conceivable to set regions by roughly dividing the state values, reduce the number of regions, and reduce the amount of processing in reinforcement learning. However, if a roughly divided region is always used, it is not possible to verify in detail what kind of command value should be output under what kind of state value. You may not be able to exercise control. Appropriate control is, for example, control that maximizes the power generation amount of the power generation system within a range that does not exceed a predetermined threshold.

Therefore, it is conceivable to reduce the amount of processing in reinforcement learning by dynamically changing the number of regions. For example, at some timing, two or more regions may be combined to reduce the number of regions. However, if it is not known how to preferably set the effective value indicating the effectiveness of each combination of command values for one or more generators in a region combining two or more regions, the power generation system It becomes difficult to perform appropriate control by

Therefore, in this embodiment, in reinforcement learning, the number of regions that can be taken by a combination of state values related to one or more generators can be dynamically changed, and an appropriate A reinforcement learning method capable of resetting effective values determined to be will be described. According to this reinforcement learning method, it is possible to reduce the amount of processing in reinforcement learning.

In the example of FIG. 1, there is one generator included in the power generation system. The state values related to the generator are, for example, the output power from the generator and the environment value related to the amount of natural energy supplied to the generator. Environmental values are, for example, wind speed and fuel consumption. The command value for the generator is, for example, a command value for switching ON and OFF of the power supply of the generator. The command value for the generator is, for example, a command value that changes the utilization efficiency of natural energy in the generator.

In FIG. 1, the reinforcement learning device 100 performs learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of areas in which the state value of the generator can take. A region is, for example, an interval or a combination of intervals. Validity information is, for example, a record of the Q table. A valid value is, for example, the Q value. For example, the reinforcement learning device 100 stores validity information in which the effective value G1 is associated with the section A1, validity information in which the effective value G2 is associated with the section A2, and validity information in which the effective value G3 is associated with the section A3. Learning is performed using a Q-table containing information. The reinforcement learning device 100 updates the Q table as a result of learning.

The reinforcement learning device 100 generates effectiveness information indicating the effectiveness of each command value for the generator in a region obtained by combining two or more consecutive regions among a plurality of regions. For example, the reinforcement learning device 100 refers to the observed state values of the generators and generates effectiveness information of the combined regions based on the characteristic function of the state values of the generators. Specifically, the reinforcement learning device 100 calculates an effective value Ga that indicates the effectiveness of the command value in the section Aa that combines the sections A2 and A3, and calculates the validity that the effective value Ga is associated with the section Aa. Generate information. Then, the reinforcement learning device 100 sets the validity information in which the effective value G2 is associated with the section A2, the validity information in which the effective value G3 is associated with the section A3, and the validity information in which the effective value Ga is associated with the section Aa. The Q table is updated by replacing it with the quality information.

The reinforcement learning device 100 performs learning using effectiveness information about the combined regions and effectiveness information about each region other than two or more regions among the plurality of regions. The reinforcement learning device 100 includes, for example, a Q table containing validity information in which the effective value G1 is associated with the section A1, and effectiveness information in which the section Aa, which is a combination of the sections A2 and A3, is associated with the effective value Ga. use to learn. The reinforcement learning device 100 updates the Q table as a result of learning.

As a result, the reinforcement learning device 100 can combine two or more regions and dynamically reduce the number of regions associated with effectiveness information. Therefore, the reinforcement learning device 100 can reduce the number of pieces of effectiveness information to be updated by learning, and reduce the amount of processing required for reinforcement learning. Further, when generating validity information, the reinforcement learning device 100 can set the valid value based on the characteristic function without initializing the valid value to 0 or setting the valid value randomly. can. For this reason, the reinforcement learning device 100 can make the generated effectiveness information accurately indicate the effectiveness of each command value for the power generator, thereby facilitating appropriate control of the power generation system. can.

Here, at some timing, the validity information about each of the two or more regions is replaced with the validity information about the combined region of the two or more regions, and the number of regions with which the validity information is associated Although the case where is dynamically decreased has been described, the present invention is not limited to this. For example, at some point in time, validity information about a combined area of two or more areas may be replaced with validity information about each of the two or more areas to dynamically increase the number of areas. may As a result, the reinforcement learning device 100 can subdivide which command value should be output for which state value. A case of replacing validity information about a combined area of two or more areas with validity information about each of the two or more areas will be specifically described later with reference to FIG. 12 .

Although the case where the number of generators included in the power generation system is one has been described here, the present invention is not limited to this. For example, there may be multiple generators included in the power generation system. In this case, for example, the reinforcement learning device 100 generates effectiveness information about a region obtained by combining two or more continuous regions among a plurality of regions that can be taken by combinations of state values of generators. Then, the reinforcement learning device 100 replaces the effectiveness information about each of the two or more areas with the effectiveness information about the combined area, and dynamically reduces the number of areas with which the effectiveness information is associated. Let As a result, the reinforcement learning device 100 can reduce the amount of processing required for reinforcement learning. A case where there are a plurality of generators included in the power generation system will be described later with reference to FIGS. 11 and 12. FIG.

Here, the reinforcement learning device 100 has been described without specifying the type of generator included in the power generation system. On the other hand, for example, the generators included in the power generation system may be wind power generators. A case where the power generator included in the power generation system is a wind power generator will be specifically described later with reference to FIGS. 11 and 12. FIG. Further, for example, the power generator included in the power generation system may be a thermal power generator. A case where the power generator included in the power generation system is a thermal power generator will be specifically described later using FIG. 15 . Also, for example, the power generation system may include both wind power generators and thermal power generators.

Here, the reinforcement learning device 100 has been described without limiting the timing of generating validity information for a region obtained by combining two or more regions. On the other hand, for example, when the observed power demand is equal to or less than the threshold, the reinforcement learning device 100 may generate effectiveness information about the combined region. Further, for example, the reinforcement learning device 100 may generate effectiveness information about the combined region when the observed power demand exceeds the threshold.

(Example of power generation system 200)
Next, an example of a power generation system 200 to which the reinforcement learning device 100 shown in FIG. 1 is applied will be described using FIG.

FIG. 2 is an explanatory diagram showing an example of the power generation system 200. As shown in FIG. In FIG. 2 , a power generation system 200 includes a reinforcement learning device 100 and one or more generators 201 .

In the power generation system 200 , the reinforcement learning device 100 and one or more power generators 201 are connected via a wired or wireless network 210 . The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like.

The power generator 201 is, for example, a machine that uses wind energy to generate power using a wind turbine. The generator 201 generates power using, for example, wind turbine torque transmitted from the wind turbine. The generator 201 is provided with a measuring device that observes state values relating to the generator 201 . The measuring machine has, for example, a sensor device. The sensor device may have at least one of an acceleration sensor, a geomagnetic sensor, an optical sensor, a vibration sensor, a power sensor, a voltage sensor, a current sensor, and the like. The power generator 201 may be, for example, a machine that uses thermal energy to generate power using a turbine.

The reinforcement learning device 100 applies reinforcement learning to the power generation system 200 and controls the power generation system 200 . The reinforcement learning device 100 controls command values for one or more generators 201 included in the power generation system 200, for example. Specifically, the reinforcement learning device 100 acquires a state value related to the generator 201 from a measuring device provided in the generator 201 . Reinforcement learning device 100 determines and outputs a command value for generator 201 based on the acquired state value and a table containing validity information. Reinforcement learning device 100 updates the table according to the result of outputting the command value. The reinforcement learning device 100 is, for example, a server, a PC (Personal Computer), a microcomputer, a PLC (Programmable Logic Controller), or the like.

(Hardware configuration example of reinforcement learning device 100)
Next, a hardware configuration example of the reinforcement learning device 100 will be described with reference to FIG.

FIG. 3 is a block diagram showing a hardware configuration example of the reinforcement learning device 100. As shown in FIG. In FIG. 3 , the reinforcement learning device 100 has a CPU (Central Processing Unit) 301 , a memory 302 , a network I/F (Interface) 303 , a recording medium I/F 304 and a recording medium 305 . Also, each component is connected by a bus 300 .

Here, the CPU 301 controls the entire reinforcement learning device 100 . The memory 302 has, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash ROM, and the like. Specifically, for example, a flash ROM or ROM stores various programs, and a RAM is used as a work area for the CPU 301 . A program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute coded processing.

Network I/F 303 is connected to network 210 through a communication line, and is connected to other computers via network 210 . Another computer is, for example, the generator 201 . A network I/F 303 serves as an internal interface with the network 210 and controls input/output of data from other computers. For the network I/F 303, for example, a modem, LAN adapter, etc. can be adopted.

A recording medium I/F 304 controls reading/writing of data from/to the recording medium 305 under the control of the CPU 301 . The recording medium I/F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. A recording medium 305 is a nonvolatile memory that stores data written under control of the recording medium I/F 304 . The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 305 may be removable from the reinforcement learning device 100 .

The reinforcement learning device 100 may have, for example, a keyboard, mouse, display, printer, scanner, microphone, speaker, etc., in addition to the components described above. Also, the reinforcement learning device 100 may have a plurality of recording medium I/Fs 304 and recording media 305 . Also, the reinforcement learning device 100 may not have the recording medium I/F 304 and the recording medium 305 .

(Example of functional configuration of reinforcement learning device 100)
Next, a functional configuration example of the reinforcement learning device 100 will be described with reference to FIG.

FIG. 4 is a block diagram showing a functional configuration example of the reinforcement learning device 100. As shown in FIG. Reinforcement learning device 100 includes storage unit 400 , acquisition unit 401 , switching unit 402 , learning unit 403 , and output unit 404 .

The storage unit 400 is implemented by, for example, a storage area such as the memory 302 or recording medium 305 shown in FIG. Although a case where the storage unit 400 is included in the reinforcement learning device 100 will be described below, the present invention is not limited to this. For example, the storage unit 400 may be included in a device different from the reinforcement learning device 100 and the storage contents of the storage unit 400 may be referenced from the reinforcement learning device 100 .

Acquisition unit 401 to output unit 404 function as an example of a control unit. Specifically, for example, the acquisition unit 401 to the output unit 404 cause the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 shown in FIG. to realize its function. The processing result of each functional unit is stored in a storage area such as the memory 302 or recording medium 305 shown in FIG. 3, for example.

The storage unit 400 stores various information that is referred to or updated in the processing of each functional unit. The storage unit 400 stores state values regarding the power generation system 200 . The state values regarding the power generation system 200 include, for example, a state value regarding each power generator 201 of the one or more power generators 201 included in the power generation system 200 and a state value regarding the power generation system 200 as a whole. The power generator 201 is, for example, a wind power generator or a thermal power generator. The state values for the generator 201 are, for example, the output wattage value and wind speed for the stall controlled wind generator and the output wattage value and wind speed for the pitch controlled wind generator. State values for generator 201 are, for example, output wattage and fuel usage for thermal generators. Moreover, the state value related to the power generation system 200 as a whole is, for example, the power demand in the power generation system 200 as a whole.

Storage unit 400 stores a command value for each of one or more generators 201 included in power generation system 200 . The command value for the generator 201 is a command value for changing the utilization efficiency of natural energy in the generator 201 . The command value is, for example, a command value for switching the power of the generator 201 between ON and OFF. The command value is, for example, a command value that changes the wind reception performance of the wind power generator. The command value is, for example, a command value indicating how much the pitch angle of the pitch-controlled wind power generator is to be changed. Specifically, the command values indicating how much the pitch angle should be changed are -ΔΘ, ±0 and +ΔΘ. The command value is, for example, a command value indicating how much the size of the fuel supply hole provided in the generator of the thermal power generator should be changed. The command value is, for example, a command value indicating how much the amount of fuel used by the thermal power generator should be changed.

The storage unit 400 stores effectiveness information indicating the effectiveness of each combination of command values for the one or more generators 201 in each of a plurality of areas where combinations of state values relating to the one or more generators 201 can take. do. A combination of state values may be one state value. The storage unit 400 also stores effectiveness information indicating the effectiveness of each combination of command values for the one or more generators 201 in a region obtained by combining two or more regions out of a plurality of regions. The storage unit 400 stores, for example, a table containing, as a record, validity information in which each of a plurality of regions is associated with a valid value indicating the validity of each combination of command values. The effective value indicates, for example, the extent to which it contributes to increasing the reward in the generator. The reward is, for example, the amount of power generation. Further, the storage unit 400 stores, for example, the validity information for each of two or more regions among the validity information for each of the plurality of regions, and the validity information for a region obtained by combining the two or more regions. Store the table replaced with sexual information.

The storage unit 400 stores characteristic functions for the generator 201 . The characteristic function indicates changes in state values for generator 201 . A characteristic function indicates, for example, the relationship between wind speed and power output from a wind power generator. The characteristic function indicates, for example, the relationship between the amount of fuel used by the thermal power generator and the power output from the thermal power generator. The storage unit 400 stores, for example, an approximate curve that approximates the characteristic function. The storage unit 400 stores, for example, different characteristic functions for each wind reception performance of the wind power generator. Specifically, the storage unit 400 stores a different characteristic function for each pitch angle of the wind power generator.

The storage unit 400 stores, for example, a processing procedure by a reinforcement learning algorithm and an action selection algorithm. A reinforcement learning algorithm is, for example, a Q-learning algorithm. Reinforcement learning algorithms may be other than Q-learning algorithms. The action selection algorithm is, for example, the ε-greedy algorithm.

Acquisition unit 401 acquires various types of information used for processing of each functional unit from storage unit 400 and outputs the information to each functional unit. The acquisition unit 401 may acquire various types of information used in the processing of each functional unit from a device different from the reinforcement learning device 100 and output the information to each functional unit. The acquisition unit 401 acquires, for example, state values regarding the power generation system 200 . Acquisition unit 401 acquires a state value related to generator 201 from a measuring device provided in generator 201 . Specifically, the acquisition unit 401 acquires the power demand in the power generation system 200 from the computer of the electric company.

The acquisition unit 401 may acquire information representing a characteristic function, for example. The obtaining unit 401 may obtain, for example, a threshold for the characteristic function and generate information representing the characteristic function. The thresholds for the characteristic function are at least rated wind speed and maximum power. The thresholds for the characteristic function may also be cut-in wind speed and cut-out wind speed. The acquisition unit 401 may acquire output power from the generator 201 at various wind speeds and generate information representing a characteristic function.

The switching unit 402 switches effectiveness information to be used for learning in reinforcement learning. The switching unit 402 sets, for example, effectiveness information for each of the plurality of areas as effectiveness information to be used for learning. For example, the switching unit 402 converts the validity information for each of two or more regions from the validity information for each region of the plurality of regions to the validity information for a region obtained by combining the two or more regions. and set it as the validity information used for learning.

Specifically, in some cases, effectiveness information for each of two or more areas is set as effectiveness information to be used for learning. In this case, the switching unit 402 refers to the obtained state value of the generator 201 and generates effectiveness information for a region obtained by combining two or more regions based on the characteristic function. Moreover, the switching unit 402 may generate validity information for a region obtained by combining two or more regions based on validity information for each region of the two or more regions. Then, the switching unit 402 replaces the validity information for each of the two or more regions among the validity information used for learning with the generated validity information.

Specifically, in some cases, effectiveness information about a region obtained by combining two or more regions is set as effectiveness information to be used for learning. In this case, the switching unit 402 refers to the obtained state value of the generator 201 and generates effectiveness information for each of the two or more regions based on the characteristic function. Moreover, the switching unit 402 may generate validity information for each of the two or more regions based on validity information for a region obtained by combining two or more regions. Then, the switching unit 402 replaces the validity information about the region obtained by combining two or more regions among the validity information used for learning with the generated validity information.

Specifically, the switching unit 402 identifies the output power corresponding to the acquired wind speed based on the characteristic function, and generates effectiveness information about the combined region based on the identified output power. In addition, specifically, the switching unit 402 selects an effective Generate sexual information.

Specifically, the switching unit 402 generates effectiveness information about the combined regions when the acquired power demand is equal to or less than the threshold. More specifically, when the acquired power demand is equal to or less than a threshold, the switching unit 402 selects a region obtained by combining two or more regions of relatively large output power among a plurality of regions of output power. generate validity information for

Specifically, the switching unit 402 generates effectiveness information about the combined regions when the acquired power demand exceeds the threshold. More specifically, when the acquired power demand exceeds the threshold, the switching unit 402 selects a region obtained by combining two or more regions of relatively small output power among a plurality of regions of output power. Generate validity information.

The learning unit 403 performs learning using the validity information set by the switching unit 402, and updates at least one of the validity information. The learning unit 403 performs learning using, for example, effectiveness information for each of the plurality of regions. For example, the learning unit 403 acquires effectiveness information about each of two or more areas from among the effectiveness information about each of the plurality of areas, and the effectiveness information about an area obtained by combining the two or more areas. Learn by replacing with

The output unit 404 outputs the processing result of each functional unit. The output format is, for example, display on a display, print output to a printer, transmission to an external device via the network I/F 303, or storage in a storage area such as the memory 302 or recording medium 305. As a result, the output unit 404 can notify the user of the processing results of each function unit, and can support management and operation of the reinforcement learning device 100, for example, updating of setting values of the reinforcement learning device 100. The convenience of the reinforcement learning device 100 can be improved.

(Example of operation of reinforcement learning device 100 for power generation system 200 including wind power generators)
Next, an operation example of the reinforcement learning device 100 for the power generation system 200 including the wind power generator will be described with reference to FIGS. 5 to 13. FIG. First, moving to the description of FIG. 5, a specific configuration example of the power generation system 200 including the wind power generator will be described.

FIG. 5 is an explanatory diagram showing a specific configuration example of a power generation system 200 including a wind power generator. In the example of FIG. 5, the power generation system 200 includes the reinforcement learning device 100, a stall control wind power generator i (i=1, . . . , n), and a pitch control wind power generator i (i=1, . , m). A stall-controlled wind power generator i receives a command value ai (i=1, . . . , n) from the reinforcement learning device 100 . The pitch-controlled wind power generator i receives command values bi (i=1, . . . , m) from the reinforcement learning device 100 .

The power generation system 200 includes an anemometer si (i=1, . . . , n) for the stall control wind power generator i and an anemometer pi (i=1, . , m). The anemometer si transmits the wind speed value F _si (t _j ) to the reinforcement learning device 100 . t _j is the time. The anemometer pi transmits the wind speed value F _pi (t _j ) to the reinforcement learning device 100 . The power generation system 200 includes a power meter for the stall-controlled wind power generator i and a power meter for the pitch-controlled wind power generator i. The power meter for the stall-controlled wind turbine i sends the output wattage value P _si (t _j ) to the reinforcement learning device 100 . The power meter for pitch-controlled wind turbine i sends the output wattage value P _pi (t _j ) to the reinforcement learning device 100 .

Reinforcement learning device 100 includes table generator 501 , interval switcher 502 , value setter 503 , action determiner 504 , state calculator 505 , reward calculator 506 , and table updater 507 . The reinforcement learning device 100 calculates the output watt value P _si (t _j ) for the stall control wind power generator _i and the pitch control An attempt is made to increase the sum with the output wattage value P _pi (t _j ) for the wind power generator i.

The table generation unit 501 creates a normal table that stores effectiveness information about a plurality of normal sections divided by the normal division method described later with reference to FIGS. 6 and 7 . The table generation unit 501 sets whether any two or more normal sections are combined to convert a plurality of normal sections into a plurality of coarsely divided sections divided by a coarse division method described later with reference to FIGS. do.

Section switching unit 502 selects wind speed value F _si (t _j ), wind speed value F _pi (t _j ), output watt value P _si (t _j ), output watt value P _pi (t _j ), power demand Watt values P'(t _j ) are received. The section switching unit 502 switches the table to be used between a normal table and a coarse division table storing effectiveness information for a plurality of coarsely divided sections based on the received various information. If threshold value α>power demand watt value P′(t _j ), section switching unit 502 sets the rough division table as the table to be used. If threshold value α≦power demand watt value P′(t _j ), section switching unit 502 sets the normal table as the table to be used.

A value setting unit 503 sets valid values in the table based on the switching result. When two or more areas are combined, the value setting unit 503 sets a valid value to the record corresponding to the combined areas. When two or more areas are separated, the value setting unit 503 sets a valid value to the record corresponding to each separated area. A value setting unit 503 outputs a table in which valid values are set.

The action determination unit 504 uses the table to select the command value ai for the wind power generator i for stall control, transmits it to the wind power generator i for stall control, and sets the command value bi for the wind power generator i for pitch control. Select and send to pitch controlled wind generator i. The action determination unit 504 selects, for example, the combination of the command value ai and the command value bi associated with the largest effective value in the table. Specifically, the action determining unit 504 uses the ε-greedy algorithm to randomly select a command value with a probability of ε, and selects the command value with a probability of 1−ε in the current state of the power generation system 200. A combination of the associated command value ai and command value bi is selected.

State calculation unit 505 calculates wind speed value F _si (t _j ), wind speed value F _pi (t _j ), output watt value P _si (t _j ), output watt value P _pi (t _j ), power demand The state of the power generation system 200 is identified based on the watt value P'(t _j ). The state calculator 505 outputs a state result indicating the record of the table corresponding to the specified state. Remuneration calculation unit 506 calculates a remuneration value based on output watt value P _si (t _j ), output watt value P _pi (t _j ), and power demand watt value P′(t _j ). The table updating unit 507 updates the effective value in the record indicated by the state result based on the calculated reward value.

6 to 13, a table generation unit 501, a section switching unit 502, a value setting unit 503, an action determination unit 504, a state calculation unit 505, a reward calculation unit 506, The operation of each part with the table updating unit 507 will be specifically described. First, moving to the description of FIGS. 6 and 7, the coarsely divided sections set in the table generation unit 501 will be specifically described.

FIG. 6 is an explanatory diagram showing normal intervals and roughly divided intervals that the state values of the wind power generator under stall control can take. For example, as shown in graph 601 of FIG. 6, the uniform division normal division method divides the entire interval for output power and wind speed into a plurality of normal intervals. For example, the overall interval of output power is usually divided into intervals 1, 2, . . . , npsi. The overall wind speed interval is divided into normal intervals 1, 2, . . . , nfsi. Then, for example, as shown in a graph 602 in FIG. 6, the two regions with the larger output power are set as objects to be combined, and the entire interval regarding the output power and the wind speed is divided into a plurality of coarsely divided intervals. . For example, the entire interval of output power is divided into coarse division intervals 1, 2, . . . , npsi-1. The entire wind speed interval is divided into coarse division intervals 1, 2, . . . , nfsi-1. Next, the description of FIG. 7 will be described.

FIG. 7 is an explanatory diagram showing normal intervals and roughly divided intervals that the state values of the pitch-controlled wind power generator can take. For example, as shown in a graph 701 of FIG. 7, the uniform division normal division method divides the entire interval for output power and wind speed into a plurality of normal intervals. For example, the entire interval of output power is usually divided into intervals 1, 2, . . . , nppi. The entire wind speed interval is divided into normal intervals 1, 2, . . . , nfpi. Then, for example, as shown in a graph 702 in FIG. 7, the two regions with the larger output power are set as objects to be combined, and the entire interval regarding the output power and the wind speed is divided into a plurality of coarsely divided intervals. . For example, the entire interval of the output power is divided into coarse division intervals 1, 2, . . . , nppi-1. The entire wind speed interval is divided into coarse division intervals 1, 2, . . . , nfpi-1.

Next, moving on to the description of FIG. 8, a specific example of realizing a normal table 801 for normal sections created by the table generation unit 501 and a rough division table 802 for rough division sections switched from the normal table 801 will be described. explained in detail.

FIG. 8 is an explanatory diagram showing an example of realizing the normal table 801 and the rough division table 802. As shown in FIG. In FIG. 8, the reinforcement learning device 100 uses the records of the normal table 801 as the records of the rough division table 802, so that the normal table 801 and the rough division table 802 can be mutually converted.

The reinforcement learning device 100 creates a normal table 801, for example. When converting the normal table 801 into the coarse division table 802, the reinforcement learning device 100 diverts a record corresponding to one normal section of two or more combined normal sections as a record corresponding to the combined coarse division section. Then, the reinforcement learning device 100 deletes the validity information set in the record of the other normal section. For example, when combining the normal section npsi-1 and the normal section npsi, the reinforcement learning device 100 uses the record corresponding to the normal section npsi-1 as the record corresponding to the coarsely divided section npsi-1. The reinforcement learning device 100 deletes, for example, validity information set in the record corresponding to the section npsi.

Further, when converting the coarsely divided table 802 into the normal table 801, the reinforcement learning device 100 converts the record corresponding to the combined section into a record corresponding to one of two or more sections divided from the combined section. Divert. Then, the reinforcement learning device 100 sets validity information again in the record of the other section. For example, when dividing a roughly divided section npsi-1 into a normal section npsi-1 and a normal section npsi, the reinforcement learning device 100 assigns a record corresponding to the roughly divided section npsi-1 to the normal section npsi-1. It is diverted as a record to be used. The reinforcement learning device 100, for example, resets validity information to the record corresponding to the normal section npsi.

Next, an example of the contents stored in the normal table 801 created by the table generation unit 501 will be described with reference to FIG. The normal table 801 is realized, for example, by a storage area such as the memory 302 or the recording medium 305 of the reinforcement learning device 100 shown in FIG. The description of the normal table 801 below corresponds to the case of using Q-learning as the reinforcement learning method, and the stored contents may differ when using different reinforcement learning methods.

FIG. 9 is an explanatory diagram showing an example of the contents stored in the normal table 801. As shown in FIG. As shown in FIG. 9, the normal table 801 has fields for status value, command value, and effective value. The normal table 801 stores validity information as a record by setting information in each field.

In the state value field, an interval in which the state value of the power generation system 200 can be set is set. The state values for the power generation system 200 include state values for the wind power generator and state values for the power generation system 200 as a whole. In the example of FIG. 9, the state values for the wind generator are the output watts and wind speed values for the stall controlled wind generator and the output watts and wind speed values for the pitch controlled wind generator. Moreover, the state value related to the power generation system 200 as a whole is the power demand watt value in the power generation system 200 as a whole.

A command value for the wind power generator is set in the command value field. In the example of FIG. 9, the command value for the wind power generator is a command value for switching the power supply of the wind power generator for stall control between ON and OFF. The command value for the wind power generator is a command value indicating how much the pitch angle of the pitch-controlled wind power generator is to be changed. Specifically, the command values indicating how much the pitch angle should be changed are -ΔΘ, ±0 and +ΔΘ. The effective value field is set with effective values indicating the effectiveness of combinations of command values for each wind power generator when each state value is included in one of the sections.

Next, moving to the description of FIG. 10, the section switching unit 502 converts the normal table 801 into the rough division table 802, or converts the rough division table 802 into the normal table 801. An example of creating a characteristic function for an aircraft will be described. A characteristic function related to the wind power generator may be input to the reinforcement learning device 100 in advance.

FIG. 10 is an explanatory diagram showing an example of creating a characteristic function. In FIG. 10, the reinforcement learning device 100 creates a characteristic function regarding the wind power generator. The reinforcement learning device 100 acquires, for example, output watt values from stall-controlled wind turbines at various wind speeds. The reinforcement learning device 100 also acquires the rated wind speed, maximum output, cut-in wind speed, and cut-out wind speed.

Next, the reinforcement learning device 100 draws a characteristic curve indicated by the characteristic function for the stall control wind power generator based on the rated wind speed, maximum output, cut-in wind speed, cut-out wind speed, and output watt values at various wind speeds. An approximation curve f _i (t) is obtained. For example, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) with a shape of y=0 from the wind speed of 0 to the cut-in wind speed.

For example, from the cut-in wind speed to the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y=a*x^3 based on the output watt values at various wind speeds. . For example, after the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the form of y=b*x^2 based on output watt values at various wind speeds. As a result, the reinforcement learning device 100 obtains the approximated curve f _i (t) as shown in the graph 1000 of FIG. 10 .

The reinforcement learning device 100 also acquires the output watt values from the pitch-controlled wind generator at various wind speeds, for example, at pitch angles Θ=0, ΔΘ, 2ΔΘ, . . . , kΔΘ. The reinforcement learning device 100 also acquires the rated wind speed, maximum output, cut-in wind speed, and cut-out wind speed.

Next, the reinforcement learning device 100 draws a characteristic curve indicated by the characteristic function for the pitch-controlled wind power generator based on the rated wind speed, maximum output, cut-in wind speed, cut-out wind speed, and output watt values at various wind speeds. An approximation curve f _i (t) is obtained. For example, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) with a shape of y=0 from the wind speed of 0 to the cut-in wind speed.

For example, from the cut-in wind speed to the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y=a*x^3 based on the output watt values at various wind speeds. . For example, after the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the form of y=b*x^2 based on output watt values at various wind speeds. As a result, the reinforcement learning device 100 obtains approximate curves f _i (t) as shown in the graph 1000 of FIG. 10 at the pitch angles Θ=0, ΔΘ, 2ΔΘ, . . . , kΔΘ.

Next, referring to FIG. 11, the section switching unit 502 converts the normal table 801 into a rough division table 802 based on the approximated curve f _i (t) of the specific function created in advance, and the table to be used is to the rough division table 802 will be described.

FIG. 11 is an explanatory diagram showing an example of switching the table to be used to the rough division table 802. As shown in FIG. In FIG. 11, the reinforcement learning device 100 sets the coarse division table 802 as a table to be used because the threshold α>power demand watt value P′(t _j ). In the example of FIG. 11, a case will be described in which a valid value is set in the field indicated by ● 1101, which corresponds to the roughly divided section npsi-1 obtained by combining the normal section nps1-1 and the normal section nps1.

Reinforcement learning device 100 identifies, for example, a record corresponding to coarsely divided section nps1-1. The reinforcement learning device 100 _acquires the wind speed values F _s1 , . The reinforcement learning device 100 also acquires the output _watt values P _p1 , . The reinforcement learning device 100 also _acquires the wind speed values F _p1 , . In addition, the reinforcement learning device 100 acquires the power demand watt value P' for the entire power generation system 200 set in the specified record.

Reinforcement learning device 100 _{predicts output power P' si = f} Calculate _i (F _si ). In addition, the reinforcement learning device 100 determines the predicted output power P' _si =0 when the power supply is turned off for the stall-controlled wind power generator i.

Further, the reinforcement learning device 100 _calculates f _{i , Θ} ( Determine the pitch angle Θ such that F _pi )≈P _pi . Next, the reinforcement learning device 100 calculates predicted output power P' _pi =f _i, Θ _- ΔΘ (F _pi ), f _i, Θ when −ΔΘ, ±0, and +ΔΘ are applied to the determined pitch angle Θ. (F _pi ), calculate f _i, Θ ₊ ΔΘ(F _pi ).

Reinforcement learning device 100 calculates predicted output power P′ _si =f _i (F _si ), predicted output power P′ _pi =f _i, Θ ₋ ΔΘ(F _pi ), f _i, Θ(F _pi ), f _{i ,} Θ ₊ ΔΘ(F _pi ) to create a predicted output power table 1100 . The reinforcement learning device 100 generates command values a1, . . . , an for the stall control wind power generator and command values b1, . to get Then, the reinforcement learning device 100 acquires the predicted output power P' _si and the predicted output power P' _pi corresponding to the obtained command value from the created predicted output power table 1100, and uses the following formula (1): Then, the predicted output power P˜ of the power generation system 200 as a whole is calculated.

Then, the reinforcement learning device 100 calculates the effective value Q=r(P") based on the difference value P" between the acquired power demand watt value P' and the calculated predicted output power P~, The field indicated by 1101 is set. r(P″) is defined by the following equations (2) to (5). Specifically, when P″ is defined by the following equation (2) and δ>0 is set, P″ > δ, r (P") is defined by the following formula (3), and when -δ ≤ P" ≤ δ, r (P") is defined by the following formula (4), and P" < In the case of −δ, r(P″) is defined by the following equation (5).

The reinforcement learning device 100 similarly sets valid values for other fields. Thereby, the reinforcement learning device 100 can calculate the effective value such that the effective value increases as the total output watt value approaches the power demand watt value. For this reason, the reinforcement learning device 100 can refer to the effective value to facilitate appropriate control of the power generation system 200 . In addition, the reinforcement learning device 100 can reduce the number of pieces of effectiveness information to be updated by learning, thereby reducing the amount of processing required for reinforcement learning.

12, the section switching unit 502 converts the rough division table 802 into the normal table 801 based on the approximated curve f _i (t) of the specific function created in advance, and converts the table to be used. to the normal table 801 will be described.

FIG. 12 is an explanatory diagram showing an example of switching the table to be used to the normal table 801. As shown in FIG. In FIG. 12, the reinforcement learning apparatus 100 sets the table to be used in the normal table 801 because the threshold value α≦power demand watt value P′(t _j ). In the example of FIG. 12, a case will be described in which a valid value is set in the field indicated by ● 1201 corresponding to the normal section nps1-1 divided from the coarsely divided section npsi-1.

Reinforcement learning device 100 identifies, for example, a record corresponding to coarsely divided section npsi-1. Next, the reinforcement learning device 100 _acquires the wind speed values F _s1 , . The reinforcement learning device 100 also acquires the output _watt values P _p1 , . The reinforcement learning device 100 also _acquires the wind speed values F _p1 , . In addition, the reinforcement learning device 100 acquires the power demand watt value P' for the entire power generation system 200 set in the specified record.

Reinforcement learning device 100 _{predicts output power P' si = f} Calculate _i (F _si ). In addition, the reinforcement learning device 100 determines the predicted output power P' _si =0 when the power supply is turned off for the stall-controlled wind power generator i.

Further, the reinforcement learning device 100 _calculates f _{i , Θ} ( Determine the pitch angle Θ such that F _pi )≈P _pi . Next, the reinforcement learning device 100 calculates predicted output power P' _pi =f _i, Θ _- ΔΘ (F _pi ), f _i, Θ when −ΔΘ, ±0, and +ΔΘ are applied to the determined pitch angle Θ. (F _pi ), calculate f _i, Θ ₊ ΔΘ(F _pi ).

Reinforcement learning device 100 calculates predicted output power P′ _si =f _i (F _si ), predicted output power P′ _pi =f _i, Θ ₋ ΔΘ(F _pi ), f _i, Θ(F _pi ), f _{i ,} Θ ₊ ΔΘ(F _pi ) to create a predicted output power table 1200 . The reinforcement learning device 100 generates command values a1, . . . , an for the stall control wind power generator and command values b1, . to get Reinforcement learning device 100 acquires predicted output power P′ _si and predicted output power P′ _pi corresponding to the obtained command value from predicted output power table 1200 created above, and uses the above equation (1). Then, the predicted output power P˜ of the power generation system 200 as a whole is calculated.

Then, the reinforcement learning device 100 calculates the effective value Q=r(P") based on the difference value P" between the acquired power demand watt value P' and the calculated predicted output power P~, The field indicated by 1201 is set. Specifically, r(P″) is defined by the above formulas (2) to (5).

Reinforcement learning apparatus 100 similarly sets valid values to other fields corresponding to the normal section nps1-1 and fields corresponding to the normal section npsi. Thereby, the reinforcement learning device 100 can calculate the effective value such that the effective value increases as the total output watt value approaches the power demand watt value. For this reason, the reinforcement learning device 100 can refer to the effective value to facilitate appropriate control of the power generation system 200 . In addition, the reinforcement learning device 100 increases the number of effectiveness information items to be updated by learning, and subdivides what kind of command value should be output for what kind of state value. can do.

As described above, the reinforcement learning device 100 has calculated effective values based on the approximated curve f _i (t) and the approximated curve f _i, Θ(t). Here, the current wind velocities are F _si and F _pi , and the wind velocities observed at the next time are F _si +ΔF _si and F _pi +ΔF _pi . In this case, the output powers observed at the following times are f _i (F _si +ΔF _si ), f _i, Θ _- ΔΘ (F _pi + ΔF _pi ), f _i, Θ (F _pi + ΔF _pi ), f _i, Θ ₊ ΔΘ(F _pi +ΔF _pi ). Furthermore, since the approximation curve is a continuous function, the output powers observed at the following times are f _i (F _si ) + ΔP _si , f _i, Θ _- ΔΘ (F _pi ) + ΔP _pi , f _i, Θ (F _pi ) + ΔP _pi , f _i, Θ ₊ ΔΘ(F _pi ) + ΔP _pi .

Also, if the demand power at the next time is P'+ΔP', the difference P″ _n between the demand power and the output power at the next time and the difference P″ between the demand power and the output power obtained at present are: The following formula (6) is established. Also, since the reward function is a continuous function, the following formula (7) holds.

If ΔF _si →0, ΔF _pi →0, and ΔP→0, then P″ _n →P″ and r(P″ _n )→r(P″) hold. Therefore, when the wind speed is stable and the change in power demand is small, P″ _{n≈P” and r(P″n )} ≈r(P″) are established. Therefore, if the reinforcement learning device 100 selects the command value combination with the largest effective value r(P″) by the ε-greedy algorithm, the total output power is brought closer to the demand power by the following equation (8). judged to be possible.

In this way, the reinforcement learning device 100 calculates the effective value based on the characteristic function, so that the wind power generator to be controlled can output power corresponding to the actual power demand. can. Therefore, the reinforcement learning device 100 can make it easier to select an appropriate command value than when initializing the valid value to 0 or setting the valid value randomly.

After that, the reinforcement learning device 100 selects and outputs a combination of the command value ai and the command value bi as an action at regular time intervals using the ε-greedy algorithm. For example, at time _tj , the reinforcement learning device 100 has a wind speed value F _si (t _j ), a wind speed value F _pi (t _j ), an output watt value P _si (t _j ), an output watt value P _pi ( t _j ) and the power demand watt value P′(t _j ) are obtained as state values.

The reinforcement learning device 100 identifies the interval F _si ∼(t _j ) to which the wind speed value F _si (t _j ) belongs and the interval F _pi ∼(t _j ) to which the wind speed value F _pi (t _j ) belongs. Reinforcement learning device 100 identifies section P _si ~(t _j ) to which output watt value P _si (t _j ) belongs and section P _pi ~(t _j ) to which output watt value P _pi (t _j ) belongs . Reinforcement learning device 100 identifies a section P'-(t _j ) to which power demand watt value P'(t _j ) belongs.

Then, the reinforcement learning device 100 randomly selects and outputs a combination of the command value ai and the command value bi with a probability of ε. Reinforcement learning device 100 identifies, with a probability of 1-ε, a record corresponding to a combination of sections to which the acquired state values belong, from normal table 801 or coarse partition table 802 set as a table to be used. Reinforcement learning device 100 selects and outputs a combination of command value ai and command value bi associated with the largest effective value in the identified record.

Further, the reinforcement learning device 100 calculates a reward value at time _tj for the combination of the command value ai and the command value bi output at time _tj-1 . The reinforcement learning device 100 calculates the reward value r(t _j ) by, for example, the following equations (9) to (12). Specifically, P″(t _j ) is defined by the following equation (9). When P″(t _j )>δ, r(t _j ) is defined by the following equation (10), When −δ≦P″(t _j )≦δ, r(t _j ) is defined by the following formula (11), and when P″(t _j )<−δ, r is defined by the following formula (12). Define (t _j ).

Next, with reference to FIG. 13, a case will be described in which the information processing apparatus updates the valid value based on the reward value calculated in response to outputting the combination of the command value ai and the command value bi. In the example of FIG. 13, the information processing device updates the effective value at time _tj+1 , for example, based on the reward value calculated for the combination of the command value ai and the command value bi output at time _tj . A case will be described.

FIG. 13 is an explanatory diagram of an example of updating valid values. In FIG. 13, the reinforcement learning device 100 updates the valid value Q set in the field indicated by ● 1301 associated with the record corresponding to the combination of the sections to which the state values acquired at time t _j belong. Reinforcement learning device 100, for example, using the following formula (13) and the following formula (14), based on the reward value calculated at time _tj+1 , calculates the effective value Q', the field indicated by ● 1301 Update the valid value Q set to

If the record corresponding to the combination of the sections to which the state values acquired at time t _j+1 belong is the record corresponding to the coarsely divided section or the record corresponding to the normal section that can be combined, the reinforcement learning device 100 An effective value Q' is calculated using the following formula (13). In addition, the reinforcement learning device 100 ensures that the record corresponding to the combination of sections to which the state value acquired at time t _j+1 belongs is a record corresponding to a coarsely divided section or a record corresponding to a normal section that can be combined. For example, the effective value Q' is calculated using the following equation (14).

In the example of FIG. 13, the reinforcement learning device 100 determines that the record corresponding to the combination of sections to which the state value acquired at time t _j+1 belongs corresponds to the record corresponding to the coarsely divided section or to the normal section that can be combined. It is determined that it is not a record that Reinforcement learning device 100 calculates effective value Q′ using the above equation (14) based on effective value Q and the state value of field 1311, the value obtained by substituting the effective value of field 1312 into the max function.

On the other hand, the record corresponding to the combination of the sections to which the state value acquired by the reinforcement learning device 100 at time t _j+1 belongs is the record corresponding to the coarsely divided section or the record corresponding to the normal section that can be combined. It may be determined that there is In this case, the reinforcement learning device 100 calculates the effective value Q′ based on the state value of the field 1310, the state value of the field 1320, and the effective value Q using the above equation (13). As a result, the reinforcement learning device 100 can update the effectiveness information and facilitate appropriate control of the power generation system 200 .

(Example of operation of reinforcement learning device 100 for power generation system 200 including a thermal power generator)
Next, an operation example of the reinforcement learning device 100 for a power generation system 200 including a thermal power generator will be described with reference to FIGS. 14 and 15. FIG. First, moving to the description of FIG. 14, a specific configuration example of a power generation system 200 including a thermal power generator will be described.

FIG. 14 is an explanatory diagram showing a specific configuration example of a power generation system 200 including a thermal power generator. In the example of FIG. 14 , the power generation system 200 includes a reinforcement learning device 100 and a fuel-controlled thermal power generator i (i=1, . . . , m). A fuel-controlled thermal power generator i receives a command value bi (i=1, . . . , m) from the reinforcement learning device 100 .

The power generation system 200 includes a fuel gauge pi (i=1, . . . , m) for a fuel controlled thermal power generator i. The fuel gauge pi transmits the fuel consumption F _pi (t _j ) to the reinforcement learning device 100 . Power generation system 200 includes a power meter for fuel controlled thermal generator i. The power meter for fuel-controlled thermal generator i sends the output wattage value P _pi to the reinforcement learning device 100 .

Reinforcement learning device 100 includes table generator 501 , interval switcher 502 , value setter 503 , action determiner 504 , state calculator 505 , reward calculator 506 , and table updater 507 . Here, the operation of each part of the reinforcement learning device 100 for the power generation system 200 including the thermal power generator is the same as the operation of each part of the reinforcement learning device 100 for the power generation system 200 including the wind power generator. , the description is omitted. Now, moving to the description of FIG. 15, an example of the contents stored in the normal table 801 in the power generation system 200 including the thermal power generator will be described.

FIG. 15 is an explanatory diagram showing an example of the contents of the normal table 801 regarding thermal power generators. As shown in FIG. 15, the normal table 801 has fields for status value, command value, and valid value. The normal table 801 stores validity information as a record by setting information in each field.

In the state value field, an interval in which the state value of the power generation system 200 can be set is set. The state values related to power generation system 200 include a state value related to the thermal power generator and a state value related to power generation system 200 as a whole. In the example of FIG. 15, the state values for the thermal generator are the output wattage value and the fuel usage for the fuel controlled thermal generator. Moreover, the state value related to the power generation system 200 as a whole is the power demand in the power generation system 200 as a whole.

A command value for the thermal power generator is set in the command value field. In the example of FIG. 15, the command value for the thermal power generator is a command value indicating how much the amount of fuel used by the thermal power generator under fuel control is to be changed. Specifically, the command values indicating how much the fuel consumption is to be changed are -X, ±0 and +X. The effective value field is set with effective values indicating the effectiveness of a combination of command values for each thermal power generator when each state value is included in one of the sections.

(Overall processing procedure)
Next, an example of an overall processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. 16 . The overall processing is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I/F 303 shown in FIG.

FIG. 16 is a flow chart showing an example of the overall processing procedure. In FIG. 16, the reinforcement learning device 100 creates a table for a plurality of normal intervals, and sets two or more normal intervals that can be roughly divided intervals (step S1601).

Next, the reinforcement learning device 100 executes switching determination processing, which will be described later with reference to FIG. 17, and sets a table to be used (step S1602). Then, the reinforcement learning device 100 executes the value setting process described later with reference to FIG. 18, and sets valid values in the set table (step S1603).

Next, the reinforcement learning device 100 identifies the state of the wind power generator based on the output watt value, wind speed value, and power demand watt value from the wind power generator (step S1604). Then, the reinforcement learning device 100 calculates a reward from the wind power generator based on the output watt value and the power demand watt value from the wind power generator (step S1605).

Next, the reinforcement learning device 100 uses the set table to determine and output a command value for the wind power generator (step S1606). Then, the reinforcement learning device 100 updates the effective value stored in the set table based on the calculated reward (step S1607). After that, the reinforcement learning device 100 returns to the process of step S1602.

(Switching determination processing procedure)
Next, an example of a switching determination processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. 17 . The switching determination process is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I/F 303 shown in FIG.

FIG. 17 is a flowchart illustrating an example of a switching determination processing procedure; In FIG. 17, the reinforcement learning device 100 sets a threshold α (step S1701). Next, the reinforcement learning device 100 determines whether or not α>P'(t _j ) for the power demand watt value P'(t _j ) (step S1702).

If α>P′(t _j ) (step S1702: Yes), the reinforcement learning device 100 proceeds to step S1707. On the other hand, if α>P'(t _j ) is not satisfied (step S1702: No), the reinforcement learning device 100 proceeds to the process of step S1703.

At step S1703, the reinforcement learning device 100 determines to use the normal table 801 for the normal section (step S1703). Next, the reinforcement learning device 100 determines whether or not the normal table 801 for the normal section was used until immediately before (step S1704).

Here, if the normal table 801 for the normal section is used (step S1704: Yes), the reinforcement learning device 100 proceeds to the process of step S1706. On the other hand, if the normal table 801 for the normal section is not used (step S1704: No), the reinforcement learning device 100 proceeds to the process of step S1705.

At step S1705, the reinforcement learning device 100 creates the normal table 801 for the normal section and sets it as the table to be used (step S1705). Then, the reinforcement learning device 100 ends the switching determination process.

At step S1706, the reinforcement learning apparatus 100 sets the table that was used immediately before as the table to be used (step S1706). Then, the reinforcement learning device 100 ends the switching determination process.

At step S1707, the reinforcement learning device 100 determines to use the coarse division table 802 for the coarse division section (step S1707). Next, the reinforcement learning device 100 determines whether or not the coarse division table 802 for the coarse division section was used until immediately before (step S1708).

Here, if the coarse division table 802 for the coarse division section is used (step S1708: Yes), the reinforcement learning device 100 proceeds to the process of step S1706. On the other hand, if the coarse division table 802 for the coarse division section is not used (step S1708: No), the reinforcement learning device 100 proceeds to the process of step S1709.

In step S1709, the reinforcement learning device 100 creates a coarse division table 802 for the coarse division section and sets it as a table to be used (step S1709). Then, the reinforcement learning device 100 ends the switching determination process.

(Value setting processing procedure)
Next, an example of the value setting processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. 18 . The value setting process is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I/F 303 shown in FIG.

FIG. 18 is a flowchart illustrating an example of a value setting procedure. In FIG. 18, the reinforcement learning device 100 identifies, from the table to be used, a record corresponding to a normal section into which coarsely divided sections are divided or a coarsely divided section into which normal sections are combined (step S1801).

Next, the reinforcement learning device 100 acquires the wind speed values F _s1 , . . . , F _sn for the stall control wind power generator i (i=1, . step S1802). Then, the reinforcement learning device 100 acquires the output watt values P _p1 , . . . , P _pm from the pitch-controlled wind power generator i (i=1, . (Step S1803).

Next, the reinforcement learning device 100 _acquires the wind speed values F _p1 , . step S1804). Then, the reinforcement learning device 100 acquires the power demand watt value P' for the entire power generation system 200 set in the specified record (step S1805).

Next, the _{reinforcement learning} device 100 performs power supply is turned on, the predicted output power f _i (F _si ) is calculated (step S1806). Then, the reinforcement learning device 100 calculates the observed wind speed F _pi , the output watt value P _pi , and the approximation curve f _i, Θ(t ), the pitch angle Θ that satisfies f _i, Θ(F _pi )≈P _pi is determined (step S1807).

Next, the reinforcement learning device 100 obtains predicted output powers f _i, Θ _- ΔΘ (F _pi ) and f _i, Θ (F _pi ) when −ΔΘ, ±0, and +ΔΘ are applied to the determined pitch angle Θ. , f _i, Θ ₊ ΔΘ(F _pi ) are calculated (step S1808). Then, the reinforcement learning device 100 calculates the predicted output power f _i (F _si ), the predicted output power f _i, Θ ₋ ΔΘ (F _pi ), f _i, Θ (F _pi ), f _i, Θ ₊ ΔΘ (F _pi ), a predicted output power table is created (step S1809).

Next, the reinforcement learning device 100 calculates command values a1, . . . , an for the stall control wind power generators and command values b1, . , bm are acquired (step S1810). Then, the reinforcement learning device 100 calculates the predicted output power P∼ of the entire power generation system 200 based on the obtained command value and the prepared predicted output power table for each field in the specified record (step S1811). .

Next, the reinforcement learning device 100 calculates and sets an effective value based on the difference value between the demand power watt value P′ and the calculated predicted output power P~ for each field in the specified record (step S1812). Then, the reinforcement learning device 100 ends the value setting process.

(Characteristic function creation processing procedure)
Next, an example of a characteristic function creation processing procedure executed by the reinforcement learning device 100 will be described with reference to FIGS. 19 and 20. FIG. The characteristic function creation process is realized by, for example, the CPU 301, storage areas such as the memory 302 and the recording medium 305, and the network I/F 303 shown in FIG.

FIG. 19 is a flow chart showing an example of a characteristic function creation processing procedure for a stall control wind power generator. In FIG. 19, the reinforcement learning device 100 observes output watt values from stall-controlled wind power generators at various wind speeds (step S1901).

Next, the reinforcement learning device 100 obtains an approximation curve f _i (t) that approximates the characteristic curve indicated by the characteristic function of the stall control wind power generator based on the output watt values at various wind speeds (step S1902). . Then, the reinforcement learning device 100 ends the characteristic function creation process for the stall control wind power generator.

FIG. 20 is a flowchart showing an example of a characteristic function creation processing procedure for a pitch-controlled wind power generator. In FIG. 20, the reinforcement learning device 100 acquires the maximum pitch angle MΘ in the pitch-controlled wind power generator (step S2001).

Next, the reinforcement learning device 100 sets the pitch angle Θ=0 for the pitch-controlled wind power generator (step S2002). Then, the reinforcement learning device 100 determines whether or not Θ<MΘ (step S2003).

Here, if Θ<MΘ (step S2003: Yes), the reinforcement learning device 100 proceeds to the process of step S2004. On the other hand, if Θ<MΘ is not true (step S2003: No), the reinforcement learning device 100 terminates the characteristic function creation process for the pitch-controlled wind power generator.

In step S2004, the reinforcement learning apparatus 100 observes output watt values from the pitch-controlled wind power generator at various wind speeds (step S2004). Next, the reinforcement learning device 100 obtains an approximation curve f _i, Θ(t) that approximates the characteristic curve indicated by the characteristic function of the pitch-controlled wind power generator based on the output watt values at various wind speeds (step S2005).

Then, the reinforcement learning device 100 sets the pitch angle Θ=Θ+ΔΘ (step S2006). After that, the reinforcement learning device 100 returns to the process of step S2003.

Here, the reinforcement learning device 100 may change the order of the processing of some steps in the various flowcharts described above. Further, the reinforcement learning device 100 may omit the processing of some steps in the various flowcharts described above.

As described above, according to the reinforcement learning device 100, the effectiveness information indicating the effectiveness of each command value for the generator 201 in each of the plurality of regions where the state value of the generator 201 can take is used. can learn. According to the reinforcement learning device 100, with reference to the observed state value of the generator 201, based on the characteristic function, for each command value for the generator 201 in a region combining two or more consecutive regions among a plurality of regions Validity information can be generated that indicates validity. According to the reinforcement learning device 100, learning can be performed using the validity information about the generated combined regions and the validity information about each region other than two or more regions among the plurality of regions. . As a result, the reinforcement learning device 100 can reduce the number of pieces of effectiveness information to be updated through learning, thereby reducing the amount of processing required for reinforcement learning.

According to the reinforcement learning device 100, the observed state value of the generator 201 is referred to, and based on the characteristic function, effectiveness information indicating the effectiveness of each command value for the generator 201 in each of two or more regions. can be generated. According to the reinforcement learning device 100, learning is performed using the validity information for each of the generated two or more regions and the validity information for each region other than the two or more regions among the plurality of regions. It can be performed. As a result, the reinforcement learning device 100 can subdivide which command value should be output for which state value.

According to the reinforcement learning device 100, learning is performed using effectiveness information indicating the effectiveness of each combination of the command values of the generator 201 in each of a plurality of regions that can be taken by the combination of the state values of the generator 201. It can be performed. According to the reinforcement learning device 100, the observed state value of the generator 201 is referred to, and based on the characteristic function, the command value of the generator 201 in a region obtained by combining two or more consecutive regions among a plurality of regions. Efficacy information can be generated that indicates the efficacy of each combination. According to the reinforcement learning device 100, learning can be performed using the validity information about the generated combined regions and the validity information about each region other than two or more regions among the plurality of regions. . Thereby, the reinforcement learning device 100 can be applied when there are a plurality of generators 201 .

The reinforcement learning device 100 can generate effectiveness information about the wind power generator. Thereby, the reinforcement learning device 100 can be applied to the power generation system 200 including the wind power generator.

According to the reinforcement learning device 100, it is possible to identify the output power corresponding to the observed wind speed based on the characteristic function, and to generate effectiveness information about the combined region based on the identified output power. Thereby, the reinforcement learning device 100 can generate effectiveness information about the stall control wind power generator.

According to the reinforcement learning device 100, among the plurality of characteristic functions that differ for each wind performance of the generator 201, based on the characteristic function corresponding to the observed wind speed and output power, generate effectiveness information about the combined area. can do. Thereby, the reinforcement learning device 100 can generate effectiveness information about the pitch-controlled wind power generator.

The reinforcement learning device 100 can generate effectiveness information about a thermal power generator. Thereby, the reinforcement learning device 100 can be applied to a power generation system 200 including a thermal power generator.

According to the reinforcement learning device 100, when the observed power demand is equal to or less than the threshold, it is possible to generate effectiveness information about the combined regions. As a result, reinforcement learning apparatus 100 can reduce the number of pieces of effectiveness information to be updated through learning when it is not necessary to perform detailed verification for regions with relatively high output power. Here, the reinforcement learning device 100 can suppress adverse effects on the control of the power generation system by combining regions of relatively large output power.

According to the reinforcement learning device 100, when the observed power demand exceeds the threshold, effectiveness information about the combined regions can be generated. As a result, the reinforcement learning apparatus 100 can reduce the number of pieces of effectiveness information to be updated by performing learning when it is not necessary to verify in detail a region of relatively low output power. Here, the reinforcement learning device 100 can suppress adverse effects on the control of the power generation system by combining regions of relatively small output power.

According to the reinforcement learning device 100, effectiveness information for a combined area of two or more areas can be generated based on effectiveness information for each of the two or more areas. As a result, even if the characteristic function is unknown, the reinforcement learning device 100 can reduce the number of pieces of effectiveness information to be updated by learning, thereby reducing the amount of processing required for reinforcement learning. .

The reinforcement learning method described in this embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. The reinforcement learning program described in the present embodiment is recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, MO, DVD, etc., and executed by being read from the recording medium by a computer. Also, the reinforcement learning program described in the present embodiment may be distributed via a network such as the Internet.

Further, the following additional remarks are disclosed with respect to the above-described embodiment.

(Appendix 1) to the computer,
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
By referring to the observed state value of the generator, and based on the characteristic function of the state value of the generator, each command value for the generator in a region combining two or more consecutive regions among the plurality of regions generate validity information indicating the validity of
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning program characterized by executing processing.

(Appendix 2) In the computer,
Effectiveness indicating the effectiveness of each command value for the generator in each of the two or more regions, with reference to the observed state value of the generator and based on a characteristic function of the state value of the generator generate information,
learning using validity information for each of the generated two or more regions and validity information for each region other than the two or more regions among the plurality of regions;
The reinforcement learning program according to Supplementary Note 1, characterized by causing a process to be executed.

(Appendix 3) In the computer,
When there are a plurality of generators, learning using effectiveness information indicating the effectiveness of each combination of command values of the generators in each of a plurality of regions that can be taken by the combination of the state values of the generators. and
Referring to the observed state values of the generator, and based on the characteristic function, the effectiveness of each combination of command values of the generator in a region obtained by combining two or more consecutive regions among the plurality of regions. generate validity information indicating
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
3. The reinforcement learning program according to appendix 1 or 2, characterized by causing a process to be executed.

(Appendix 4) The generator is a wind power generator,
4. The reinforcement learning program according to any one of appendices 1 to 3, wherein the state values relating to the generator are wind speed and output power.

(Appendix 5) The characteristic function indicates the relationship between the wind speed and the output power from the generator,
The process of generating effectiveness information for the combined region includes identifying output power corresponding to the observed wind speed based on the characteristic function, and determining effectiveness for the combined region based on the identified output power. The reinforcement learning program according to appendix 4, which generates sex information.

(Additional remark 6) The generator can change the wind reception performance,
The command value is a command value for controlling wind reception performance,
The characteristic function indicates the relationship between wind speed and power output from the generator,
The process of generating effectiveness information for the combined region includes, from among the plurality of characteristic functions that differ for each wind reception performance of the generator, based on the characteristic function corresponding to the observed wind speed and output power, 5. The reinforcement learning program of Claim 4, wherein the program generates efficacy information for bound regions.

(Appendix 7) The generator is a thermal power generator,
4. The reinforcement learning program according to any one of Appendices 1 to 3, wherein the state values relating to the generator are fuel consumption and output power.

(Supplementary note 8) A supplementary note characterized in that the process of generating validity information about the combined area generates the validity information about the combined area when the observed power demand is equal to or less than a threshold. The reinforcement learning program according to any one of 1 to 7.

(Supplementary note 9) Supplementary note 1 characterized in that the process of generating validity information about the combined area generates the validity information about the combined area when the observed power demand exceeds a threshold. 8. The reinforcement learning program according to any one of 7.

(Appendix 10) The computer
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
By referring to the observed state value of the generator, and based on the characteristic function of the state value of the generator, each command value for the generator in a region combining two or more consecutive regions among the plurality of regions generate validity information indicating the validity of
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning method characterized by executing a process.

(Appendix 11) learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of areas that the state value related to the generator can take,
By referring to the observed state value of the generator, and based on the characteristic function of the state value of the generator, each command value for the generator in a region combining two or more consecutive regions among the plurality of regions generate validity information indicating the validity of
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning device comprising a control unit.

(Appendix 12) to the computer,
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
Based on the effectiveness information indicating the effectiveness of each command value for the generator in each of two or more consecutive areas among the plurality of areas, for the generator in the area combining the two or more areas Generate effectiveness information that indicates the effectiveness of each command value,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning program characterized by executing processing.

(Appendix 13) The computer
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
Based on the effectiveness information indicating the effectiveness of each command value for the generator in each of two or more consecutive areas among the plurality of areas, for the generator in the area combining the two or more areas Generate effectiveness information that indicates the effectiveness of each command value,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning method characterized by executing a process.

(Appendix 14) learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of areas that the state value related to the generator can take,
Based on the effectiveness information indicating the effectiveness of each command value for the generator in each of two or more consecutive areas among the plurality of areas, for the generator in the area combining the two or more areas Generate effectiveness information that indicates the effectiveness of each command value,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning device comprising a control unit.

100 Reinforcement Learning Device 200 Power Generation System 201 Generator 210 Network 300 Bus 301 CPU
302 memory 303 network I/F
304 recording medium I/F
305 recording medium 400 storage unit 401 acquisition unit 402 switching unit 403 learning unit 404 output unit 501 table generation unit 502 section switching unit 503 value setting unit 504 action determination unit 505 state calculation unit 506 reward calculation unit 507 table update unit 601, 602, 701, 702, 1000 graph 801 normal table 802 rough division table 1100, 1200 predicted output power table 1310 to 1312, 1320 field

Claims (9)

to the computer,
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
If the power demand is less than or equal to a predetermined threshold, the state value including the observed output voltage of the generator is referred to, and based on a characteristic function of the state value of the generator, two consecutive regions out of the plurality of regions are selected. Generating effectiveness information indicating the effectiveness of each command value for the generator in an area obtained by combining the above areas,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning program characterized by executing processing. to the computer;
When the demand power exceeds a predetermined threshold, referring to the state value including the observed output voltage for the generator, based on the characteristic function for the state value for the generator, in each of the two or more regions generating validity information indicating the validity of each command value for the generator;
learning using validity information for each of the generated two or more regions and validity information for each region other than the two or more regions among the plurality of regions;
2. The reinforcement learning program according to claim 1, causing a process to be executed. to the computer;
When there are a plurality of generators, learning using effectiveness information indicating the effectiveness of each combination of command values of the generators in each of a plurality of regions that can be taken by the combination of the state values of the generators. and
Referring to the observed state values of the generator, and based on the characteristic function, the effectiveness of each combination of command values of the generator in a region obtained by combining two or more consecutive regions among the plurality of regions. generate validity information indicating
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
3. The reinforcement learning program according to claim 1, causing a process to be executed. The generator is a wind power generator,
4. The reinforcement learning program according to any one of claims 1 to 3, wherein the state values relating to the generator are wind speed and output power. The characteristic function indicates the relationship between wind speed and power output from the generator,
The process of generating effectiveness information for the combined region includes identifying output power corresponding to the observed wind speed based on the characteristic function, and determining effectiveness for the combined region based on the identified output power. 5. The reinforcement learning program according to claim 4, wherein sexual information is generated. The generator can change the wind reception performance,
The command value is a command value for controlling wind reception performance,
The characteristic function indicates the relationship between wind speed and power output from the generator,
The process of generating effectiveness information for the combined region includes, from among the plurality of characteristic functions that differ for each wind reception performance of the generator, based on the characteristic function corresponding to the observed wind speed and output power, 5. The reinforcement learning program according to claim 4, wherein the program generates efficacy information about the connected regions. The generator is a thermal power generator,
4. The reinforcement learning program according to any one of claims 1 to 3, wherein the state values relating to the generator are fuel consumption and output power. the computer
Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
If the power demand is less than or equal to a predetermined threshold, the state value including the observed output voltage of the generator is referred to, and based on a characteristic function of the state value of the generator, two consecutive regions out of the plurality of regions are selected. Generating effectiveness information indicating the effectiveness of each command value for the generator in an area obtained by combining the above areas,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning method characterized by executing a process. Learning using effectiveness information indicating the effectiveness of each command value for the generator in each of a plurality of regions that the state value related to the generator can take,
If the demand power is equal to or less than a predetermined threshold, the observed state value of the generator is referred to, and based on a characteristic function of the state value including the output voltage of the generator, two consecutive regions out of the plurality of regions are selected. Generating effectiveness information indicating the effectiveness of each command value for the generator in an area obtained by combining the above areas,
learning using validity information about the generated combined regions and validity information about each region other than the two or more regions among the plurality of regions;
A reinforcement learning device comprising a control unit.

Images