JP2020061909A - Reinforcement learning program, reinforcement learning method, and reinforcement learning device - Google Patents

Abstract

To provide a reinforcement learning device capable of reducing the processing amount in reinforcement learning.SOLUTION: A reinforcement learning device 100 performs learning by using a piece of validity information that represents the effectiveness of each command value for a power generator in each area of the multiple areas where the state value for the power generator can be. The reinforcement learning device 100 generates a piece of validity information that represents the effectiveness of each command value for the power generator in an area that combines two or more consecutive areas in the multiple areas. The reinforcement learning device 100 learns by using the validity information about the combined area and each area other than two or more areas in multiple areas.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a reinforcement learning program, a reinforcement learning method, and a reinforcement learning device.

  Conventionally, a power generation system including one or more generators that utilize natural energy may be controlled by reinforcement learning. In the reinforcement learning, for example, a table in which valid values indicating the validity of the command value for the generator are associated with each other for each region where the state value related to the generator can be used. The table is, for example, a Q table. In the reinforcement learning, for example, learning for estimating an effective value for the command value is repeatedly performed, and the table is updated.

Japanese Patent Publication No. 2016-517104

  However, in the conventional technique, the processing amount in reinforcement learning may be increased. For example, as the number of regions in which the state value relating to the generator can be taken increases, the number of times of learning for estimating the effective value of the command value increases, resulting in an increase in the amount of processing in reinforcement learning.

  In one aspect, the present invention aims to reduce the amount of processing in reinforcement learning.

  According to one embodiment, the learning is performed using the validity information indicating the validity of each command value for the generator in each of the plurality of regions in which the state value related to the generator can be obtained, and the observation is performed. Effectiveness of each command value for the generator in a region in which two or more continuous regions are combined among the plurality of regions based on a characteristic function of the state value for the generator with reference to the state value for the generator Is generated, and learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions. A reinforcement learning program to be performed, a reinforcement learning method, and a reinforcement learning device are proposed.

  According to one aspect, it is possible to reduce the processing amount in reinforcement learning.

FIG. 1 is an explanatory diagram illustrating an example of the reinforcement learning method according to the embodiment. FIG. 2 is an explanatory diagram showing an example of the power generation system 200. FIG. 3 is a block diagram showing a hardware configuration example of the reinforcement learning device 100. FIG. 4 is a block diagram showing a functional configuration example of the reinforcement learning device 100. FIG. 5: is explanatory drawing which shows the specific structural example of the power generation system 200 containing a wind power generator. FIG. 6 is an explanatory diagram showing a normal section and a roughly divided section that can be taken by the state value of the stall-controlled wind power generator. FIG. 7: is explanatory drawing which shows the normal area and coarse division area which the state value of the pitch control wind power generator can take. FIG. 8 is an explanatory diagram showing an example of realizing the normal table 801 and the coarse division table 802. FIG. 9 is an explanatory diagram showing an example of the stored contents of the normal table 801. FIG. 10 is an explanatory diagram showing an example of creating the characteristic function. FIG. 11 is an explanatory diagram showing an example of switching the table to be used to the coarse division table 802. FIG. 12 is an explanatory diagram showing an example of switching the table to be used to the normal table 801. FIG. 13 is an explanatory diagram showing an example of updating valid values. FIG. 14: is explanatory drawing which shows the specific structural example of the power generation system 200 containing a thermal power generator. FIG. 15 is an explanatory diagram showing an example of the stored contents of the normal table 801 relating to the thermal power generator. FIG. 16 is a flowchart showing an example of the overall processing procedure. FIG. 17 is a flowchart showing an example of the switching determination processing procedure. FIG. 18 is a flowchart showing an example of the value setting processing procedure. FIG. 19 is a flowchart showing an example of a characteristic function creation processing procedure for a stall-controlled wind power generator. FIG. 20 is a flowchart showing an example of a characteristic function creation processing procedure for a pitch-controlled wind power generator.

  Hereinafter, embodiments of a reinforcement learning program, a reinforcement learning method, and a reinforcement learning apparatus according to the present invention will be described in detail with reference to the drawings.

(One Example of Reinforcement Learning Method According to Embodiment)
FIG. 1 is an explanatory diagram illustrating an example of the reinforcement learning method according to the embodiment. The reinforcement learning device 100 is a computer that applies reinforcement learning to a power generation system including one or more power generators and controls a power generation system including one or more power generators. The generator is, for example, a wind power generator or a thermal power generator.

  In the reinforcement learning, for example, the effective value indicating the effectiveness of each combination of the command values for the one or more generators in each of the plurality of regions in which the combination of the state values of the one or more generators can be associated is associated A table to represent is used. The table is, for example, a Q table. In reinforcement learning, for example, the state value related to the generator is observed, the command value for the generator is determined, the determined command value is input to the generator, and the input command value is valid in the area including the observed state value. The learning for estimating the effective value indicating the sex is repeated and the table is updated. Reinforcement learning is realized by, for example, Q learning or SARSA.

  Here, it is conceivable that the amount of processing in reinforcement learning may increase. For example, as the number of regions that can be combined by the state values of one or more generators increases, the number of times learning is performed increases, resulting in an increase in the amount of processing in reinforcement learning. Specifically, if the state values are finely divided to set areas, the number of areas increases. Further, specifically, as the number of generators increases, the number of areas also increases. Therefore, it is desired to reduce the amount of processing in reinforcement learning.

  On the other hand, it is conceivable to reduce the number of regions to reduce the amount of processing in reinforcement learning. For example, it is conceivable that the state value is roughly divided to set regions, the number of regions is reduced, and the processing amount in reinforcement learning is reduced. However, if a coarsely divided area is used at all times, it is not possible to verify in detail what kind of command value should be output in what kind of state value, and it is appropriate for the power generation system. It may not be possible to control. The appropriate control is, for example, control that maximizes the amount of power generation of the power generation system within a range that does not exceed a predetermined threshold value.

  Therefore, it is possible to reduce the processing amount in the reinforcement learning by dynamically changing the number of regions. For example, it is possible to combine two or more regions and reduce the number of regions at some timing. However, if it is not clear how to set an effective value that indicates the effectiveness of each combination of command values for one or more generators in a region that combines two or more regions, It becomes difficult to perform appropriate control.

  Therefore, in the present embodiment, in reinforcement learning, it is possible to dynamically change the number of regions that can be taken by a combination of state values related to one or more generators, and it is appropriate according to the number of regions being dynamically changed. A reinforcement learning method capable of resetting a valid value determined as follows will be described. According to this reinforcement learning method, it is possible to reduce the amount of processing in reinforcement learning.

  In the example of FIG. 1, the number of generators included in the power generation system is one. The state value related to the generator is, for example, output power from the generator and an environmental value related to the amount of natural energy supplied to the generator. The environmental value is, for example, the wind speed or the amount of fuel used. The command value for the generator is, for example, a command value for switching the power supply of the generator between ON and OFF. The command value for the generator is, for example, a command value for changing the utilization efficiency of natural energy in the generator.

  In FIG. 1, the reinforcement learning device 100 performs learning by using validity information indicating the validity of each command value for a generator in each of a plurality of regions in which a state value related to the generator can be taken. The region is, for example, a section or a combination of sections. The validity information is, for example, a record of the Q table. The effective value is, for example, the Q value. The reinforcement learning device 100, for example, validity information in which the valid value G1 is associated with the section A1, validity information in which the valid value G2 is associated with the section A2, and validity in which the valid value G3 is associated with the section A3. Learning is performed using the Q table including information and. The reinforcement learning device 100 updates the Q table as a result of learning.

  The reinforcement learning device 100 generates validity information indicating the validity of each command value for the generator in a region in which two or more continuous regions are combined among a plurality of regions. The reinforcement learning apparatus 100 refers to, for example, the observed state value regarding the generator, and generates validity information regarding the combined region based on the characteristic function regarding the state value regarding the generator. Specifically, the reinforcement learning device 100 calculates the effective value Ga indicating the effectiveness of the command value in the section Aa, which is a combination of the section A2 and the section A3, and the effectiveness in which the effective value Ga is associated with the section Aa. Generate information. Then, the reinforcement learning device 100 associates the validity information in which the valid value G2 is associated with the section A2, the validity information in which the valid value G3 is associated with the section A3, and the validity information Ga is associated with the section Aa. The Q table is updated by replacing with the sex information.

  The reinforcement learning device 100 performs learning by using the effectiveness information about the combined areas and the effectiveness information about each area other than two or more areas of the plurality of areas. The reinforcement learning device 100 includes, for example, a Q table including validity information in which the valid value G1 is associated with the section A1, and validity information in which the valid value Ga is associated with the section Aa in which the section A2 and the section A3 are combined. Use to learn. The reinforcement learning device 100 updates the Q table as a result of learning.

  Thereby, the reinforcement learning device 100 can combine two or more regions and dynamically reduce the number of regions to which validity information is associated. For this reason, the reinforcement learning apparatus 100 can reduce the number of validity information items to be learned and updated, and reduce the amount of processing required for reinforcement learning. Further, the reinforcement learning device 100 may set the valid value based on the characteristic function without initializing the valid value to 0 or randomly setting the valid value when generating the validity information. it can. Therefore, the reinforcement learning device 100 can accurately generate the validity information that is generated, indicating the validity of each command value for the power generator, and facilitate the appropriate control of the power generation system. it can.

  Here, the number of areas to which the validity information is associated by replacing the validity information about each of the two or more areas with the validity information about the area obtained by combining the two or more areas at some timing. However, the present invention is not limited to this. For example, at some timing, the validity information about a region obtained by combining two or more regions may be replaced with the validity information about each of the two or more regions to dynamically increase the number of regions. May be. Thereby, the reinforcement learning device 100 can subdivide and execute what kind of command value should be output in what kind of state value. A case where the validity information about the area obtained by combining the two or more areas is replaced with the validity information about each of the two or more areas will be specifically described later with reference to FIG. 12.

  Here, the case where the power generation system includes only one generator has been described, but the present invention is not limited to this. For example, there may be a plurality of generators included in the power generation system. In this case, for example, the reinforcement learning device 100 generates validity information about an area obtained by combining two or more continuous areas out of a plurality of areas that the combination of the state values of the generator can take. Then, the reinforcement learning device 100 replaces the validity information about each of the two or more regions with the validity information about the combined region, and dynamically reduces the number of regions to which the validity information is associated. Let As a result, the reinforcement learning apparatus 100 can reduce the amount of processing required for reinforcement learning. The case where there are a plurality of generators included in the power generation system will be described later with reference to FIGS. 11 and 12.

  Here, the reinforcement learning device 100 has been described without specifying the type of the generator included in the power generation system. On the other hand, for example, the generator included in the power generation system may be a wind power generator. A case where the generator included in the power generation system is a wind power generator will be specifically described later with reference to FIGS. 11 and 12. Further, for example, the generator included in the power generation system may be a thermal power generator. A case where the generator included in the power generation system is a thermal power generator will be specifically described later with reference to FIG. 15. Further, for example, the power generation system may include both a wind power generator and a thermal power generator.

  Here, the reinforcement learning device 100 has been described without limiting the timing of generating the validity information for the region in which two or more regions are combined. On the other hand, for example, the reinforcement learning device 100 may generate the validity information about the combined regions when the observed power demand is less than or equal to the threshold value. Further, for example, the reinforcement learning device 100 may generate validity information about the combined regions when the observed demand power exceeds a threshold value.

(Example of power generation system 200)
Next, an example of the power generation system 200 to which the reinforcement learning device 100 shown in FIG. 1 is applied will be described with reference to FIG.

  FIG. 2 is an explanatory diagram showing an example of the power generation system 200. In FIG. 2, the power generation system 200 includes the reinforcement learning device 100 and one or more power generators 201.

  In the power generation system 200, the reinforcement learning device 100 and one or more power generators 201 are connected via a wired or wireless network 210. The network 210 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or the Internet.

  The generator 201 is, for example, a machine that uses wind energy to generate electricity using a wind turbine. The generator 201 generates power using, for example, the wind turbine torque transmitted from the wind turbine. The generator 201 is provided with a measuring instrument for observing a state value related to the generator 201. The measuring instrument has, for example, a sensor device. The sensor device may include at least one of an acceleration sensor, a geomagnetic sensor, an optical sensor, a vibration sensor, a power sensor, a voltage sensor, and a current sensor. The generator 201 may be, for example, a machine that utilizes thermal energy and uses a turbine to generate electricity.

  The reinforcement learning device 100 applies reinforcement learning to the power generation system 200 and controls the power generation system 200. The reinforcement learning device 100 controls, for example, command values for one or more generators 201 included in the power generation system 200. The reinforcement learning device 100 specifically acquires a state value related to the generator 201 from a measuring device provided in the generator 201. The reinforcement learning device 100 determines and outputs a command value for the generator 201 based on the acquired state value and the table including the validity information. The reinforcement learning device 100 updates the table according to the result of outputting the command value. The reinforcement learning device 100 is, for example, a server, a PC (Personal Computer), a microcomputer, a PLC (Programmable Logic Controller), or the like.

(Example of Hardware Configuration of Reinforcement Learning Device 100)
Next, a hardware configuration example of the reinforcement learning device 100 will be described with reference to FIG.

  FIG. 3 is a block diagram showing a hardware configuration example of the reinforcement learning device 100. In FIG. 3, the reinforcement learning device 100 includes a CPU (Central Processing Unit) 301, a memory 302, a network I / F (Interface) 303, a recording medium I / F 304, and a recording medium 305. Further, each component is connected by a bus 300.

  Here, the CPU 301 controls the entire reinforcement learning device 100. The memory 302 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), and a flash ROM. Specifically, for example, a flash ROM or a ROM stores various programs, and a RAM is used as a work area of the CPU 301. The program stored in the memory 302 is loaded into the CPU 301 to cause the CPU 301 to execute the coded processing.

  The network I / F 303 is connected to the network 210 via a communication line, and is connected to another computer via the network 210. The other computer is, for example, the generator 201. The network I / F 303 administers an internal interface with the network 210 and controls the input / output of data from / to another computer. For the network I / F 303, for example, a modem or a LAN adapter can be adopted.

  The recording medium I / F 304 controls reading / writing of data with respect to the recording medium 305 under the control of the CPU 301. The recording medium I / F 304 is, for example, a disk drive, an SSD (Solid State Drive), a USB (Universal Serial Bus) port, or the like. The recording medium 305 is a non-volatile memory that stores data written under the control of the recording medium I / F 304. The recording medium 305 is, for example, a disk, a semiconductor memory, a USB memory, or the like. The recording medium 305 may be detachable from the reinforcement learning device 100.

  The reinforcement learning device 100 may include, for example, a keyboard, a mouse, a display, a printer, a scanner, a microphone, a speaker, and the like, in addition to the above-described components. Further, the reinforcement learning device 100 may include a plurality of recording medium I / Fs 304 and recording media 305. Further, the reinforcement learning device 100 may not include the recording medium I / F 304 or the recording medium 305.

(Example of functional configuration of the reinforcement learning device 100)
Next, a functional configuration example of the reinforcement learning device 100 will be described with reference to FIG.

  FIG. 4 is a block diagram showing a functional configuration example of the reinforcement learning device 100. The reinforcement learning device 100 includes a storage unit 400, an acquisition unit 401, a switching unit 402, a learning unit 403, and an output unit 404.

  The storage unit 400 is realized by a storage area such as the memory 302 and the recording medium 305 illustrated in FIG. 3, for example. The case where the storage unit 400 is included in the reinforcement learning device 100 will be described below, but the storage unit 400 is not limited to this. For example, the storage unit 400 may be included in a device different from the reinforcement learning device 100, and the storage content of the storage unit 400 may be referred to by the reinforcement learning device 100.

  The acquisition unit 401 to the output unit 404 function as an example of a control unit. Specifically, the acquisition unit 401 to the output unit 404, for example, by causing the CPU 301 to execute a program stored in a storage area such as the memory 302 or the recording medium 305 illustrated in FIG. 3, or the network I / F 303. To realize that function. The processing result of each functional unit is stored in a storage area such as the memory 302 or the recording medium 305 illustrated in FIG. 3, for example.

  The storage unit 400 stores various information that is referred to or updated in the processing of each functional unit. The storage unit 400 stores a state value regarding the power generation system 200. The state value regarding the power generation system 200 includes, for example, a state value regarding each of the one or more generators 201 included in the power generation system 200 and a state value regarding the entire power generation system 200. The generator 201 is, for example, a wind power generator or a thermal power generator. The state values for the generator 201 are, for example, the output watt value and wind speed for the stall-controlled wind power generator, and the output watt value and wind speed for the pitch-controlled wind power generator. The state value regarding the generator 201 is, for example, the output wattage value and the fuel consumption amount regarding the thermal power generator. Further, the state value related to the entire power generation system 200 is, for example, demand power in the entire power generation system 200.

  The storage unit 400 stores a command value for each of the one or more generators 201 included in the power generation system 200. The command value for the generator 201 is a command value for changing the utilization efficiency of natural energy in the generator 201. The command value is, for example, a command value for switching the power supply of the generator 201 between ON and OFF. The command value is, for example, a command value that changes the wind receiving performance of the wind power generator. The command value is, for example, a command value indicating how much the pitch angle of the pitch-controlled wind power generator is changed. Specifically, the command values indicating how much the pitch angle is changed are -ΔΘ, ± 0, and + ΔΘ. The command value is, for example, a command value indicating how much to change the size of the fuel supply hole provided in the generator of the thermal power generator. The command value is, for example, a command value indicating how much the fuel usage amount of the thermal power generator should be changed.

  The storage unit 400 stores validity information indicating the effectiveness of each combination of command values for the one or more generators 201 in each of a plurality of regions in which a combination of state values for the one or more generators 201 can be taken. To do. The combination of state values may be one state value. The storage unit 400 also stores validity information indicating the validity of each combination of command values for one or more generators 201 in a region obtained by combining two or more regions out of a plurality of regions. The storage unit 400 stores, for example, a table including, as a record, validity information in which a valid value indicating validity for each combination of command values is associated with each of the plurality of regions. The effective value indicates, for example, the degree of contribution to the increase of the reward in the generator. The reward is, for example, the amount of power generation. Further, the storage unit 400, for example, among the validity information about each area of the plurality of areas, the validity information about each area of two or more areas, the validity information about the area combining two or more areas. The table replaced with the sex information is stored.

  The storage unit 400 stores the characteristic function of the generator 201. The characteristic function indicates the change of the state value regarding the generator 201. The characteristic function indicates, for example, the relationship between the wind speed and the output power from the wind power generator. The characteristic function indicates, for example, the relationship between the fuel usage amount of the thermal power generator and the output power from the thermal power generator. The storage unit 400 stores, for example, an approximate curve that approximates the characteristic function. The memory | storage part 400 memorize | stores the characteristic function which changes for every wind receiving performance of a wind power generator, for example. The memory | storage part 400 memorize | stores the characteristic function which changes specifically for every pitch angle of a wind power generator.

  The storage unit 400 stores, for example, processing procedures by the reinforcement learning algorithm and the action selection algorithm. The reinforcement learning algorithm is, for example, a Q learning algorithm. The reinforcement learning algorithm may be other than the Q learning algorithm. The action selection algorithm is, for example, the ε-greedy algorithm.

  The acquisition unit 401 acquires various kinds of information used for the processing of each functional unit from the storage unit 400 and outputs it to each functional unit. The acquisition unit 401 may acquire various information used for the processing of each functional unit from a device different from the reinforcement learning device 100 and output the acquired information to each functional unit. The acquisition unit 401 acquires, for example, a state value regarding the power generation system 200. The acquisition unit 401 acquires a state value regarding the generator 201 from a measuring device provided in the generator 201. The acquisition unit 401 specifically acquires the demand power in the power generation system 200 from the computer of the electric company.

  The acquisition unit 401 may acquire information indicating the characteristic function, for example. The acquisition unit 401 may acquire, for example, a threshold related to the characteristic function and generate information representing the characteristic function. The threshold for the characteristic function is at least the rated wind speed and the maximum output. The threshold for the characteristic function may be a cut-in wind speed and a cut-out wind speed. The acquisition unit 401 may acquire output power from the generator 201 at various wind speeds and generate information representing a characteristic function.

  The switching unit 402 switches validity information used for learning in reinforcement learning. The switching unit 402 sets, for example, the validity information about each of the plurality of regions as the validity information used for learning. The switching unit 402, for example, among the validity information about each of the plurality of regions, the validity information about each of the two or more regions, the validity information about the region obtained by combining the two or more regions. Replaced with and set to the validity information used for learning.

  Specifically, the validity information for each of the two or more regions may be set as the validity information used for learning. In this case, the switching unit 402 refers to the acquired state value regarding the power generator 201, and generates validity information about the area in which two or more areas are combined based on the characteristic function. Further, the switching unit 402 may generate the validity information about the area obtained by combining the two or more areas based on the validity information about each of the two or more areas. Then, the switching unit 402 replaces the validity information for each of the two or more regions in the validity information used for learning with the generated validity information.

  Specifically, the validity information about the area in which two or more areas are combined may be set as the validity information used for learning. In this case, the switching unit 402 refers to the acquired state value regarding the generator 201, and generates validity information for each of the two or more regions based on the characteristic function. Further, the switching unit 402 may generate the validity information about each of the two or more areas based on the validity information about the area in which the two or more areas are combined. Then, the switching unit 402 replaces the validity information about the area in which two or more areas are combined, with the generated validity information, out of the validity information used for learning.

  Specifically, the switching unit 402 specifies the output power corresponding to the acquired wind speed based on the characteristic function, and generates the validity information about the combined region based on the specified output power. In addition, the switching unit 402 is specifically effective for the combined region based on the characteristic function corresponding to the acquired wind speed and output power among the plurality of characteristic functions that differ for each wind receiving performance of the generator 201. Generate sex information.

  Specifically, when the acquired demand power is equal to or less than the threshold, the switching unit 402 generates validity information about the combined area. More specifically, the switching unit 402, in the case where the acquired demand power is less than or equal to a threshold value, regarding a region that is a combination of two or more regions for relatively large output power among a plurality of regions for output power. Generate validity information for.

  Specifically, when the acquired demand power exceeds the threshold value, the switching unit 402 generates validity information about the combined areas. More specifically, when the acquired demand power exceeds the threshold value, the switching unit 402, regarding a region obtained by combining two or more regions for relatively small output power, among a plurality of regions for output power, Generate validity information.

  The learning unit 403 performs learning using the validity information set by the switching unit 402, and updates at least one of the validity information. The learning unit 403 performs learning using, for example, the validity information about each of the plurality of areas. The learning unit 403, for example, among the validity information about each area of the plurality of areas, the validity information about each area of two or more areas, the validity information about the area combining two or more areas. Replace with to learn.

  The output unit 404 outputs the processing result of each functional unit. The output format is, for example, display on a display, print output to a printer, transmission to an external device by the network I / F 303, or storage in a storage area such as the memory 302 or the recording medium 305. Thereby, the output unit 404 can notify the user of the processing result of each functional unit, and can support the management and operation of the reinforcement learning apparatus 100, for example, the update of the set value of the reinforcement learning apparatus 100, The convenience of the reinforcement learning device 100 can be improved.

(Operation example of the reinforcement learning device 100 for the power generation system 200 including the wind power generator)
Next, an operation example of the reinforcement learning device 100 for the power generation system 200 including the wind power generator will be described with reference to FIGS. 5 to 13. First, shifting to the description of FIG. 5, a specific configuration example of the power generation system 200 including a wind power generator will be described.

  FIG. 5: is explanatory drawing which shows the specific structural example of the power generation system 200 containing a wind power generator. In the example of FIG. 5, the power generation system 200 includes a reinforcement learning device 100, a stall control wind power generator i (i = 1, ..., N), and a pitch control wind power generator i (i = 1 ,. .., m) are included. The stall-controlled wind power generator i receives the command value ai (i = 1, ..., N) from the reinforcement learning device 100. The pitch-controlled wind power generator i receives the command value bi (i = 1, ..., M) from the reinforcement learning device 100.

The power generation system 200 includes an anemometer si (i = 1, ..., N) for a stall-controlled wind power generator i and an anemometer pi (i = 1, ..., N) for a pitch-controlled wind power generator i. , M) and. The anemometer si transmits the wind speed value F _si (t _j ) to the reinforcement learning device 100. t _j is the time. The anemometer pi transmits the wind speed value F _pi (t _j ) to the reinforcement learning device 100. The power generation system 200 includes a power meter for the stall-controlled wind power generator i and a power meter for the pitch-controlled wind power generator i. The power meter for the stall-controlled wind power generator i sends the output wattage value P _si (t _j ) to the reinforcement learning device 100. The power meter for the pitch-controlled wind power generator i sends the output wattage value P _pi (t _j ) to the reinforcement learning device 100.

The reinforcement learning device 100 includes a table generation unit 501, a section switching unit 502, a value setting unit 503, an action determination unit 504, a state calculation unit 505, a reward calculation unit 506, and a table update unit 507. The reinforcement learning device 100 outputs the output watt value P _si (t _j ) of the stall-controlled wind power generator i and the pitch control of the pitch control in a range that does not exceed the demand power watt value P ′ (t _j ) of the entire power generation system 200. Increase the sum with the output wattage value P _pi (t _j ) for the wind power generator i.

  The table generation unit 501 creates a normal table that stores validity information about a plurality of normal sections divided by the normal division method described later in FIGS. 6 and 7. The table generation unit 501 sets which two or more normal intervals are combined to convert the plurality of normal intervals into a plurality of coarse division intervals divided by a coarse division method described later in FIGS. 6 and 7. To do.

The section switching unit 502 uses the wind speed value F _si (t _j ), the wind speed value F _pi (t _j ), the output watt value P _si (t _j ), the output watt value P _pi (t _j ), and the power demand. Receive the wattage value P ′ (t _j ). The section switching unit 502 switches the table to be used between a normal table and a coarse division table that stores validity information for a plurality of coarse division sections, based on the received various information. If the threshold value α> the demand power watt value P ′ (t _j ), the section switching unit 502 sets the rough division table to the table to be used. The section switching unit 502 sets the normal table to the table to be used if the threshold value α ≦ the demand power watt value P ′ (t _j ).

  The value setting unit 503 sets a valid value in the table based on the switching result. When two or more areas are combined, the value setting unit 503 sets a valid value in the record corresponding to the combined area. When two or more areas are separated, the value setting unit 503 sets a valid value in the record corresponding to each separated area. The value setting unit 503 outputs a table in which valid values are set.

  The action determination unit 504 uses the table to select the command value ai for the stall-controlled wind power generator i, and transmits the command value ai to the stall-controlled wind power generator i to obtain the command value bi for the pitch-controlled wind power generator i. Select and send to pitch controlled wind power generator i. The action determination unit 504 selects, for example, a combination of the command value ai and the command value bi associated with the largest effective value in the table. Specifically, the action determination unit 504 randomly selects a command value with a probability of ε by using the ε-greedy algorithm, and has a probability of 1-ε to obtain the largest effective value in the current state of the power generation system 200. A combination of the associated command value ai and command value bi is selected.

The state calculation unit 505 determines the wind speed value F _si (t _j ), the wind speed value F _pi (t _j ), the output watt value P _si (t _j ), the output watt value P _pi (t _j ), and the power demand. The state of the power generation system 200 is specified based on the watt value P ′ (t _j ). The state calculation unit 505 outputs a state result indicating a record in the table corresponding to the specified state. The reward calculation unit 506 calculates a reward value based on the output watt value P _si (t _j ), the output watt value P _pi (t _j ), and the demand power watt value P ′ (t _j ). The table updating unit 507 updates the effective value in the record indicated by the state result based on the calculated reward value.

  Next, shifting to the description of FIGS. 6 to 13, the table generation unit 501, the section switching unit 502, the value setting unit 503, the action determination unit 504, the state calculation unit 505, the reward calculation unit 506, The operation of each part with the table updating unit 507 will be specifically described. First, shifting to the description of FIG. 6 and FIG. 7, the rough division section set in the table generation unit 501 will be specifically described.

  FIG. 6 is an explanatory diagram showing a normal section and a roughly divided section that can be taken by the state value of the stall-controlled wind power generator. For example, as shown by a graph 601 in FIG. 6, the entire section regarding the output power and the wind speed is divided into a plurality of normal sections by the normal division method of uniformly dividing. For example, the entire section of output power is divided into normal sections 1, 2, ..., Npsi. The whole section of wind speed is divided into normal sections 1, 2, ..., Nfsi. Then, for example, as shown in the graph 602 of FIG. 6, the two regions having the larger output power are set as targets to be combined, and the entire section regarding the output power and the wind speed is divided into a plurality of coarse division sections. . For example, the entire section of output power is divided into coarsely divided sections 1, 2, ..., Npsi-1. The whole section of wind speed is divided into coarsely divided sections 1, 2, ..., Nfsi-1. Next, the description moves to FIG. 7.

  FIG. 7: is explanatory drawing which shows the normal area and coarse division area which the state value of the pitch control wind power generator can take. For example, as shown in a graph 701 in FIG. 7, the entire section regarding the output power and the wind speed is divided into a plurality of normal sections by the normal division method for uniform division. For example, the entire output power section is divided into normal sections 1, 2, ..., Nppi. The entire wind speed section is divided into normal sections 1, 2, ..., Nfpi. Then, for example, as shown in the graph 702 of FIG. 7, the two regions having the larger output power are set as the target to be combined, and the entire section regarding the output power and the wind speed is divided into a plurality of coarse division sections. . For example, the entire section of output power is divided into coarsely divided sections 1, 2, ..., Nppi-1. The entire wind speed section is divided into coarsely divided sections 1, 2, ..., Nfpi-1.

  Next, shifting to the description of FIG. 8, a specific example of realizing the normal table 801 for the normal sections created by the table generation unit 501 and the rough division table 802 for the coarse division sections switched from the normal table 801 will be described. To explain.

  FIG. 8 is an explanatory diagram showing an example of realizing the normal table 801 and the coarse division table 802. In FIG. 8, the reinforcement learning device 100 uses the record of the normal table 801 as the record of the coarse division table 802 to realize the mutual conversion of the normal table 801 and the coarse division table 802.

  The reinforcement learning device 100 creates the normal table 801, for example. When converting the normal table 801 into the coarse division table 802, the reinforcement learning apparatus 100 diverts a record corresponding to one ordinary section of two or more ordinary sections to be combined as a record corresponding to the combined coarse division section. Then, the reinforcement learning device 100 deletes the validity information set in the record of the other normal section. For example, when the normal section npsi-1 and the normal section npsi are combined, the reinforcement learning apparatus 100 diverts the record corresponding to the normal section npsi-1 as the record corresponding to the coarse division section npsi-1. The reinforcement learning device 100 deletes the validity information set in the record corresponding to the section npsi, for example.

  Further, when converting the rough division table 802 into the normal table 801, the reinforcement learning apparatus 100 regards the record corresponding to the combined section as a record corresponding to one of the two or more sections divided from the combined section. Divert. Then, the reinforcement learning device 100 sets the validity information again in the record of the other section. The reinforcement learning device 100, for example, when dividing the rough division section npsi-1 into the normal division npsi-1 and the normal division npsi, corresponds the record corresponding to the rough division division npsi-1 to the normal division npsi-1. It will be used as a record. The reinforcement learning device 100 sets the validity information again in the record corresponding to the normal section npsi, for example.

  Next, with reference to FIG. 9, an example of the stored contents of the normal table 801 created by the table creation unit 501 will be described. The normal table 801 is realized by a storage area such as the memory 302 or the recording medium 305 of the reinforcement learning device 100 shown in FIG. The following description of the normal table 801 corresponds to the case where Q learning is used as the reinforcement learning method, and the stored contents may be different when the different reinforcement learning method is used.

  FIG. 9 is an explanatory diagram showing an example of the stored contents of the normal table 801. As shown in FIG. 9, the normal table 801 has fields for status values, command values, and valid values. The normal table 801 stores validity information as a record by setting information in each field.

  In the state value field, a section where the state value regarding the power generation system 200 can be set is set. The state value regarding the power generation system 200 includes a state value regarding the wind power generator and a state value regarding the entire power generation system 200. In the example of FIG. 9, the state values for the wind power generator are the output watt value and the wind speed value for the stall control wind power generator and the output watt value and the wind speed value for the pitch control wind power generator. Further, the state value regarding the entire power generation system 200 is a demand power watt value in the entire power generation system 200.

  A command value for the wind power generator is set in the command value field. In the example of FIG. 9, the command value for the wind power generator is a command value for switching the power source of the stall-controlled wind power generator between ON and OFF. The command value for the wind power generator is a command value indicating how much the pitch angle of the pitch-controlled wind power generator is changed. Specifically, the command values indicating how much the pitch angle is changed are -ΔΘ, ± 0, and + ΔΘ. In the effective value field, an effective value indicating the effectiveness of the combination of command values for each wind power generator when each state value is included in any section is set.

  Next, shifting to the description of FIG. 10, the wind power generation that is used when the section switching unit 502 converts the normal table 801 into the coarse partition table 802 or when converting the coarse partition table 802 into the normal table 801. An example of creating a machine-related characteristic function will be described. The characteristic function regarding the wind power generator may be input in advance to the reinforcement learning device 100.

  FIG. 10 is an explanatory diagram showing an example of creating the characteristic function. In FIG. 10, the reinforcement learning device 100 creates a characteristic function regarding a wind power generator. The reinforcement learning device 100 acquires output wattage values from a stall-controlled wind power generator at various wind speeds, for example. Further, the reinforcement learning device 100 acquires the rated wind speed, the maximum output, the cut-in wind speed, and the cut-out wind speed.

Next, the reinforcement learning device 100 determines the characteristic curve indicated by the characteristic function for the stall-controlled wind power generator based on the rated wind speed, the maximum output, the cut-in wind speed, the cut-out wind speed, and the output wattage at various wind speeds. An approximate curve f _i (t) to be approximated is obtained. The reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y = 0 from the wind speed of 0 to the cut-in wind speed.

The reinforcement learning apparatus 100 obtains a part of the approximated curve f _i (t) in the shape of y = a * x ^ 3 based on the output wattage values at various wind speeds from the cut-in wind speed to the rated wind speed, for example. . For example, after the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y = b * x ^ 2 based on the output wattage values at various wind speeds. Thereby, the reinforcement learning device 100 obtains an approximated curve f _i (t) as shown in the graph 1000 of FIG. 10.

  Further, the reinforcement learning device 100 acquires the output wattage value from the pitch-controlled wind power generator at various wind speeds, for example, at pitch angles Θ = 0, ΔΘ, 2ΔΘ, ..., kΔΘ. Further, the reinforcement learning device 100 acquires the rated wind speed, the maximum output, the cut-in wind speed, and the cut-out wind speed.

Next, the reinforcement learning device 100 determines the characteristic curve indicated by the characteristic function of the pitch-controlled wind power generator based on the rated wind speed, the maximum output, the cut-in wind speed, the cut-out wind speed, and the output wattage at various wind speeds. An approximate curve f _i (t) to be approximated is obtained. The reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y = 0 from the wind speed of 0 to the cut-in wind speed.

The reinforcement learning apparatus 100 obtains a part of the approximated curve f _i (t) in the shape of y = a * x ^ 3 based on the output wattage values at various wind speeds from the cut-in wind speed to the rated wind speed, for example. . For example, after the rated wind speed, the reinforcement learning device 100 obtains a part of the approximated curve f _i (t) in the shape of y = b * x ^ 2 based on the output wattage values at various wind speeds. Thereby, the reinforcement learning device 100 obtains an approximated curve f _i (t) as shown in the graph 1000 of FIG. 10 at the pitch angles Θ = 0, ΔΘ, 2ΔΘ, ..., KΔΘ.

Next, shifting to the description of FIG. 11, the section switching unit 502 converts the normal table 801 into a coarse division table 802 based on an approximated curve f _i (t) of a specific function created in advance, and uses the table. An example will be described in which is switched to the coarse division table 802.

FIG. 11 is an explanatory diagram showing an example of switching the table to be used to the coarse division table 802. In FIG. 11, the reinforcement learning apparatus 100 sets the rough division table 802 to a table that uses the threshold value α> the power demand watt value P ′ (t _j ). In the example of FIG. 11, a case will be described in which a valid value is set in the field indicated by ● 1101 corresponding to the rough division section npsi-1 in which the normal section nps1-1 and the normal section nps1 are combined.

The reinforcement learning device 100 specifies, for example, a record corresponding to the rough division section nps1-1. The reinforcement learning device 100 acquires the wind speed values F _s1 , ..., F _sn for the stall-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the output watt value P _p1 , ..., P _pm from the pitch-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the wind speed values F _p1 , ..., F _pm regarding the pitch-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the demand power watt value P ′ for the entire power generation system 200 set in the identified record.

The reinforcement learning device 100 predicts the output power P ′ _si = f when the power is turned on for the stall-controlled wind power generator i based on the observed wind speed F _si and the approximate curve f _i (t). _i (F _si ) is calculated. Further, the reinforcement learning device 100 determines the predicted output power P ′ _si = 0 when the power is turned off for the stall-controlled wind power generator i.

Further, the reinforcement learning device 100, based on the observed wind speed F _pi , the output watt value P _pi, and the approximated curve f _i, Θ (t) for the pitch-controlled wind power generator i, f _i, Θ ( Determine the pitch angle Θ such that F _pi ) ≈P _pi . Next, reinforcement learning apparatus 100, determined -ΔΘ the pitch angle theta, ± 0, + predicted output power in the case where the _{ΔΘ P 'pi = f i,} Θ - ΔΘ (F pi), f i, Θ (F _pi ), fi _, Θ ₊ ΔΘ (F _pi ) is calculated.

The reinforcement learning device 100 predicts the predicted output power P ′ _si = f _i (F _si ) and the predicted output power P ′ _pi = f _i, Θ ₋ ΔΘ (F _pi ), f _i, Θ (F _pi ), f _{i. ,} Θ ₊ ΔΘ (F _pi ) and the predicted output power table 1100 is created. The reinforcement learning device 100 has command values a1, ..., An for the stall control wind power generator and command values b1 ,. To get. Then, the reinforcement learning device 100 acquires the predicted output power P ′ _si corresponding to the acquired command value and the predicted output power P ′ _pi from the created predicted output power table 1100, and uses the following formula (1). Then, the predicted output power P ~ in the entire power generation system 200 is calculated.

  Then, the reinforcement learning device 100 calculates an effective value Q = r (P ″) based on the difference value P ″ between the acquired demand power watt value P ′ and the calculated predicted output power P˜, The fields 1101 are set. r (P ″) is defined by the following equations (2) to (5). Specifically, when P ″ is defined by the following equation (2) and δ> 0 is set, P ″ is set. In the case of> δ, r (P ″) is defined by the following formula (3), and in the case of −δ ≦ P ″ ≦ δ, r (P ″) is defined by the following formula (4), and P ″ < In the case of −δ, r (P ″) is defined by the following equation (5).

  The reinforcement learning device 100 similarly sets valid values in other fields. Thereby, the reinforcement learning device 100 can calculate the effective value such that the effective value increases as the total output watt value approaches the demand power watt value. Therefore, the reinforcement learning device 100 can easily control the power generation system 200 appropriately by referring to the effective value. Further, the reinforcement learning device 100 can reduce the number of pieces of validity information to be learned and updated, and reduce the amount of processing required for reinforcement learning.

Next, shifting to the description of FIG. 12, the section switching unit 502 converts the rough division table 802 into a normal table 801 based on an approximated curve f _i (t) of a specific function created in advance, and a table to be used. An example of switching the table to the normal table 801 will be described.

FIG. 12 is an explanatory diagram showing an example of switching the table to be used to the normal table 801. In FIG. 12, the reinforcement learning apparatus 100 sets the normal table 801 to a table that uses the threshold value α ≦ the demand power watt value P ′ (t _j ). In the example of FIG. 12, a case will be described in which a valid value is set in the field indicated by ● 1201 corresponding to the normal section nps1-1 divided from the rough division section npsi-1.

The reinforcement learning device 100 identifies, for example, a record corresponding to the rough division section npsi-1. Next, the reinforcement learning device 100 acquires the wind speed values F _s1 , ..., F _sn for the stall-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the output watt value P _p1 , ..., P _pm from the pitch-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the wind speed values F _p1 , ..., F _pm regarding the pitch-controlled wind power generator i set in the identified record. Further, the reinforcement learning device 100 acquires the demand power watt value P ′ for the entire power generation system 200 set in the identified record.

The reinforcement learning device 100 predicts the output power P ′ _si = f when the power is turned on for the stall-controlled wind power generator i based on the observed wind speed F _si and the approximate curve f _i (t). _i (F _si ) is calculated. Further, the reinforcement learning device 100 determines the predicted output power P ′ _si = 0 when the power is turned off for the stall-controlled wind power generator i.

Further, the reinforcement learning device 100, based on the observed wind speed F _pi , the output watt value P _pi, and the approximated curve f _i, Θ (t) for the pitch-controlled wind power generator i, f _i, Θ ( Determine the pitch angle Θ such that F _pi ) ≈P _pi . Next, reinforcement learning apparatus 100, determined -ΔΘ the pitch angle theta, ± 0, + predicted output power in the case where the _{ΔΘ P 'pi = f i,} Θ - ΔΘ (F pi), f i, Θ (F _pi ), fi _, Θ ₊ ΔΘ (F _pi ) is calculated.

The reinforcement learning device 100 predicts the predicted output power P ′ _si = f _i (F _si ) and the predicted output power P ′ _pi = f _i, Θ ₋ ΔΘ (F _pi ), f _i, Θ (F _pi ), f _{i. ,} Θ ₊ ΔΘ (F _pi ), a predicted output power table 1200 is created. The reinforcement learning device 100 has command values a1, ..., An for the stall control wind power generator and command values b1, ..., Bm for the pitch control wind power generator corresponding to the field indicated by 1201. To get. Then, the reinforcement learning device 100 acquires the predicted output power P ′ _si and the predicted output power P ′ _pi corresponding to the acquired command value from the created predicted output power table 1200, and uses the above formula (1). Then, the predicted output power P ~ in the entire power generation system 200 is calculated.

  Then, the reinforcement learning device 100 calculates an effective value Q = r (P ″) based on the difference value P ″ between the acquired demand power watt value P ′ and the calculated predicted output power P˜, The field 1201 is set. Specifically, r (P ″) is defined by the above formulas (2) to (5).

  The reinforcement learning device 100 similarly sets valid values in other fields corresponding to the normal section nps1-1 and fields corresponding to the normal section npsi. Thereby, the reinforcement learning device 100 can calculate the effective value such that the effective value increases as the total output watt value approaches the demand power watt value. Therefore, the reinforcement learning device 100 can easily control the power generation system 200 appropriately by referring to the effective value. Further, the reinforcement learning device 100 increases the number of validity information to be learned and updated, and subdivides what kind of command value should be output in what kind of state value and executes it. can do.

In the above, the reinforcement learning device 100 calculates the effective value based on the approximated curve f _i (t) and the approximated curve f _i, Θ (t). Here, the current wind speed is F _si and F _pi , and the wind speed observed at the next time is F _si + ΔF _si and F _pi + ΔF _pi . In this case, the output power observed at the next time is f _i (F _si + ΔF _si ), f _i, Θ ₋ ΔΘ (F _pi + ΔF _pi ), f _i, Θ (F _pi + ΔF _pi ), f _i, Θ ₊ ΔΘ (F _pi + ΔF _pi ). Further, since the approximated curve is a continuous function, the output power observed at the next time is f _i (F _si ) + ΔP _si , f _i, Θ ₋ ΔΘ (F _pi ) + ΔP _pi , f _i, Θ (F _pi ) + ΔP _pi , f _i, Θ ₊ ΔΘ (F _pi ) + ΔP _pi .

If the demand power at the next time is P ′ + ΔP ′, the difference P ″ _n between the demand power and the output power at the next time and the difference P ″ between the demand power and the output power currently obtained are The following formula (6) is established. Since the reward function is a continuous function, the following expression (7) is established.

Here, if ΔF _si → 0, ΔF _pi → 0, ΔP → 0, then P ″ _n → P ″ and r (P ″ _n ) → r (P ″). Therefore, when the wind speed is stable and the change in the demand power is small, P ″ _n ≈P ″ and r (P ″ _n ) ≈r (P ″) are established. Therefore, if the reinforcement learning apparatus 100 selects the combination of the command values having the largest effective value r (P ″) by the ε-greedy algorithm, the total output power is brought close to the demand power by the following formula (8). It is judged that it is possible.

  As described above, the reinforcement learning device 100 calculates the effective value based on the characteristic function so that the wind power generator to be controlled has an output power according to the actual demand power. it can. Therefore, the reinforcement learning device 100 can facilitate selection of an appropriate command value as compared with the case where the effective value is initialized to 0 or the effective value is randomly set.

After that, the reinforcement learning device 100 selects and outputs the combination of the command value ai and the command value bi as the action using the ε-greedy algorithm at regular intervals. The reinforcement learning device 100, for example, at time t _j , the wind speed value F _si (t _j ), the wind speed value F _pi (t _j ), the output watt value P _si (t _j ), and the output watt value P _pi ( t _j ) and the demand power watt value P ′ (t _j ) are acquired as state values.

The reinforcement learning device 100 specifies the section F _si to (t _j ) to which the wind speed value F _si (t _j ) belongs and the section F _pi to (t _j ) to which the wind speed value F _pi (t _j ) belongs. Reinforcement learning apparatus 100 includes an output wattage value P _si interval P _si ~ Field of _{(t j) (t j)} , identifies belongs interval P _pi ~ a (t _j) of the output wattage value P _pi (t _j) . Reinforcement learning apparatus 100 specifies the section belongs demand power watt value P '(t _j) P'~ a (t _j).

  Then, the reinforcement learning apparatus 100 randomly selects and outputs a combination of the command value ai and the command value bi with the probability of ε. The reinforcement learning device 100 identifies, with a probability of 1-ε, a record corresponding to the combination of the sections to which the acquired state value belongs in the normal table 801 or the rough division table 802 set as the table to be used. The reinforcement learning apparatus 100 selects and outputs the combination of the command value ai and the command value bi associated with the largest effective value in the identified record.

Further, the reinforcement learning device 100 calculates the reward value at time t _j for the combination of the command value ai and the command value bi output at time t _j-1 . The reinforcement learning device 100 calculates the reward value r (t _j ) by the following equations (9) to (12), for example. Specifically, P ″ (t _j ) is defined by the following equation (9). When P ″ (t _j )> δ, r (t _j ) is defined by the following equation (10), When −δ ≦ P ″ (t _j ) ≦ δ, r (t _j ) is defined by the following formula (11), and when P ″ (t _j ) <−δ, r (t _j ) is defined by the following formula (12). Define (t _j ).

Next, a case where the information processing device updates the effective value based on the reward value calculated in response to the output of the combination of the command value ai and the command value bi will be described with reference to FIG. 13. In the example of FIG. 13, the information processing apparatus updates the effective value, for example, at time t _{j + 1} based on the reward value calculated for the combination of the command value ai and the command value bi output at time t _j . The case will be described.

FIG. 13 is an explanatory diagram showing an example of updating valid values. In FIG. 13, the reinforcement learning device 100 updates the valid value Q set in the field indicated by ● 1301 associated with the record corresponding to the combination of the sections to which the state value acquired at time t _j belongs. The reinforcement learning device 100 calculates the effective value Q ′ based on the reward value calculated at the time t _{j + 1} by using, for example, the following formulas (13) and (14), and a field indicated by ● 1301. The effective value Q set in is updated.

In the reinforcement learning device 100, if the record corresponding to the combination of the sections to which the state value acquired at time t _{j + 1} belongs is a record corresponding to the rough division section or a record corresponding to the normal section that can be combined, The effective value Q ′ is calculated using the following equation (13). Further, in the reinforcement learning device 100, the record corresponding to the combination of the sections to which the state value acquired at time t _{j + 1} belongs must be a record corresponding to the rough division section or a record corresponding to the normal section that can be combined. For example, the effective value Q ′ is calculated using the following formula (14).

In the example of FIG. 13, in the reinforcement learning device 100, the record corresponding to the combination of the sections to which the state value acquired at the time t _{j + 1} belongs corresponds to the record corresponding to the coarse division section or the normal section that can be combined. It is determined that it is not a record to be executed. The reinforcement learning device 100 calculates the effective value Q ′ by using the above equation (14) based on the state value of the field 1311, the value obtained by substituting the effective value of the field 1312 into the max function, and the effective value Q.

On the other hand, in the reinforcement learning device 100, the record corresponding to the combination of the sections to which the state value acquired at the time t _{j + 1} belongs is the record corresponding to the rough division section or the record corresponding to the normal section that can be combined. It may be determined that there is. In this case, the reinforcement learning device 100 calculates the effective value Q ′ using the above equation (13) based on the state value of the field 1310, the state value of the field 1320, and the effective value Q. As a result, the reinforcement learning device 100 can update the validity information and facilitate appropriate control of the power generation system 200.

(Operation example of the reinforcement learning device 100 for the power generation system 200 including a thermal power generator)
Next, an operation example of the reinforcement learning device 100 for the power generation system 200 including a thermal power generator will be described with reference to FIGS. 14 and 15. First, shifting to the description of FIG. 14, a specific configuration example of the power generation system 200 including a thermal power generator will be described.

  FIG. 14: is explanatory drawing which shows the specific structural example of the power generation system 200 containing a thermal power generator. In the example of FIG. 14, the power generation system 200 includes the reinforcement learning device 100 and a fuel-controlled thermal power generator i (i = 1, ..., M). The fuel-controlled thermal power generator i receives the command value bi (i = 1, ..., M) from the reinforcement learning device 100.

The power generation system 200 includes a fuel meter pi (i = 1, ..., M) for the fuel-controlled thermal power generator i. The fuel gauge pi transmits the fuel usage amount F _pi (t _j ) to the reinforcement learning device 100. The power generation system 200 includes a power meter for the fuel-controlled thermal power generator i. The power meter for the fuel-controlled thermal power generator i sends the output wattage value P _pi to the reinforcement learning device 100.

  The reinforcement learning device 100 includes a table generation unit 501, a section switching unit 502, a value setting unit 503, an action determination unit 504, a state calculation unit 505, a reward calculation unit 506, and a table update unit 507. Here, the operation of each part of the reinforcement learning apparatus 100 for the power generation system 200 including a thermal power generator is the same as the operation of each part of the reinforcement learning apparatus 100 for the power generation system 200 including a wind power generator. , Description is omitted. Now, shifting to the description of FIG. 15, an example of the stored contents of the normal table 801 in the power generation system 200 including a thermal power generator will be described.

  FIG. 15 is an explanatory diagram showing an example of the stored contents of the normal table 801 relating to the thermal power generator. As shown in FIG. 15, the normal table 801 has fields of a state value, a command value, and a valid value. The normal table 801 stores validity information as a record by setting information in each field.

  In the state value field, a section where the state value regarding the power generation system 200 can be set is set. The state value regarding the power generation system 200 includes a state value regarding the thermal power generator and a state value regarding the entire power generation system 200. In the example of FIG. 15, the state value regarding the thermal power generator is the output watt value and the fuel usage amount regarding the fuel-controlled thermal power generator. Further, the state value regarding the entire power generation system 200 is the demand power in the entire power generation system 200.

  A command value for the thermal power generator is set in the command value field. In the example of FIG. 15, the command value for the thermal power generator is a command value indicating how much the fuel usage amount of the fuel-controlled thermal power generator should be changed. The command values indicating how much the fuel usage amount is changed are specifically −X, ± 0, and + X. In the effective value field, an effective value indicating the effectiveness of the combination of command values for each thermal power generator when each state value is included in any section is set.

(Overall processing procedure)
Next, an example of the overall processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. The entire process is realized by, for example, the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

  FIG. 16 is a flowchart showing an example of the overall processing procedure. In FIG. 16, the reinforcement learning device 100 creates a table for a plurality of normal intervals and sets two or more normal intervals that can be coarsely divided intervals (step S1601).

  Next, the reinforcement learning device 100 executes a switching determination process described later in FIG. 17 and sets a table to be used (step S1602). Then, the reinforcement learning device 100 executes a value setting process described later in FIG. 18, and sets a valid value in the set table (step S1603).

  Next, the reinforcement learning device 100 identifies the state of the wind power generator based on the output watt value from the wind power generator, the wind speed value, and the demand power watt value (step S1604). Then, the reinforcement learning device 100 calculates the reward from the wind power generator based on the output watt value and the demand power watt value from the wind power generator (step S1605).

  Next, the reinforcement learning apparatus 100 determines and outputs the command value for the wind power generator using the set table (step S1606). Then, the reinforcement learning device 100 updates the effective value stored in the set table based on the calculated reward (step S1607). After that, the reinforcement learning device 100 returns to the process of step S1602.

(Switching determination processing procedure)
Next, an example of a switching determination processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. The switching determination process is realized by, for example, the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

FIG. 17 is a flowchart showing an example of the switching determination processing procedure. In FIG. 17, the reinforcement learning device 100 sets a threshold value α (step S1701). Next, the reinforcement learning device 100 determines whether or not α> P ′ (t _j ) with respect to the demand power watt value P ′ (t _j ) (step S1702).

Here, if α> P ′ (t _j ) (step S1702: Yes), the reinforcement learning apparatus 100 proceeds to the process of step S1707. On the other hand, when α> P ′ (t _j ) is not satisfied (step S1702: No), the reinforcement learning apparatus 100 proceeds to the process of step S1703.

  In step S1703, the reinforcement learning device 100 determines to use the normal table 801 for the normal section (step S1703). Next, the reinforcement learning apparatus 100 determines whether or not the normal table 801 for the normal section has been used until immediately before (step S1704).

  Here, when the normal table 801 for the normal section is used (step S1704: Yes), the reinforcement learning apparatus 100 proceeds to the process of step S1706. On the other hand, when the normal table 801 for the normal section is not used (step S1704: No), the reinforcement learning apparatus 100 moves to the process of step S1705.

  In step S1705, the reinforcement learning apparatus 100 creates the normal table 801 for the normal section and sets it as the table to be used (step S1705). Then, the reinforcement learning device 100 ends the switching determination process.

  In step S1706, the reinforcement learning device 100 sets the table used until immediately before to the table to be used as it is (step S1706). Then, the reinforcement learning device 100 ends the switching determination process.

  In step S1707, the reinforcement learning device 100 determines to use the coarse division table 802 for the coarse division section (step S1707). Next, the reinforcement learning apparatus 100 determines whether or not the coarse division table 802 for the coarse division section has been used until immediately before (step S1708).

  Here, when the coarse division table 802 for the coarse division section is used (step S1708: Yes), the reinforcement learning apparatus 100 proceeds to the process of step S1706. On the other hand, when the coarse division table 802 for the coarse division section is not used (step S1708: No), the reinforcement learning apparatus 100 proceeds to the process of step S1709.

  In step S1709, the reinforcement learning device 100 creates a coarse division table 802 for the coarse division section and sets it as a table to be used (step S1709). Then, the reinforcement learning device 100 ends the switching determination process.

(Value setting procedure)
Next, an example of the value setting processing procedure executed by the reinforcement learning device 100 will be described with reference to FIG. The value setting process is realized by, for example, the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

  FIG. 18 is a flowchart showing an example of the value setting processing procedure. In FIG. 18, the reinforcement learning apparatus 100 identifies a record corresponding to a normal section that divides a rough division section or a rough division section that combines the normal sections from the table to be used (step S1801).

Next, the reinforcement learning device 100 acquires the wind speed values F _s1 , ..., F _sn for the stall-controlled wind power generator i (i = 1, ..., N) set in the specified record ( Step S1802). Then, the reinforcement learning apparatus 100 acquires the output wattage values P _p1 , ..., P _pm from the pitch-controlled wind power generator i (i = 1, ..., M) set in the identified record. (Step S1803).

Next, the reinforcement learning apparatus 100 acquires the wind speed values F _p1 , ..., F _pm regarding the pitch-controlled wind power generator i (i = 1, ..., M) set in the specified record (( Step S1804). Then, the reinforcement learning device 100 acquires the demand power watt value P ′ for the entire power generation system 200 set in the identified record (step S1805).

Next, the reinforcement learning device 100 supplies power to the stall-controlled wind power generator i (i = 1, ..., N) based on the observed wind speed F _si and the approximate curve f _i (t). The predicted output power f _i (F _si ) when the switch is turned on is calculated (step S1806). Then, the reinforcement learning device 100 observes the wind speed F _pi , the output watt value P _pi, and the approximated curve f _i, Θ (t) for the pitch-controlled wind power generator i (i = 1, ..., M). ) And P _i, Θ (F _pi ) ≈P _pi , the pitch angle Θ is determined (step S1807).

Next, the reinforcement learning apparatus 100 predicts the predicted output powers f _i, Θ _- ΔΘ (F _pi ), f _i, Θ (F _pi ) when the determined pitch angle Θ is −ΔΘ, ± 0, + ΔΘ. , F _i, Θ ₊ ΔΘ (F _pi ) is calculated (step S1808). Then, the reinforcement learning device 100 calculates the predicted output power f _i (F _si ) and the predicted output powers f _i, Θ ₋ ΔΘ (F _pi ), f _i, Θ (F _pi ), f _i, Θ ₊ ΔΘ (F _pi ) and a predicted output power table is created (step S1809).

  Next, the reinforcement learning device 100, for each field in the identified record, command values a1, ..., An for the stall control wind power generator and command values b1, ... For the pitch control wind power generator. , Bm are acquired (step S1810). Then, the reinforcement learning device 100 calculates the predicted output power P ~ in the entire power generation system 200 based on the acquired command value and the created predicted output power table for each field in the identified record (step S1811). .

  Next, the reinforcement learning device 100 calculates and sets an effective value for each field in the identified record, based on the difference value between the demand power watt value P ′ and the calculated predicted output power P˜ (step). S1812). Then, the reinforcement learning device 100 ends the value setting process.

(Characteristic function creation processing procedure)
Next, an example of a characteristic function creation processing procedure executed by the reinforcement learning device 100 will be described with reference to FIGS. 19 and 20. The characteristic function creating process is realized by, for example, the CPU 301 shown in FIG. 3, a storage area such as the memory 302 and the recording medium 305, and the network I / F 303.

  FIG. 19 is a flowchart showing an example of a characteristic function creation processing procedure for a stall-controlled wind power generator. In FIG. 19, the reinforcement learning device 100 observes the output watt value from the stall-controlled wind power generator at various wind speeds (step S1901).

Next, the reinforcement learning device 100 obtains an approximate curve f _i (t) that approximates the characteristic curve indicated by the characteristic function of the stall-controlled wind power generator, based on the output wattage values at various wind speeds (step S1902). . Then, the reinforcement learning device 100 ends the characteristic function creating process for the stall-controlled wind power generator.

  FIG. 20 is a flowchart showing an example of a characteristic function creation processing procedure for a pitch-controlled wind power generator. In FIG. 20, the reinforcement learning device 100 acquires the maximum pitch angle MΘ in the pitch-controlled wind power generator (step S2001).

  Next, the reinforcement learning device 100 sets the pitch angle Θ = 0 for the pitch-controlled wind power generator (step S2002). Then, the reinforcement learning device 100 determines whether or not Θ <MΘ (step S2003).

  Here, if Θ <MΘ (step S2003: Yes), the reinforcement learning apparatus 100 proceeds to the process of step S2004. On the other hand, when Θ <MΘ is not satisfied (step S2003: No), the reinforcement learning apparatus 100 ends the characteristic function creation processing for the pitch-controlled wind power generator.

In step S2004, the reinforcement learning device 100 observes the output wattage value from the pitch-controlled wind power generator at various wind speeds (step S2004). Next, the reinforcement learning apparatus 100 obtains an approximate curve f _i, Θ (t) that approximates the characteristic curve indicated by the characteristic function of the pitch-controlled wind power generator, based on the output wattage values at various wind speeds (step). S2005).

  Then, the reinforcement learning device 100 sets the pitch angle Θ = Θ + ΔΘ (step S2006). After that, the reinforcement learning device 100 returns to the process of step S2003.

  Here, the reinforcement learning apparatus 100 may perform the processing by changing the order of some of the steps in the above-described various flowcharts. Further, the reinforcement learning device 100 may omit the processing of some steps in the various flowcharts described above.

  As described above, according to the reinforcement learning device 100, the validity information indicating the validity of each command value for the generator 201 in each of the plurality of regions in which the state value regarding the generator 201 can be used is used. Can learn. According to the reinforcement learning device 100, by referring to the observed state value regarding the generator 201, based on the characteristic function, for each command value for the generator 201 in a region in which two or more continuous regions are combined among a plurality of regions. Effectiveness information indicating effectiveness can be generated. According to the reinforcement learning device 100, learning can be performed using the generated validity information about the combined region and the validity information about each region other than the two or more regions of the plurality of regions. . As a result, the reinforcement learning device 100 can reduce the number of validity information that is the target of learning and updating, and can reduce the processing amount required for reinforcement learning.

  According to the reinforcement learning device 100, the validity information indicating the validity of each command value for the generator 201 in each of the two or more regions is referred to by referring to the observed state value regarding the generator 201. Can be generated. According to the reinforcement learning apparatus 100, learning is performed by using the validity information about each of the generated two or more areas and the validity information about each area other than the two or more areas of the plurality of areas. It can be performed. Thereby, the reinforcement learning device 100 can subdivide and execute what kind of command value should be output in what kind of state value.

  According to the reinforcement learning device 100, learning is performed by using the validity information indicating the validity of each combination of the command values of the generator 201 in each of the plurality of regions that the combination of the state values of the generator 201 can take. It can be performed. According to the reinforcement learning device 100, referring to the observed state value of the power generator 201, based on the characteristic function, the command value of the power generator 201 in the region in which two or more continuous regions are combined among the plurality of regions is Effectiveness information indicating the effectiveness for each combination can be generated. According to the reinforcement learning device 100, learning can be performed using the generated validity information about the combined region and the validity information about each region other than the two or more regions of the plurality of regions. . Thereby, the reinforcement learning device 100 can be applied when there are a plurality of generators 201.

  The reinforcement learning device 100 can generate validity information about the wind power generator. Thereby, the reinforcement learning device 100 can be applied to the power generation system 200 including the wind power generator.

  According to the reinforcement learning device 100, the output power corresponding to the observed wind speed can be specified based on the characteristic function, and the validity information about the combined region can be generated based on the specified output power. Thereby, the reinforcement learning device 100 can generate the effectiveness information about the stall-controlled wind power generator.

  According to the reinforcement learning device 100, the validity information about the combined region is generated based on the characteristic function corresponding to the observed wind speed and the output power among the plurality of characteristic functions that differ depending on the wind receiving performance of the generator 201. can do. Thereby, the reinforcement learning device 100 can generate the effectiveness information about the pitch-controlled wind power generator.

  The reinforcement learning device 100 can generate validity information about the thermal power generator. Thereby, the reinforcement learning device 100 can be applied to the power generation system 200 including a thermal power generator.

  According to the reinforcement learning device 100, when the observed demand power is less than or equal to the threshold value, it is possible to generate the validity information about the combined regions. As a result, the reinforcement learning device 100 can reduce the number of pieces of validity information to be learned and updated when detailed verification is not required for a relatively large output power region. Here, the reinforcement learning device 100 can suppress the adverse effect on the control of the power generation system by combining the regions of relatively large output power.

  According to the reinforcement learning device 100, when the observed demand power exceeds the threshold value, it is possible to generate the validity information about the combined regions. As a result, the reinforcement learning device 100 can reduce the number of pieces of validity information to be learned and updated when detailed verification is not required for a region of relatively small output power. Here, the reinforcement learning device 100 can suppress the adverse effect on the control of the power generation system if the regions of relatively small output power are combined.

  According to the reinforcement learning device 100, it is possible to generate the validity information about the area obtained by combining the two or more areas based on the validity information about each area of the two or more areas. As a result, the reinforcement learning apparatus 100 can reduce the number of pieces of validity information to be learned and updated to reduce the amount of processing required for reinforcement learning even if the characteristic function is unknown. .

  The reinforcement learning method described in the present embodiment can be realized by executing a prepared program on a computer such as a personal computer or a workstation. The reinforcement learning program described in the present embodiment is recorded in a computer-readable recording medium such as a hard disk, a flexible disk, a CD-ROM, an MO, or a DVD, and is executed by being read from the recording medium by the computer. Further, the reinforcement learning program described in the present embodiment may be distributed via a network such as the Internet.

  Regarding the above-described embodiment, the following supplementary notes are further disclosed.

(Supplementary note 1) For a computer,
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning program characterized by executing processing.

(Supplementary note 2) In the computer,
Effectiveness indicating the effectiveness of each command value for the generator in each of the two or more regions based on the characteristic function of the observed state value for the generator with reference to the observed state value for the generator Generate information,
Learning is performed by using the generated validity information about each of the two or more regions and the validity information about each of the plurality of regions other than the two or more regions,
The reinforcement learning program according to appendix 1, wherein the program is executed.

(Supplementary note 3) In the computer,
When there are a plurality of generators, learning is performed by using effectiveness information indicating effectiveness of each combination of command values of the generator in each of a plurality of areas that can be taken by a combination of state values of the generator. And then
By referring to the observed state value of the generator, based on the characteristic function, the effectiveness of each combination of the command values of the generator in the region in which two or more continuous regions are combined among the plurality of regions is shown. Generate the validity information shown,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
3. The reinforcement learning program according to supplementary note 1 or 2, which executes a process.

(Supplementary Note 4) The generator is a wind power generator,
The reinforcement learning program according to any one of appendices 1 to 3, wherein the state values related to the generator are wind speed and output power.

(Supplementary Note 5) The characteristic function indicates a relationship between wind speed and output power from the generator,
The process of generating the validity information about the combined area is based on the characteristic function, specifies the output power corresponding to the observed wind speed, and based on the specified output power, the effective about the combined area. A reinforcement learning program according to appendix 4, wherein the reinforcement learning program generates sex information.

(Supplementary Note 6) The wind power of the generator can be changed,
The command value is a command value for controlling the wind performance,
The characteristic function indicates the relationship between the wind speed and the output power from the generator,
The process of generating the validity information about the combined region is based on the characteristic function corresponding to the observed wind speed and output power among the plurality of characteristic functions that differ for each wind-receiving performance of the generator, and The reinforcement learning program according to appendix 4, wherein validity information about the combined areas is generated.

(Supplementary Note 7) The generator is a thermal power generator,
The reinforcement learning program according to any one of appendices 1 to 3, wherein the state value related to the generator is a fuel usage amount and an output power.

(Supplementary Note 8) The processing for generating the validity information about the combined area generates the validity information about the combined area when the observed demand power is less than or equal to a threshold value. The reinforcement learning program according to any one of 1 to 7.

(Supplementary note 9) The process of generating the validity information about the combined area generates the validity information about the combined area when the observed demand power exceeds a threshold value. Reinforcement learning program described in any one of.

(Supplementary note 10) The computer
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning method characterized by executing processing.

(Supplementary Note 11) Learning is performed by using validity information indicating validity of each command value for the generator in each of a plurality of regions in which a state value related to the generator can be taken,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
A reinforcement learning device having a control unit.

(Supplementary note 12) In a computer,
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning program characterized by executing processing.

(Supplementary note 13) Computer
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning method characterized by executing processing.

(Supplementary Note 14) Learning is performed by using validity information indicating validity of each command value for the generator in each of a plurality of regions in which a state value related to the generator can be taken,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
A reinforcement learning device having a control unit.

100 Reinforcement Learning Device 200 Power Generation System 201 Generator 210 Network 300 Bus 301 CPU
302 memory 303 network I / F
304 recording medium I / F
305 recording medium 400 storage unit 401 acquisition unit 402 switching unit 403 learning unit 404 output unit 501 table generation unit 502 section switching unit 503 value setting unit 504 action determination unit 505 state calculation unit 506 reward calculation unit 507 table update unit 601, 602 701, 702, 1000 Graph 801 Normal table 802 Coarse division table 1100, 1200 Predicted output power table 1310-1312, 1320 fields

Claims (13)

On the computer,
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning program characterized by executing processing. On the computer,
Effectiveness indicating the effectiveness of each command value for the generator in each of the two or more regions based on the characteristic function of the observed state value for the generator with reference to the observed state value for the generator Generate information,
Learning is performed by using the generated validity information about each of the two or more regions and the validity information about each of the plurality of regions other than the two or more regions,
The reinforcement learning program according to claim 1, which executes a process. On the computer,
When there are a plurality of generators, learning is performed by using effectiveness information indicating effectiveness of each combination of command values of the generator in each of a plurality of areas that can be taken by a combination of state values of the generator. And then
By referring to the observed state value of the generator, based on the characteristic function, the effectiveness of each combination of the command values of the generator in the region in which two or more continuous regions are combined among the plurality of regions is shown. Generate the validity information shown,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
The reinforcement learning program according to claim 1 or 2, which executes a process. The generator is a wind generator,
The reinforcement learning program according to any one of claims 1 to 3, wherein the state value regarding the generator is wind speed and output power. The characteristic function indicates the relationship between the wind speed and the output power from the generator,
The process of generating the validity information about the combined area is based on the characteristic function, specifies the output power corresponding to the observed wind speed, and based on the specified output power, the effective about the combined area. The reinforcement learning program according to claim 4, wherein sex information is generated. The generator is capable of changing the wind receiving performance,
The command value is a command value for controlling the wind performance,
The characteristic function indicates the relationship between the wind speed and the output power from the generator,
The process of generating the validity information about the combined region is based on the characteristic function corresponding to the observed wind speed and output power among the plurality of characteristic functions that differ for each wind-receiving performance of the generator, and The reinforcement learning program according to claim 4, wherein validity information about the combined regions is generated. The generator is a thermal power generator,
The reinforcement learning program according to any one of claims 1 to 3, wherein the state value regarding the generator is a fuel usage amount and an output power.   The processing of generating the validity information about the combined area generates the validity information about the combined area when the observed demand power is less than or equal to a threshold value. Reinforcement learning program described in any one of. Computer
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning method characterized by executing processing. Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
For each command value for the generator in a region obtained by combining two or more continuous regions of the plurality of regions with reference to the observed state value for the generator and the characteristic function of the state value for the generator Generates validity information that shows the effectiveness of
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
A reinforcement learning device having a control unit. On the computer,
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning program characterized by executing processing. Computer
Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
Reinforcement learning method characterized by executing processing. Learning is performed by using the validity information indicating the validity of each command value for the generator in each of the plurality of regions where the state value related to the generator can take,
Based on the validity information indicating the validity of each command value for the generator in each of two or more consecutive regions of the plurality of regions, the generator for the region in which the two or more regions are combined Generates validity information indicating the validity of each command value,
Learning is performed by using the generated validity information about the combined region and the validity information about each region other than the two or more regions among the plurality of regions,
A reinforcement learning device having a control unit.

Images