KR20230127001A - Method and apparatus for controlling based on reinforcement learning for reducing electricity bill using energy stroage system - Google Patents

Abstract

A reinforcement learning-based control method and apparatus for an energy storage device are disclosed. Determining a control signal for charging and discharging the energy storage device based on a reinforcement learning-based control model that takes as inputs the electricity rate for each time period and the cost of depreciation prediction due to the use of the energy storage device, and based on the control signal, energy Charging or discharging the storage device may be included.

Description

Reinforcement learning-based control method and device for reducing electricity bills using energy storage devices

The following embodiments relate to a reinforcement learning-based control technology that reduces electricity bills using an energy storage device.

A typical energy storage system is composed of a secondary battery, a lithium-ion battery, and based on the conventional operating method of charging the battery at low electricity rates and preferentially using the energy of the energy storage system during high electricity rates, It works. However, conventional operating methods do not take into account depreciation costs that may be incurred by the use of energy storage devices. Therefore, the conventional operating method cannot be said to be an optimal operating method. Energy storage systems are expensive equipment, and the depreciation cost is enormous. Therefore, there is a need for research on an operating method that considers both depreciation cost and reduction of electricity charges due to battery use.

The determining of the control signal may include charging the energy storage device such that a sum of a total electric charge for electricity consumed when charging the energy storage device and a predicted depreciation cost due to use of the energy storage device is minimized. A step of generating the control signal for controlling discharge may be included.

The generating of the control signal may include the energy storage device based on a current remaining lifespan of the energy storage device, an aging factor of the energy storage device, a current charging amount of the energy storage device, and a stress factor of the energy storage device, using a control model. Calculating a depreciation prediction cost by use of ; Calculating the total electricity price based on predicted future power demand and electricity price for each time period; and generating the control signal for controlling charging and discharging of the energy storage device for each time period such that the sum of the estimated depreciation cost due to the use of the energy storage device and the total electricity cost becomes a minimum value.

The control model is configured to determine the amount of charging and discharging based on the current charging amount of the energy storage device, the predicted future power demand, the electricity rate for each time slot, and the predicted depreciation cost due to the use of the energy storage device. The learning agent can be trained.

The control model is the amount of charging and discharging using at least one deep neural network of DQN (Deep Q-learning Network), dbl-DQN (Double Deep Q-learning Network) and duel-DQN (Dueling Deep Q-learning Network). can determine

According to an embodiment, an optimal energy storage device operating method capable of minimizing the sum of the depreciation cost due to the use of the energy storage device and the total electricity cost incurred due to the use of electricity may be determined.

According to one embodiment, it is possible to minimize the total electricity cost and the cost required for operating the battery.

According to an embodiment, depreciation applicable to the entire period of battery life can accurately reflect both non-linear and quasi-linear deterioration applied to the initial period of battery life as well as linear deterioration applied only after the initial period of battery life. A cost estimation process can be provided.

1 is a diagram illustrating a reinforcement learning-based control system for an energy storage device according to an embodiment.
2 is a diagram for explaining a control method according to an exemplary embodiment.
3 is a diagram for explaining electricity rates for each time zone according to an embodiment.
4 to 6 are diagrams for explaining a reinforcement learning algorithm according to an embodiment.
7 is a diagram showing the configuration of a control device according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed and implemented in various forms. Therefore, the form actually implemented is not limited only to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, such terms should only be construed for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element.

It should be understood that when an element is referred to as being “connected” to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle.

Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in this specification, it should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same reference numerals are given to the same components regardless of reference numerals, and overlapping descriptions thereof will be omitted.

The reinforcement learning-based control system for an energy storage device (or energy storage system; ESS) described in this specification is based on timely electricity price fluctuations and energy storage caused by an imbalance in power supply and demand per hour. An optimized battery management system for minimizing the sum of the depreciation cost of the energy storage device and the total electricity cost may be provided based on costs that may occur due to deterioration of the device (or battery). The control device of the control system may learn the control model based on reinforcement learning. The control model may determine a control signal for charging and discharging the energy storage device such that the sum of the total electricity cost and the estimated depreciation cost is minimized. The depreciation cost may be a cost incurred due to a decrease in the economic value of the energy storage device. The degree of degradation of the energy storage device may be determined based on a usage pattern of the energy storage device and an operating environment such as temperature.

The control model described in this specification can complete a function corresponding to the human brain through a long-term learning experience. That is, the control model may receive an input and output an output. Inputs are usually information about reality, and outputs can be correct actions. The control model can receive information about the energy storage device and output actions including correct decisions about charging and discharging the energy storage device. The control model may receive inputs including a current time, a current charge amount of the energy storage device, a charging capacity that is a maximum capacity in which power can be charged in the energy storage device, and a predicted value of future power demand. The control model may output an output including a control signal for charging and discharging the energy storage device based on the input.

1 is a diagram illustrating a reinforcement learning-based control system for an energy storage device according to an embodiment.

Referring to FIG. 1 , the control system may include an energy storage device 130 and a control device 140 for controlling charging and discharging of the energy storage device. The energy storage device 130 may be charged by receiving power from the power supply 120 and may discharge the charged power to provide power to the facility 110 . The control device 140 may control a charging time, a charging amount, a discharging time, and a discharging amount of the energy storage device 130 . If the controller 140 determines that it is a more cost-effective action for the facility 110 to receive power directly from the power supply 120 than to charge and discharge the energy storage device 130, the facility 110 ) may receive power directly from the power supply 120 .

2 is a diagram for explaining a control method according to an exemplary embodiment.

Referring to FIG. 2 , in step 210, the controller may predict future power demand of a facility to which the energy storage device supplies energy. In one embodiment, the control device may predict future power demand using a prediction algorithm. For example, the control device may predict future power demand of a facility to which the energy storage device supplies energy based on the measured power demand for each time zone of a predetermined period. The control device may calculate an average of the measured power demand for each time zone of a predetermined period, and predict future power demand for each time zone of a facility supplying energy from the energy storage device based on the average. In an example, the controller may predict future power demand of a facility supplied with energy by the energy storage device based on an average of power demand of the facility supplied with energy by the energy storage device for each time period for 10 weeks. In another example, the controller may predict future power demand of a facility powered by the energy storage device based on a trend of power demand of the facility powered by the energy storage device for a predetermined period of time. For example, if the power demand of a facility supplying energy from an energy storage device has been trending upward for the past 10 weeks, the control device does not predict the future power demand based on the average of the power demand, but predicts the future power demand based on the trend. You can also predict demand.

In step 220, the control device receives reinforcement learning using the current state of charge (SoC) of the energy storage device, predicted future power demand, electricity rates for each time slot, and the estimated depreciation cost due to the use of the energy storage device as inputs. A control signal for charging and discharging of the energy storage device may be determined based on the based control model. In one embodiment, the control model is a reinforcement learning agent that determines the amount of charging and discharging based on the current charging amount of the energy storage device, the predicted future power demand, the electricity rate for each time zone, and the predicted cost of depreciation due to the use of the energy storage device. can be learned. The reinforcement learning agent is a virtual entity representing decision-making and may be a control model described herein or at least one of deep neural networks generated by the control model.

The control model is a given input using, for example, a deep neural network of at least one of DQN (Deep Q-learning Network), dbl-DQN (Double Deep Q-learning Network), and duel-DQN (Dueling Deep Q-learning Network). An optimal charge/discharge amount corresponding to the value may be determined. The input layer may receive a state including a current amount of charge, power demand, and current time of the energy storage device.

Descriptions related to reinforcement learning are described in more detail in FIGS. 4 to 6 . In addition, electricity rates for each time zone are described in FIG. 3 .

In one embodiment, the control device controls charging and discharging of the energy storage device so that the sum of the total electricity cost for electricity consumed when charging the energy storage device and the estimated depreciation cost due to the use of the energy storage device is minimized. signal can be generated. In more detail, the control device uses the control model, based on the remaining life of the current energy storage device, the aging factor of the energy storage device, the current charge amount of the energy storage device, and the stress factor of the energy storage device. Depreciation forecast cost can be calculated. The control device may calculate the total electricity price based on the predicted future power demand and the electricity price for each time period. The control device may generate a control signal for controlling charging and discharging of the energy storage device for each time period such that the sum of the estimated cost of depreciation due to the use of the energy storage device and the total electricity cost becomes a minimum value.

The control signal may include, for example, a signal for controlling charging/discharging for each time period. When the energy storage device determines to charge the energy storage device in a specific time period, the control signal may further include information about the charging amount.

Depending on the embodiment, the control device may determine the control signal as not charging/discharging the energy storage device. In this case, energy stored in the energy storage device is not provided to the facility, but energy from the power supply may be directly provided to the facility.

In step 230, the control device may charge or discharge the energy storage device based on the control signal.

3 is a diagram for explaining electricity rates for each time zone according to an embodiment. FIG. 3 may show electricity rates based on Time-of-Use (TOU) applied in Korea. Electricity rates can be lowest between 23:00 and 8:00 the next day, and highest between 10:00 and 11:00 and 13:00 and 16:00. That is, the demand for electricity may be lowest between 11:00 and 8:00 the next day, and may be highest between 10:00 and 11:00 and between 13:00 and 16:00.

4 to 6 are diagrams for explaining a reinforcement learning algorithm according to an embodiment. 4 to 6 may show a deep neural network for learning a control model. Referring to FIGS. 4 to 6 , the deep neural network in each algorithm may include a vanilla neural network including an input layer, a hidden layer, and an output layer. Here, the input layer can receive a state. state is vector can be defined as where the demand vector is may include 13 elements including D _t , D _t+1 , ..., D _t+12 . As a result, the length of the state vector may be 16. There can be two hidden layers in the deep neural network of each algorithm, and each layer can contain 64 nodes. The deep neural network of FIG. 6 may further include a state dependent action advantage function layer between the hidden layer and the output layer. In one example, a ReLu function may be used as a parameter of a hidden layer of a deep neural network.

The control model may be learned based on a reinforcement learning algorithm. Reinforcement learning algorithms can be based on the Markov decision process (MDP), in which the current state and actions influence the stochastic transition to the next state. Based on these properties, the reinforcement learning algorithm can find the optimal action to minimize the sum of current and future depreciation prediction costs and total electricity bills. A Markov decision process can be represented as a tuple < S, A, R, γ, P >. where S is the state space for all states s, and A can be the action space for all operations. R can be a bivariate reward function R(s, a) that provides an immediate reward to the reinforcement learning agent when the control model in state s chooses action a. γ is a discount factor that quantifies the relative importance between the immediate reward and the next reward, and P can represent a probabilistic transition between two adjacent time steps. The conversion function may be as shown below.

The transition function can measure the probability that state s' will occur at time t+1, if the reinforcement learning agent selects action a in state s at time t.

Reinforcement learning may be a statistical learning method in which a reinforcement learning agent is trained to learn an optimal course of action in an MDP environment, and the course of the optimal course of action may be referred to as an optimal policy. By following the optimal policy, the reinforcement learning agent can minimize the sum of the depreciation prediction cost and the total electricity bill. Reinforcement learning agents can learn optimal policies through experience without prior knowledge. In reinforcement learning, the Q-function can represent a bivariate function for the expected return from choosing action a in state s. For a reinforcement learning agent whose policy is ð, the Q-function can be formulated as:

is a return, which is the sum of rewards reduced at time t, R _t is a reward at time t, and γ may be a discount rate. Q-learning can be in the same family as reinforcement learning algorithms. In Q-learning, starting from an initial policy ð ₀ , the reinforcement learning agent can iteratively update policy ð until it reaches an optimal policy ð. The core of Q-learning can be the estimation of a policy (evaluation of a policy) and the update of the running policy according to the current estimation (policy improvement).

Q-learning can tabulate the Q-function if both the state space and the action space are small enough, but a functional approximation of the Q-function may be needed when either the state space or the action space becomes large. A Deep Q-learning Network (DQN), a functional approximation for Q-learning using deep neural networks, can be proposed. The structure of the deep-Q-learning network may be as shown in FIG. 4 . Deep-Q-learning networks can use a single Q-function estimate at each time step. The output of the last hidden layer is It is passed to the layer for (s,a) so that it can estimate the value of each of the five possible operations. In the last layer, the reinforcement learning agent can select the best action with the highest Q value.

5 may show the structure of a Double DEEP Q-learning Network (dbl-DQN). Dual deep Q-learning networks can reduce overestimation that can occur in DQNs using single Q-function estimation. dbl-DQN can use two separate Q-learning networks. At each stage, one network may perform policy evaluation and another network may perform policy remediation. However, the role of each step is not limited to the contents disclosed in this specification and may be continuously changed throughout the learning process.

6 may show the structure of a duel deep Q-learning network (duel-DQN). Dual DQN may be a decomposition of a bivariate Q-function into a state value function and a state-dependent behavioral advantage function. The first component, the state value function, can estimate the contribution of the state input to the Q-function. The second component, the state-dependent action advantage function, can estimate how each action contributes to the Q-function in terms of the average outcome of all possible actions conditioned by a particular state. A reinforcement learning agent can formulate a single action variable as a preliminary charge strength using a dual DQN algorithm to check if the algorithm is suitable when there are many actions with similar values. Here, the strength of the reinforcement learning agent's possible actions is different, but not completely opposite, so in this regard, the reinforcement learning agent can use dual DQN. Dual DQNs can include one layer for state value functions and another two layers for behavioral benefit functions.

The predicted depreciation cost may be a cost expected to be incurred due to a decrease in the economic value of the energy storage device. The degree of deterioration of the energy storage device may be determined based on a usage pattern of the energy storage device and an operating environment such as temperature. When the control model calculates the depreciation forecast cost, the current state of the energy storage device may be taken into account, in which future use of the energy storage device may affect the degree of degradation of the energy storage device performance. Any relevant information about whether or not there is a In this process, the control model determines the remaining life of the energy storage device at the next moment ( _t+1 ) can be formulated to be a function of the factors. The factors are currently remaining energy storage device life ( _t ), the current SoC (X _t ), the next instant SoC (X _t+1 ), and other environment variables. The goal of reinforcement learning is _t+1 equals f( _t , X _t , X _t+1 , operating environment) may be to find a recursive degrading function, f(·). This formula allows us to estimate the estimated cost of depreciation if the SoC changes from x% to y% when the current remaining life is 1%.

Factors causing deterioration of the energy storage device may include aging factors and stress factors. The aging factor may include at least one of calendar aging, which is an aging factor irrespective of use, and cycle aging, which is an aging factor caused by charging and discharging. The stress factors include DoD stress (depth of discharge stress) caused by a change in the charge amount of the energy storage device, SoC stress (state of charge stress) caused by maintaining a high charge amount, and temperature during operation of the energy storage device. It may include at least one of temperature stress (temperature stress) caused by and time stress (time stress) caused by repetitive use of the energy storage device.

The lifetime of current energy storage devices can be calculated based on the following equation.

constant α and deals with aging factors, N is the number of cycles and f _d,1 can mean stress deterioration for one cycle including all stress factors. The cycle may mean a charge/discharge cycle of 20-80 or 30-80. f _d,1 can be expressed as the following equation.

DoD denoted by δ can be denoted by δ = |X _t +1 -X _t |, and SoC denoted by σ can be denoted by σ = (X _t + X _t+1 )/2.

The life cycle of an energy storage device is characterized by 1) rapid non-linear degradation at the beginning of its life and 2) quasi-linear degradation at the beginning of its life. Given the state at time t, the expected remaining life at time t+1 can be estimated based on the steps below.

(Step 1) parameters (α and ) and given operating conditions such as temperature.

(Step 2) Current conditions after planned use between time t and time t+1(X _t+1 ) (X _t , _t ) and the quantity for SoC is known.

(Step 3) Calculate f _d,1 from the parameters of Step 2 and the parameters of Step 1 using Equation 3.

(Step 4) In Equation 3, LHS is set to t; N is a uniquely unknown variable, which can be determined numerically using either a fixed-point algorithm or a bisection algorithm. value can be said this is The energy storage lifetime at t is from X _t to X _t+1 It can be said to be equivalent to repeating a cycle.

(Step 5) Using Equation 3 with factor N equal to +1, Calculate t.

(Step 6) Depreciation forecast cost is t - _t+1 ) Calculated as Х (the price of a new energy storage unit).

In general, factors that cause a decrease in the amount of charge to be stored in an energy storage device may include efficiency and deterioration of the energy storage device. In terms of efficiency, the charge/discharge efficiency of an energy storage device may not generally be 100%. Although it may vary depending on the operating conditions, the charge/discharge efficiency may generally be less than that, and may be even less when discharged without using electricity after charging. Deterioration may also occur when the energy storage device is in operation. Depreciation forecasting costs may occur when the SoC changes from X _t to X _t+1 regardless of the charging and discharging of the energy storage device.

In the MDP formula, an MDP problem is described as a tuple of (S, A, R, γ, P), where the elements of the tuple can represent states, actions, rewards, discount rates, and stochastic transitions, respectively.

State S is the reinforcement learning agent at time t at the current time (h _t ), ESS ( ) of capacity, current SoC (X _t ), power demand for the next 12 hours may have been observed.

Assuming the process starts at time t = 0, the current time h _t can be specified as mod(t, 24). The current time h _t may provide information about electricity rates based on TOU.

ESS( ) and the current SoC (X _t ) may indicate the state of the energy storage device at time t. demand vector may represent the future power demand of a facility to which the energy storage device supplies energy. State S _t is We can complete the vectorized notation for the state as ESS( ) is the capacity of ESS may be proportional to the remaining life of in other words,

can be the same as the above expression, where may represent the capacity of the new energy storage device.

The control model can be learned through experience to find the optimal course of action in any state. For example, if future electricity demand is high and current electricity rates are low, the control model can purchase electricity and store it in an energy storage device for future use. However, the current state of the energy storage device _t and X _t ) would result in high depreciation forecasting costs, the control model can avoid this task.

Conversion P allows the facility to meet the demand vector D _t , which represents the amount of electricity demand based on the purchase of new energy without charging the energy storage device, discharging the energy storage device, or a combination of the two. P _t may represent the amount of energy (or amount of power) purchased at time t. Based on D _t and P _{t ,} the equation below can be verified.

LHS means an increase or decrease in SoC level, and RHS can indicate additional power after meeting immediate power demand. That is, when the amount of power purchased is greater than the amount of immediate power demand (when P _t > D _t ), the SoC of the energy storage device may increase. On the other hand, when P _t < D _t , the SoC of the energy storage device may decrease. If P _t and D _t are equal, the SoC of the energy storage device can remain the same. of energy storage The capacity conversion of can be made based on the degradation process.

decision at time t , the control model can meet the power demand of the facility and determine the amount of charge to be stored in the energy storage device. Several variables may be involved in the process by which the control model determines the amount of charge to be stored in the energy storage device. When formulating the MDP, the action variable |S| ^|A| Determining behavioral variables can be important, as it determines the entire search space known to be proportional to . where |S| is the size of the state space and |A| It can be the size of the action space.

Actions are univariate and you may need to develop an action space such that it is a percentage value, i.e. A _t ∈ [0, 1]. This development may be based on the amount of electricity purchased by the facility at time t, P _t . Since the facility must meet its immediate demand, D _{t ,} the amount of electricity purchased by the facility at time t is can be bigger here, may be the positive part of x. If the current charge of the energy storage is sufficient to meet the immediate demand, the facility does not need to purchase power, but if not, the facility may need to purchase more than the shortfall (D _t -X _t ). Therefore, the minimum value of the amount of electricity (P _t ) that the facility will purchase at time t is can be On the other hand, the current charge amount available from the current charge amount of the energy storage device and the instantaneous power demand may be X _t -D _t . At this time, the energy storage device is can be charged up to Therefore, the maximum value of the amount of electricity to be purchased by the facility at time t may be as follows.

Based on the above, A _t is defined as a percentage variable in the amount of power to be purchased so that when A t is 0, the amount of _power the facility will purchase is the minimum value, and when A _t is 1, the amount of power the facility will purchase is the maximum value It can be. In the general case of A _t ∈ [0, 1], P _t , which is the amount of electricity to be purchased, can be defined as follows.

where x- = min(x, 0) can be the negative part of x.

Since the definition of action A _t requires that conversions be formulated using action variables, the description of conversions in Equation 6 can be modified. In the equation for X _t , the modified equation of the couple related to the transition may be as follows.

For the reward R, the reward R _t may quantify the utilization of the control model at time t. Compensation may be a negative sum of an electricity price according to the amount of power to be purchased, which is P _t , and an estimated depreciation cost of the energy storage device. Based on this, the compensation may be as follows.

c _h is the electricity price per unit at time h, and the function can measure the cost of depreciation in units of currency.

For the depreciation rate γ, the control model can maximize the return G _t using the factor γ. Here, γ can be set to 1 because the time unit is given as 1 hour and the depreciation rate can be ignored. The control model can choose the optimal action to maximize returns in any state. Revenue can be expressed as:

7 is a diagram showing the configuration of a control device according to an embodiment.

Referring to FIG. 7 , the control device 700 may include a processor 710 and a memory 720. Depending on the embodiment, the control device 700 may further include a database 730.

The memory 720 is connected to the processor 710 and may store instructions executable by the processor 710 , data to be calculated by the processor 710 , or data processed by the processor 710 . Memory 720 may include non-transitory computer-readable media such as high-speed random access memory and/or non-volatile computer-readable storage media (e.g., one or more disk storage devices, flash memory devices, or other non-volatile solid state memory devices). can include

The database 730 may store data necessary for the control device 700 to perform a control method. For example, the database 730 may store electricity rates for each time period and past power demand of facilities.

The processor 710 executes functions and instructions to be executed in the control device 700 and controls the overall operation of the control device 700 . The processor 710 may perform one or more operations related to the operation of the control device described herein.

For example, the processor 710 may allow the controller 700 to predict the future power demand of a facility to which the energy storage device 740 supplies energy, the current charging amount of the energy storage device 740, and the predicted future power demand. Control to determine a control signal for charging and discharging of the energy storage device 740 based on a reinforcement learning-based control model that takes as inputs the electricity rate for each time period and the cost of predicting depreciation due to the use of the energy storage device 740. Device 700 can be controlled. Also, the processor 710 may control the control device 700 to charge or discharge the energy storage device 740 based on the control signal.

The hardware device described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited drawings, those skilled in the art can apply various technical modifications and variations based on this. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

110: facility 120: power supply
130, 740: energy storage device 140, 700: control device
710: processor 720: memory
730: database

Claims (5)

In the reinforcement learning-based control method for an energy storage device,
predicting future power demand of a facility to which the energy storage device supplies energy;
of the energy storage device based on a reinforcement learning-based control model that takes as inputs the current charging amount of the energy storage device, the predicted future power demand, electricity rates for each time zone, and the predicted cost of depreciation due to the use of the energy storage device. determining a control signal for charging and discharging; and
Charging or discharging the energy storage device based on the control signal.
including,
control method. According to claim 1,
Determining the control signal,
Generating the control signal for controlling the charging and discharging of the energy storage device so that the sum of the total electricity cost for electricity consumed when charging the energy storage device and the estimated depreciation cost due to the use of the energy storage device is minimized step to do
including,
control method. According to claim 2,
Generating the control signal,
Using the control model, the estimated depreciation cost due to the use of the energy storage device based on the remaining life of the current energy storage device, the aging factor of the energy storage device, the current charge amount of the energy storage device, and the stress factor of the energy storage device calculating;
Calculating the total electricity price based on predicted future power demand and electricity price for each time period; and
Generating the control signal for controlling the charging and discharging of the energy storage device for each time period such that the sum of the estimated depreciation cost due to the use of the energy storage device and the total electricity cost becomes a minimum value.
including,
control method. According to claim 1,
The control model,
Training a reinforcement learning agent to determine the amount of charging and discharging based on the current charge amount of the energy storage device, the predicted future power demand amount, the electricity rate for each time slot, and the predicted cost of depreciation due to the use of the energy storage device ,
control method. According to claim 1,
The control model,
Determining the amount of charging and discharging using at least one deep neural network of DQN (Deep Q-learning Network), dbl-DQN (Double Deep Q-learning Network) and duel-DQN (Dueling Deep Q-learning Network),
control method.

Images