CN110598925A - Energy storage in-trading market decision optimization method based on double-Q learning algorithm - Google Patents

Abstract

A method for optimizing decision-making of energy storage in a trading market based on a double-Q learning algorithm, comprising the following steps: establishing a mathematical model for decision-making of energy storage in a trading market; describing energy storage operations as a Markov decision-making process; using real historical market transactions For the price data, use the Double-Q learning algorithm to iteratively train the two data sets to obtain the trained Q table; the energy storage executes the action of maximizing the decision-making goal in the trained Q table, and obtains the cumulative value under the joint arbitrage award. The Double-Q learning algorithm of the present invention uses two functions to iteratively update the Q table, which reduces the impact of the Q-learning algorithm overestimation problem, and the designed arbitrage strategy is more stable, making the long-term arbitrage income of energy storage higher; the source of arbitrage is not limited to The electricity market, joining the carbon market, has significantly increased arbitrage income.

Description

A decision-making optimization method for energy storage in the trading market based on double Q learning algorithm

technical field

The invention belongs to the technical field of engineering.

Background technique

With the increasing penetration of renewable resources, it is important to efficiently achieve this balance given the high uncertainty of wind and solar energy. The energy storage system can continuously absorb energy and release energy in a timely manner to meet the large demand of users for electricity, relieve the overload of electricity on the power grid, optimize the configuration of the power grid system, maintain the complete and stable operation of the power grid, and meet the needs of different users for power. demand, is a complement to variable renewable energy, and its economic viability is gaining increasing attention. One of the most frequently discussed sources of income for energy storage is real-time price arbitrage, that is, energy storage uses the price difference in real-time electricity market prices to charge when electricity prices are low and discharge when electricity prices are high to achieve profitability.

Due to the high volatility of real-time electricity market prices caused by the increasing popularity of intermittent renewable generation, the decision-making of energy storage in trading markets has received great attention from the research community. However, even with rising price spreads, designing a good strategy to make significant profits is still not an easy task. The first method that comes to mind is to predict the price, but the accuracy of the prediction is difficult to guarantee. Later, approximate dynamic programming was used to derive bidding strategies for energy storage without prior knowledge of the price distribution. However, this strategy is often computationally expensive due to the high dimensionality of the state space. Reinforcement learning is an online learning technology that is different from supervised learning and unsupervised learning. The Q-learning algorithm in reinforcement learning provides people with an energy storage decision-making strategy under a data-driven framework.

The existing energy storage decision-making methods based on reinforcement learning have the following defects: only electricity price information is used for decision-making, and the source of decision-making information is single; the Q-learning algorithm has an overestimation problem, and the performance of the algorithm is unstable.

Contents of the invention

The purpose of the present invention is to overcome the deficiencies of the prior art, and propose a decision-making optimization method for energy storage in the trading market based on a double-Q learning algorithm.

The present invention is achieved through the following technical solutions.

A method for optimizing decision-making of energy storage in a trading market based on a double-Q learning algorithm according to the present invention comprises the following steps:

Step 1: Establish a mathematical model for decision-making of energy storage in the trading market;

Step 2: Describe the energy storage operation as a Markov decision process;

Step 3: Use the real historical market transaction price data and use the Double-Q learning algorithm to iteratively train the two data sets to obtain the trained Q table;

Step 4: Energy storage executes the action of maximizing the decision-making goal in the trained Q table, and obtains the cumulative reward under joint arbitrage.

Further, said step 1 includes the following steps:

Step 1-1: Determine the objective function of energy storage decision-making;

Step 1-2: Determine the energy storage constraints of the energy storage system;

Step 1-3: Determine the charging and discharging power constraints of the energy storage system.

Further, said step 2 includes the following steps:

Step 2-1: Set the energy storage action as a function of price;

Step 2-2: Determine the energy storage state space;

Step 2-3: Determine the energy storage action space;

Steps 2-4: Determine the action reward function.

Further, said step 3 includes the following steps:

Step 3-1: Determine the state of the energy storage system;

Step 3-2: Select an energy storage action according to the ∈-greedy strategy;

Step 3-3: Randomly select one of the two functions to update the Q table value, iterate 3000 times, and obtain the Q value table.

Compared with the prior art, the beneficial effects of the present invention are: (1) the Double-Q learning algorithm uses two functions to iteratively update the Q table, which reduces the impact of the Q-learning algorithm overestimation problem, and the designed arbitrage strategy is more Stability makes the long-term decision-making benefits of energy storage higher. (2) The price data is not limited to the electricity market, but has joined the carbon market, which has significantly increased the cumulative reward.

Description of drawings

Attached Figure 1 is a flow chart of the decision-making method for energy storage in the trading market.

Accompanying drawing 2 is the block diagram of Markov decision process.

Accompanying drawing 3 is the decision-making flowchart of Double-Q learning algorithm.

Detailed ways

The specific implementation will be described below in conjunction with the working principle of the accompanying drawings.

The double-Q learning algorithm-based energy storage decision optimization method in the trading market proposed by the present invention uses the double-Q learning algorithm to make decisions in the electricity and carbon market transactions in a certain place to maximize the cumulative reward. The method flow chart is shown in Figure 1 Specifically, the following steps are included:

Step 1: Establish a mathematical model for decision-making of energy storage in the electricity market and carbon market;

Step 2: Describe the energy storage operation as a Markov decision process;

Step 3: Use the real historical carbon price and electricity price data of a trading market in a certain place, and use the Double-Q learning algorithm to iteratively train the two data sets to obtain the trained Q table;

Step 4: Energy storage executes the action of maximizing the decision-making goal in the trained Q table, and obtains the cumulative reward under the decision-making of this method.

Further, said step 1 includes the following steps:

Step 1-1: Determine the objective function of energy storage joint arbitrage as:

Step 1-2: Determine the energy storage constraints of the energy storage system as:

Step 1-3: Determine the charging and discharging power constraints of the energy storage system:

Further, to describe the energy storage operation as a Markov decision process:

Step 2-1: Set the energy storage action as a function of price:

Step 2-2: Determine the energy storage state space function;

S=(P,Q)*E (5)

Step 2-3: Determine the energy storage action space function;

Steps 2-4: Determine the action reward function.

Further, said step 3, according to the algorithm flow chart of Fig. 3 includes the following steps:

Step 3-1: Obtain the historical price data of electricity and carbon market transactions in a certain place, and determine the state S of the energy storage system according to the state space function (5). What is different from the past is that we add another kind of price information in the decision state space, Make a decision between two price distributions;

Step 3-2: Calculate the reward value of all actions in each state according to the reward function (7), which is used to determine the choice of subsequent actions, as shown in Table 1:

Table 1

Award Value Table Action A1 Action A2 State S1 R(S1,A1) R(S1,A2) State S2 R(S2,A1) R(s2,A2) .... .... .... State Sn R(Sn,A1) R(Sn,A2)

Step 3-3: According to the ∈-greedy strategy, the algorithm will randomly select the action in the action function (6) with the probability ∈∈[0,1]. There is a probability of (1-∈) to choose the action with the largest reward value in the reward value table, which will avoid the algorithm iteration local optimum;

Step 3-4: Q-Learning is a model-free reinforcement learning technique that can find an optimal action selection strategy in MDP problems. It learns through an action-value function, and finally can give the desired action according to the current state and the optimal policy. One of its advantages is that it does not require knowledge of a model of the environment to perform expected value comparisons of actions. The max operation in standard Q-learning uses the same value to select and measure an action. This is actually more likely to pick an estimate that is too high, leading to an overly optimistic value estimate. In order to avoid this situation, we can decouple the selection and measurement. Every time the state is updated, one of the functions (8) and (9) is randomly selected to update the Q table value, so as to avoid the overestimation of the action value. Iteration 3000 times, get the Q value table, as shown in Table 2, choose the action with the largest Q value in the Q value table, and get the cumulative reward.

Table 2

Q value table Action A1 Action A2 State S1 Q(S1,A1) Q(S1,A2) State S2 Q(S2,A1) Q(s2,A2) .... .... .... State Sn Q(Sn,A1) Q(Sn,A2)

Figure 2 is a block diagram of the Markov decision process. The purpose of the Markov decision problem is to find an optimal strategy, that is, a series of actions that maximize the evaluation function. For the state S at each moment, the agent will select the appropriate action through the optimal strategy. To maximize the cumulative reward of the decision objective, we describe the energy storage operation as a Markov decision process. Identify elements such as states, actions, strategies, and rewards. The charging and discharging decision is defined as a function of price information, and a double Q learning strategy is designed to optimally control the real-time decision of the energy storage system in the electricity market and carbon market.

Figure 3 is a flow chart of the double Q learning algorithm. When the price data is input into the state space for training, the performance of the double Q learning algorithm is more stable than that of the Q learning algorithm, reducing overestimation. When applied to energy storage joint arbitrage, first initialize, then determine the current state, select the energy storage action, and adopt the ε-greedy action selection strategy. The algorithm will randomly select the action with the probability of ε∈[0,1], and the 1-ε Probabilistically select the optimal action, the key lies in the two update functions used in double Q learning, one is used to determine the value generated by the action, and the other is used to update the Q value table.

Claims (4)

1. A decision optimization method of energy storage in trading market based on double Q learning algorithm is characterized by comprising the following steps:

step 1: establishing a mathematical model for energy storage in a trading market decision;

step 2: describing the energy storage operation as a Markov decision process;

and step 3: performing iterative training on the two data sets by adopting real historical market trading price data and applying a Double-Qlearning algorithm to obtain a trained Q table;

and 4, step 4: the stored energy executes the action of decision-making goal maximization in the trained Q table, and the accumulated reward under the joint arbitrage is obtained.

2. The method for energy storage and market trading decision optimization based on the double-Q learning algorithm as claimed in claim 1, wherein the step 1 comprises the following steps:

step 1-1: determining an objective function of an energy storage decision;

step 1-2: determining a stored electricity constraint of the energy storage system;

step 1-3: and determining charge and discharge power constraints of the energy storage system.

3. The method for energy storage and market trading decision optimization based on the double-Q learning algorithm as claimed in claim 1, wherein the step 2 comprises the following steps:

step 2-1: setting the action of storing energy as a function of price;

step 2-2: determining an energy storage state space;

step 2-3: determining an energy storage action space;

step 2-4: an action reward function is determined.

4. The method for energy storage and market trading decision optimization based on the double-Q learning algorithm as claimed in claim 1, wherein the step 3 comprises the following steps:

step 3-1: determining the state of the energy storage system;

step 3-2: selecting an energy storage action according to an element-greedy strategy;

step 3-3: one of the two functions is randomly selected to update the Q value table, and the Q value table is obtained after 3000 iterations.