CN110276698B - Distributed renewable energy transaction decision method based on multi-agent double-layer collaborative reinforcement learning - Google Patents

Abstract

The invention discloses a distributed renewable energy transaction decision method based on multi-agent double-layer collaborative reinforcement learning, which comprises the following main steps of: 1) constructing a double-layer random decision optimization model of distributed renewable energy trading; 2) introducing a multi-agent double-layer collaborative reinforcement learning algorithm, carrying out learning training according to a theoretical framework of the multi-agent double-layer collaborative reinforcement learning algorithm, and establishing a function approximator and a collaborative reinforcement learning working mechanism; 3) calculating an estimated value of an optimal Q value function by using an iterative calculation method on the basis of the frame in the step 2); 4) and solving an optimization model by using the trained multi-agent double-layer collaborative reinforcement learning algorithm to complete optimization calculation. The invention considers the uncertainty in the distributed renewable energy transaction, can improve the income of the power generator in consideration of risks, and simultaneously maximizes the comprehensive benefit.

Description

Distributed renewable energy trading decision method based on multi-agent double-layer collaborative reinforcement learning

technical field

The invention relates to the field of intelligent power distribution networks, in particular to a distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning.

Background technique

With the progress and development of society, the global demand for green, clean and efficient power is increasing. More and more distributed renewable energy is connected to the distribution network. Distributed energy has the advantages of reasonable energy efficiency, low loss, It has the characteristics of less pollution, flexible operation and good system economy. The development mainly involves problems such as grid connection, power supply quality, capacity reserve, and fuel supply.

Although distributed photovoltaic and wind power generation has no fuel cost, its construction cost, operation cost and maintenance cost are high. At present, my country's new energy distributed generators mainly make profits through national and local government electricity price subsidies. However, with the increase in the penetration rate of distributed power, the profit model obviously does not conform to the laws of the market. Subsidizing distributed generators through user subscription fees can help generators to participate in market competition and make reasonable quotations based on their own potential benefits and power generation costs, thereby maximizing social benefits. At the same time, considering various uncertain information such as generator quotations, distributed power output fluctuations, and user subscriptions, the model can be solved by the multi-agent double-layer collaborative reinforcement learning solution method, which can quickly calculate the optimal scheduling decision and reduce risks. Improve economic efficiency.

SUMMARY OF THE INVENTION

In order to overcome the shortcomings of the existing transaction decision-making methods, the present invention proposes a distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning, which takes into account the quotations of power generators, the output fluctuation of distributed power sources, and user subscriptions. This is a two-layer stochastic programming model for distributed energy under uncertain information. The model is solved by a multi-agent two-layer collaborative reinforcement learning solution method, which can quickly calculate the optimal scheduling decision, reduce risks, and improve economic benefits.

The present invention realizes above-mentioned purpose through following technical scheme:

A distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning, comprising the following steps:

Step 1) Build a two-layer stochastic decision-making optimization model for distributed renewable energy trading;

Step 2) Introduce a multi-agent double-layer collaborative reinforcement learning algorithm, carry out learning and training according to the theoretical framework of the multi-agent double-layer collaborative reinforcement learning algorithm, and establish a function approximator and a collaborative reinforcement learning working mechanism; the function approximator adopts a A series of adjustable parameters and features extracted from the state action space are used to estimate the Q value. The approximator establishes a mapping from the parameter space to the state action to space of the Q value function. The mapping can be linear or nonlinear. The linear map analyzes solvability, and the typical form of the function approximator is as follows:

是一种可调近似参数向量，

是状态-动作对的特征向量，

是基函数(BF)，(·)^T表示矩阵转置操作；in

is a tunable approximate parameter vector,

is the eigenvector of the state-action pair,

is the basis function (BF), ( ) ^T represents the matrix transpose operation;

Step 3) utilizes iterative calculation method to obtain the estimated value of the optimal Q value function on the basis of the frame described in step 2);

Step 4) Use the trained multi-agent double-layer collaborative reinforcement learning algorithm to solve the optimization model, and complete the optimization calculation.

Preferably, the two-layer stochastic decision-making optimization model for distributed renewable energy trading in the step 1) includes upper-level planning modeling and lower-level planning modeling, respectively corresponding to two parts of the energy trading link.

Preferably, the opportunity-constrained programming that maximizes the optimistic value of the objective function constructed in the upper-level planning modeling, the optimization goal is to maximize economic benefits, and the constraints are composed of objective constraints and opportunity constraints, and the upper-level planning modeling The mathematical expression is as follows:

Constraint function:

—风电、光伏真实值与预测值偏差不确定性造成的随机变量，

—报价为λ时，在ξ与

场景下的发电商收益，β—承受风险置信度，

—满足β置信度下的预期收益，q^t,ξ—ξ场景下，下层规划中得到的该发电商在t时段中标电量，

—ξ场景下，下层决策得到的单位电量用户新能源认购补偿(下层决策输出)，c_base—单位发电成本，

—发电商在ξ与

场景下的违约罚款，γ—未完成电量的单位罚款，

—t时刻，ξ场景中标电量超出

场景下最大出力的不平衡电量，

—在

场景下在t时刻分布式电源实际出力上限，T—一个时段，默认值为一小时。In the formula, λ—the time-sharing quotation of the generator, where λ ^t is the quotation at time t, ξ—the random variable caused by the unknown quotation of the bidder,

- Random variables caused by the uncertainty of the deviation between the actual value and the predicted value of wind power and photovoltaics,

—When quoted as λ, when ξ and

The income of the generator under the scenario, β—the confidence level of risk tolerance,

—Meet the expected revenue under the β confidence, q ^t,ξ —ξ scenario, the power generation company obtained in the lower-level planning won the bid electricity in the t period,

- In the scenario of ξ, the user's new energy subscription compensation per unit of electricity obtained by the lower-level decision (the output of the lower-level decision), c _base - unit power generation cost,

- Power generators are

The penalty for breach of contract in the scenario, γ—the unit penalty for uncompleted electricity,

-At time t, the standard power in the ξ scene exceeds

The unbalanced power with the maximum output in the scene,

-exist

In the scenario, the upper limit of the actual output of the distributed power supply at time t, T—a period, and the default value is one hour.

Preferably, the lower-level planning modeling is used for the bidding scenario, aiming at the comprehensive benefit of market operation, optimizing the scheduling, and allocating the bid-winning power of each generator. The mathematical expression of the lower-level planning is as follows:

Constraint function:

—t时刻从外电网购电的单位成本，

—t时刻从i号光伏、风力发电商购电成本，

—t时刻从外电网购电电量，

—t时刻从i号光伏、风力发电商购电电量，

—t时刻第i号电力用户的负荷量，comp_pv、comp_wp—用户认购范围为光伏、风电等可再生能源内付出的每度电认购补偿，Q_load-pv-i、Q_load-wp-i—i号用户当日结算下应付费的光伏、风电认购电量，Q_pv、Q_wp、Q_grid—当日区域内消耗的光伏、风电、外部电量，υ_pv、υ_wp—当日区域内光伏、风力发电比例，α_i、β_i—第i号用户认购的光伏、风电比例，

—i号光伏、风力发电商申报的t时刻最大发电量。In the formula: N _pv , N _wp - the total number of photovoltaic and wind power generators in the area, L - the total number of power users in the area,

—The unit cost of purchasing electricity from the external grid at time t,

—The cost of purchasing electricity from PV and wind power generators at time t,

- Purchase electricity from the external power grid at time t,

— Purchase electricity from photovoltaic and wind power generators at time t,

—The load of the ith power user at time t, comp _pv , comp _wp — The subscription scope of the user is the subscription compensation for each kilowatt-hour of electricity paid in renewable energy such as photovoltaic and wind power, Q _load-pv-i , Q _{load-wp- i} —the amount of photovoltaic and wind power subscription electricity payable under the settlement of the user on the day, Q _pv , Q _wp , Q _grid —the photovoltaic, wind power, and external electricity consumed in the area on the day, υ _pv , υ _wp —the photovoltaic and wind power in the area on the day Proportion of power generation, α _i , β _i — the ratio of photovoltaic and wind power subscribed by the i-th user,

- The maximum power generation at time t declared by PV and wind power generators.

Preferably, in the step 2), multiple agents are used to deal with the randomness problem of the upper-level planning modeling and the lower-level planning modeling itself, and the mutual iteration between the upper-level planning modeling and the lower-level planning modeling; the two-layer collaborative strengthening algorithm The diffusion strategy is introduced in the reinforcement learning process, and the adaptive combination (ATC) mechanism is introduced into the reinforcement learning algorithm. The collaborative reinforcement learning algorithm can adapt to the randomness and uncertainty brought by distributed renewable energy, and can adapt to dual The layer stochastic decision optimization model is computationally complex; in addition, in order to avoid the storage of a large number of Q-value tables, a function approximator is used to record the Q-values of complex continuous state and action spaces.

Preferably, the diffusion strategy can achieve faster convergence and can achieve a lower mean square deviation than the consensus strategy, and the diffusion strategy is as follows:

是一个由扩散策略引入的中间项，x_i(k+1)是通过组合智能体i的所有中间项来更新的状态；N_i是与智能体i相邻点的集合；b_ij是由智能体i分配给相邻智能体j的权重；定义一个矩阵B＝[b_ij]∈R^n×n作为微电网通信网络的拓扑矩阵；拓扑矩阵B是一个随机矩阵，B1_n＝1_n，其中1_n∈Rⁿ是单位向量。in

is an intermediate term introduced by the diffusion strategy, _xi (k+1) is the state updated by combining all intermediate terms of agent _i ; _Ni is the set of points adjacent to agent i; The weight assigned by the body i to the adjacent agent j; define a matrix B=[b _ij ]∈R ^n×n as the topology matrix of the microgrid communication network; the topology matrix B is a random matrix, B1 _n =1 _n , where 1 _n ∈ R ⁿ is a unit vector.

Beneficial effects:

1. The two-layer decision-making optimization model established by the present invention can comprehensively consider the uncertain scenarios caused by these random variables, and make better decisions. Therefore, it is very suitable for optimal decision-making of distributed generators.

2. The algorithm proposed in the present invention is a two-layer collaborative reinforcement learning algorithm, which can be well integrated into the two-layer stochastic decision-making optimization model, and provides a new idea for the intensive energy trading decision of the future information network and energy network. .

3. The present invention introduces a plurality of agents to deal with the randomness of the upper and lower level planning itself and the mutual iteration of the upper and lower levels, so that the collaborative reinforcement learning algorithm is more suitable for the two-level planning problem.

4. Multi-agent double-layer collaborative reinforcement learning, as a multi-agent reinforcement learning algorithm with self-learning and collaborative learning capabilities, is more suitable for solving large-scale distributed access energy problems with strong randomness and uncertainty . After a certain training update, the algorithm can quickly perform dynamic optimization while ensuring the stability of global convergence.

5. The diffusion strategy is introduced in the reinforcement learning process, which enables distributed information exchange in the microgrid while reducing the computational cost, enabling faster convergence, and achieving a lower mean square deviation than the consensus strategy.

Description of drawings

Fig. 1 is the overall frame diagram of the present invention;

FIG. 2 is a flowchart of the multi-agent double-layer collaborative reinforcement learning of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

The distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning described in the present invention uses the distribution network as a medium to simultaneously dispatch distributed power sources and controllable loads to achieve economic benefit optimization. The schematic diagram of the model is shown in Figure 1.

The present invention proposes a distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning, and the method includes the following steps:

Step 1) Build a two-layer stochastic decision-making optimization model for distributed renewable energy trading;

Step 2) Introduce a multi-agent double-layer collaborative reinforcement learning algorithm, carry out learning and training according to the theoretical framework of the multi-agent double-layer collaborative reinforcement learning algorithm, and establish a function approximator and a collaborative reinforcement learning working mechanism;

Step 3) utilizes iterative calculation method to obtain the estimated value of the optimal Q value function on the basis of the frame described in step 2);

Step 4) Use the trained multi-agent double-layer collaborative reinforcement learning algorithm to solve the optimization model, and complete the optimization calculation.

The two-layer stochastic decision-making optimization model for distributed renewable energy trading in the step 1) includes upper-level planning modeling and lower-level planning modeling, respectively corresponding to two parts of the energy trading link

In the step 2), multiple agents are used to deal with the randomness of the upper-level planning modeling and the lower-level planning modeling itself, and the mutual iteration between the upper-level planning modeling and the lower-level planning modeling; the two-layer collaborative reinforcement algorithm is used in reinforcement learning. The diffusion strategy is introduced in the process, and the adaptive combination (ATC) mechanism is introduced into the reinforcement learning algorithm. The collaborative reinforcement learning algorithm can adapt to the randomness and uncertainty brought by distributed renewable energy, and can adapt to two-layer random decision-making. The optimization model is computationally complex; in order to avoid the storage of a large number of Q-value tables, a function approximator is used to record the Q-values of complex continuous state and action spaces.

The step 3) iterative calculation process includes the following steps (see Figure 2):

S1: Initialization: θ ₀ , ω ₀

S2: number of repetitions k = 1to T

S3: Each agent calculates i=1to n in turn

和状态s_i(k)S4: Calculate eigenvectors

and state s _i (k)

S5: Choose action a _i (k) according to policy π

S6: Observe the reward value ri ( _k )

S7: TD error δ _i (k)

S8: Estimation

S9: Update parameters θ _i (k), ω _i (k)

S10: Back to S3

S11: Back to S2

S12: Return the result.

The basic steps and explanations of the distributed renewable energy framework application of the multi-agent double-layer collaborative reinforcement learning described in the present invention are as follows:

A1: Decompose the objective function and constraint function of the upper and lower planning into the respective Rewards of the reinforcement learning algorithm, as the reference value of the reward, in which the upper planning objective function should get the maximum expectation, so it is set as a positive reward, and the lower planning objective function I hope the price is the lowest, so it is set as a reverse reward, and the constraints of the upper and lower planning are used as the penalty item, and the coefficient is set according to the actual debugging situation. coefficient can be.

A2: Build the first reinforcement learning module, which is essentially a combination of two (usually multiple) reinforcement learning agents. The lower layer planning is used as a module to build reinforcement learning agent units. Since there are multiple generators in the upper layer, each A power generator is used as a module to establish a reinforcement learning agent unit, and finally the upper-layer intelligent unit and the lower-layer intelligent unit are integrated through a whole intelligent unit, as shown in the agent II in Figure 1, at this time the intelligent The Reward composition of Body II aims to maximize the total reward of each agent unit.

A3: Build a function approximator. The storage of the Q value will take up a lot of computer resources, in order to reduce the occupation of computer resources and speed up the calculation.

A4: Establish a collaborative reinforcement learning working mechanism. In order to speed up the computational efficiency before multi-agent, an adaptive combination (ATC) diffusion strategy is established to integrate the parameter update process of Greedy-GQ.

A5: Construct a second reinforcement learning module, using Agent II as the environment of the agent, and using conventional Q-learning (or Sarsar, DQN, etc.) update rules to establish an update policy.

Upper-level planning modeling:

The chance-constrained programming that maximizes the optimistic value of the objective function constructed in the upper-level planning, its optimization objective is to maximize economic benefits, and the constraints are composed of objective constraints and opportunity constraints. Moreover, the upper layer optimization takes the optimistic value of economic benefit as the optimization goal (that is, the economic benefit obtained is better than this value under a certain confidence level) to minimize the operating cost of the distribution network. The objective constraints are limited to constraints on deterministic objects, including unit power generation costs, fines per unit of uncompleted power generation, and upper and lower limits for the actual processing of distributed power sources. The opportunity constraint is limited to the constraint conditions for the uncertainty object of the distribution network, including the probability constraint of the execution degree of risk tolerance, the safety limit of the power flow, and the like. The sources of the uncertainty factors include distributed photovoltaics, wind power output, uncertainty of bids by power generators, uncertainty of traditional load forecast deviation, and the like.

Therefore, the mathematical expression of the upper-level programming modeling is as follows:

Constraint function:

in the formula

λ——The time-of-use quotation of the generator, where λ ^t is the quotation at time t

ξ——Random variable caused by unknown bidder’s bid

——风电、光伏真实值与预测值偏差不确定性造成的随机变量

——Random variables caused by the uncertainty of the deviation between the actual value and the predicted value of wind power and photovoltaic power

——报价为λ时，在ξ与

场景下的发电商收益

——When quoted as λ, between ξ and

Power generator revenue in the scenario

β - the confidence level of risk taking

——满足β置信度下的预期收益

- Satisfy the expected return under the β confidence level

q ^t,ξ ——In the ξ scenario, the power generation company's winning bid power in the t period obtained from the lower-level planning

——ξ场景下，下层决策得到的单位电量用户新能源认购补偿(下层决策输出)

——In the ξ scenario, the user's new energy subscription compensation per unit of electricity obtained by the lower-level decision (the output of the lower-level decision)

c _base - unit power generation cost

——发电商在ξ与

场景下的违约罚款

——The power generator is in ξ and

Penalty for breach of contract

γ - unit fine for uncompleted electricity

——t时刻，ξ场景中标电量超出

场景下最大出力的不平衡电量

——At time t, the standard power in the ξ scene exceeds

The unbalanced power of the maximum output in the scene

——在

场景下在t时刻分布式电源实际出力上限

--exist

The upper limit of the actual output of the distributed power supply at time t in the scenario

T - a time period, the default value is one hour.

Lower-level planning modeling:

The lower-level planning, aiming at the comprehensive benefit of market operation, optimizes the dispatching and allocation of bidding rights of each power plant. The lower-level programming model is actually a market equilibrium scheduling model for regional retail markets. The accuracy of the model determines whether the regional market can function normally according to the rules. Due to the neglect of energy storage, the power purchase sources in this region include distributed generators and external grids, and the sum of the power purchase costs for each period constitutes the cost source of the system. In addition, considering that users are willing to pay a certain cost to order new energy and enjoy green power, this user group can also be included in the overall benefit. Therefore, the optimal target can minimize the cost of electricity purchase and increase the subscription fee for green electricity.

Therefore, the mathematical expression of the lower-level programming modeling is as follows:

Constraint function:

where:

N _pv , N _wp - the total number of photovoltaic and wind power generators in the region

L - the total number of electricity users in the area

——t时刻从外电网购电的单位成本

——The unit cost of purchasing electricity from the external grid at time t

——t时刻从i号光伏、风力发电商购电成本

——The cost of purchasing electricity from PV and wind power generators at time t

——t时刻从外电网购电电量

—— Purchase electricity from the external power grid at time t

——t时刻从i号光伏、风力发电商购电电量

——Purchasing electricity from photovoltaic and wind power generators at time t

——t时刻第i号电力用户的负荷量

——The load of the ith power user at time t

comp _pv , comp _wp - the subscription scope of users is the subscription compensation for each kilowatt-hour of electricity paid in renewable energy sources such as photovoltaics and wind power

Q _load-pv-i , Q _load-wp-i ——the amount of photovoltaic and wind power subscription electricity payable under the settlement of user i on the day

Q _pv , Q _wp , Q _grid - photovoltaic, wind power, external power consumption in the area on the day

υ _pv , υ _wp —— the ratio of photovoltaic and wind power generation in the region on the day

α _i , β _i - the proportion of photovoltaic and wind power subscribed by the i-th user

——i号光伏、风力发电商申报的t时刻最大发电量

——Maximum power generation at time t declared by PV and wind power generators

Function approximator:

The function approximator uses a series of tunable parameters and features extracted from the state action space to estimate the Q value. The approximator then builds a mapping from the parameter space to the state action-to-space of the Q-value function. The mapping can be linear or non-linear. Solvability can be analyzed using linear mapping. A typical form of a linear approximator looks like this:

是一种可调近似参数向量，

是状态-动作对的特征向量，由下式可得：in

is a tunable approximate parameter vector,

is the eigenvector of the state-action pair, which can be obtained by:

是基函数(BF)，例如高斯径向BF，其中心是状态空间中的选择不动点。一般情况下，对应于固定点的BFs集合均匀分布在状态空间中。在本文中，如果没有指定，所有向量被认为是列向量。(·)^T表示矩阵转置操作。径向基神经网络已被用于随机非线性互联系统，并已证明其具有良好的泛化性能。in

is a basis function (BF), such as a Gaussian radial BF, whose center is a choice fixed point in the state space. In general, the set of BFs corresponding to fixed points is uniformly distributed in the state space. In this article, if not specified, all vectors are considered column vectors. (·) ^T represents the matrix transpose operation. Radial Basis Neural Networks have been used in stochastic nonlinear interconnected systems and have demonstrated good generalization performance.

Diffusion strategy:

The reinforcement learning algorithm introduces the diffusion strategy in the reinforcement learning process, and the adaptive composition (ATC) mechanism is introduced into the reinforcement learning algorithm. Diffusion strategies can achieve faster convergence and can achieve lower mean squared deviations than consistent strategies. In addition, the diffusion strategy has better response performance to continuous real-time signals and is insensitive to adjacent weights. The basic idea of the diffusion strategy is to combine cooperative terms based on neighboring states during the self-state update process of each agent. Consider an agent _i with state xi and its dynamics.

x _i (k+1)=x _i (k)+f(x _i (k))

The diffusion strategy is as follows:

是一个由扩散策略引入的中间项，x_i(k+1)是通过组合智能体i的所有中间项来更新的状态。N_i是与智能体i相邻点的集合。此外，b_ij是由智能体i分配给相邻智能体j的权重。在这里，我们可以定义一个矩阵B＝[b_ij]∈R^n×n作为微电网通信网络的拓扑矩阵。一般来说，拓扑矩阵B是一个随机矩阵，这意味着B1_n＝1_n，其中1_n∈Rⁿ是单位向量。in

is an intermediate term introduced by the diffusion strategy, and _xi (k+1) is the state updated by combining all intermediate terms of agent i. Ni is the set of points adjacent to agent _i . Furthermore, b _ij is the weight assigned by agent i to neighboring agent j. Here, we can define a matrix B=[b _ij ]∈R ^n×n as the topology matrix of the microgrid communication network. In general, the topological matrix B is a random matrix, which means B1 _n = 1 _n , where 1 _n ∈ R ⁿ is the unit vector.

A collaborative reinforcement learning algorithm is presented by integrating the adaptive composition (ATC) diffusion strategy into the parameter update process of Greedy-GQ.

和

实际近似参数向量θ_i(k+1)和校正参数向量ω_i(k+1)是相邻代理的中间矢量的组合。在所提出的算法中，学习速率参数α(k)和β(k)可以用条件P(1)～P(4)来设定。It should be noted that the proposed collaborative reinforcement learning algorithm introduces two intermediate vectors:

and

The actual approximation parameter vector θ _i (k+1) and the correction parameter vector ω _i (k+1) are the combination of the intermediate vectors of adjacent agents. In the proposed algorithm, the learning rate parameters α(k) and β(k) can be set with conditions P(1)~P(4).

α(k)＞0, β(k)＞0 P(1)

α(k)/β(k)→0 P(4)

The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art can still modify or equivalently replace the specific embodiments of the present invention. , and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention are all within the protection scope of the claims of the present invention for which the application is pending.

Claims (3)

1. A distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning, is characterized in that, comprises the following steps: Step 1) constructing a two-layer stochastic decision-making optimization model for distributed renewable energy trading, and the two-layer stochastic decision-making optimization model for distributed renewable energy trading in step 1) includes upper-level planning modeling and lower-level planning modeling, respectively. Corresponding to the two parts of the energy trading link; Step 2) Introduce a multi-agent double-layer collaborative reinforcement learning algorithm, carry out learning and training according to the theoretical framework of the multi-agent double-layer collaborative reinforcement learning algorithm, and establish a function approximator; the function approximator adopts a series of adjustable parameters and from The features extracted from the state action space are used to estimate the Q value. The approximator establishes the mapping of the state action from the parameter space to the Q value function to the space. The mapping is linear or non-linear, and the linear mapping is used to analyze the solvability. The function approximator The typical form of is as follows:

in

is a tunable approximate parameter vector,

is the eigenvector of the state-action pair,

is the basis function (BF); ( ) ^T represents the matrix transpose operation; Step 3) utilizes iterative calculation method to obtain the estimated value of the optimal Q value function on the basis of the frame described in step 2); Step 4) use the trained multi-agent double-layer collaborative reinforcement learning algorithm to solve the optimization model, and complete the optimization calculation; the opportunity-constrained planning of maximizing the optimistic value of the objective function constructed in the upper-level planning modeling, the optimization goal of which is economic benefit Maximum, the constraints are composed of objective constraints and chance constraints, and the mathematical expression of the upper-level programming modeling is as follows:

Constraint function:

In the formula, λ—the time-sharing quotation of the generator, where λ ^t is the quotation at time t, ξ—the random variable caused by the unknown quotation of the bidder,

- Random variables caused by the uncertainty of the deviation between the actual value and the predicted value of wind power and photovoltaics,

—When quoted as λ, when ξ and

The income of the generator under the scenario, β—the confidence level of risk tolerance,

—Meet the expected revenue under the confidence of β, in the scenario of q ^{t ,ξ} —ξ, the power generation company’s winning bid power in the t period obtained in the lower-level planning, in the scenario of c _s ^ξ —ξ, the lower-level decision-making obtains the new energy for users per unit of electricity Subscription compensation, c _base — unit power generation cost,

- Power generators are

The penalty for breach of contract in the scenario, γ—the unit penalty for uncompleted electricity,

-At time t, the standard power in the ξ scene exceeds

The unbalanced power with the maximum output in the scene,

-exist

In the scenario, the upper limit of the actual output of the distributed power supply at time t, T—a period, and the default value is one hour; The lower-level planning modeling is used for the bidding scenario, aiming at the comprehensive benefit of market operation, optimizing scheduling, and allocating the bid-winning power of each generator. The mathematical expression of the lower-level planning is as follows:

Constraint function:

In the formula: N _pv , N _wp - the total number of photovoltaic and wind power generators in the area, L - the total number of power users in the area,

—The unit cost of purchasing electricity from the external grid at time t,

—The cost of purchasing electricity from PV and wind power generators at time t,

- Purchase electricity from the external power grid at time t,

— Purchase electricity from photovoltaic and wind power generators at time t,

—The load amount of the ith power user at time t, comp _pv , comp _wp — The subscription scope of the user is the subscription compensation for each kilowatt-hour of electricity paid in photovoltaic and wind power renewable energy, Q _load-pv-i , Q _load-wp-i —The amount of photovoltaic and wind power subscription electricity payable under the settlement of the user on the day, Q _pv , Q _wp , Q _grid — the photovoltaic, wind power, and external electricity consumed in the region on the day, υ _pv , υ _wp — the photovoltaic and wind power generation in the region on the day Proportion, α _i , β _i - the proportion of photovoltaic and wind power subscribed by the i-th user,

- The maximum power generation at time t declared by PV and wind power generators. 2. A distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning according to claim 1, characterized in that: in the step 2), multiple agents are used to process upper-level planning respectively The randomness problem of modeling and lower-level planning modeling itself and the mutual iteration of upper-level planning modeling and lower-level planning modeling; the two-layer collaborative reinforcement learning algorithm introduces a diffusion strategy in the reinforcement learning process, and the adaptive combination ATC mechanism is introduced into reinforcement learning. In the algorithm, the two-layer collaborative reinforcement learning algorithm can adapt to the randomness and uncertainty caused by distributed renewable energy, and can adapt to the complex calculation problem of the two-layer random decision optimization model; in order to avoid the storage of a large number of Q-value tables. , using a function approximator to record Q-values for complex continuous state and action spaces. 3. A distributed renewable energy transaction decision-making method based on multi-agent double-layer collaborative reinforcement learning according to claim 2, characterized in that: the diffusion strategy can achieve faster convergence, and can achieve more consistent lower mean squared deviation of the strategy, the diffusion strategy is as follows:

in,

is an intermediate term introduced by the diffusion strategy, _xi (k+1) is the state updated by combining all intermediate terms of agent _i ; _Ni is the set of points adjacent to agent i; The weight assigned by the body i to the adjacent agent j; here, a matrix B=[b _ij ]∈R ^n×n is defined as the topology matrix of the microgrid communication network; the topology matrix B is a random matrix, B1 _n =1 _n , where 1 _n ∈ R ⁿ is the unit vector.

Images