CN115759604A - An Optimal Scheduling Method for Integrated Energy System - Google Patents

Abstract

The invention discloses an optimized scheduling method of a comprehensive energy system, belonging to the technical field of algorithm optimized scheduling and comprising the following steps: constructing an integrated energy system, and obtaining a scheduling model based on the integrated energy system and a reinforcement learning algorithm, wherein the scheduling model comprises an intelligent agent and an environment; and correcting a Q value function in the reinforcement learning algorithm based on advantage learning to obtain a comprehensive algorithm, and training the intelligent agent based on the comprehensive algorithm to obtain an optimized scheduling strategy. The invention utilizes the dominant learning value function theory framework to combine with the SAC algorithm, improves the dominant learning value function theory framework and realizes the optimized dispatching of the comprehensive energy system by taking low carbon and economy as targets.

Description

An Optimal Scheduling Method for Integrated Energy System

technical field

The invention belongs to the technical field of algorithm optimization scheduling, and in particular relates to an optimal scheduling method of an integrated energy system.

Background technique

As an emerging energy management model, the integrated energy system aims to use advanced communication and control technologies to realize the efficient application of various energy sources, which is conducive to improving energy utilization efficiency and increasing the proportion of renewable energy consumption.

In the existing technology, for the optimal scheduling of integrated energy systems, deep reinforcement learning (deepreinforce-ment learning, DRL) is often used as an effective means to deal with sequential decision-making problems, but in the optimal scheduling of integrated energy systems, DRL based on policy gradient There are two difficulties in optimal scheduling: one is the problem of overestimation. The greedy thought of the algorithm will overestimate the Q value corresponding to some non-optimal actions, disturbing the generation of scheduling strategies, resulting in wrong judgments and generalization in the new environment. Reduced capacity. Second, the convergence speed of algorithm training is slow. The agent needs to obtain more data samples in new scenes to improve its scheduling strategy, but it needs to re-collect samples every time the strategy is improved, so the sample utilization efficiency is low, which reduces the learning efficiency of the agent, and with the new With the addition of training samples, the convergence speed of DRL will be slower.

Contents of the invention

The purpose of the present invention is to provide an integrated energy system optimization scheduling method to solve the problems of overestimation and slow convergence during training in the above-mentioned prior art.

In order to achieve the above purpose, the present invention provides a comprehensive energy system optimization scheduling method, including:

Construct an integrated energy system, and obtain a scheduling model based on the integrated energy system and a reinforcement learning algorithm, wherein the scheduling model includes an agent and an environment;

Based on advantage learning, the Q value function in the reinforcement learning algorithm is modified to obtain a comprehensive algorithm, and the agent is trained based on the comprehensive algorithm to obtain an optimal scheduling strategy.

Preferably, the integrated energy system runs several equipment models through grid connection, wherein the equipment models include: hydrogen energy storage model, electric energy storage model, cogeneration model, electric heating boiler model, gas boiler model and heat exchange device Model.

Preferably, the process of obtaining the scheduling model includes:

The constraint balance in the integrated energy system is obtained, and based on the constraint balance, a scheduling model is constructed through a reinforcement learning algorithm, wherein the constraint balance includes: power grid balance, heat network balance and gas network balance.

Preferably, the reinforcement learning algorithm includes: algorithm iteration and algorithm parameter update.

Preferably, the process of obtaining the comprehensive algorithm includes:

Obtain the Q value network loss function in the algorithm iteration, calculate the rate of decline of the loss function, judge the start of advantage learning based on the rate of decline, and modify the Q value function in the reinforcement learning algorithm based on the judgment result, and finally obtain the comprehensive algorithm.

Preferably, the Q-value function is a function between state parameters and action parameters in the integrated energy system at time t.

Preferably, it also includes the process of combining the comprehensive algorithm with transfer learning:

Based on the comprehensive algorithm, the scheduling knowledge is obtained, and the scheduling knowledge is migrated to the target task; based on the migration result, the scheduling strategy is fine-tuned to obtain an optimized scheduling strategy.

Preferably, the process of transferring the scheduling knowledge to the target task includes:

Based on the scheduling knowledge, the parameters of the deep neural network are transferred, and at the same time, the environment of the target task is judged through the k-means clustering algorithm, and the scheduling knowledge is transferred to the target task based on the judgment result.

Technical effect of the present invention is:

The present invention constructs an integrated energy system, and obtains a scheduling model based on the integrated energy system and a reinforcement learning algorithm, wherein the scheduling model includes an agent and an environment; based on advantage learning, the Q value function in the reinforcement learning algorithm is corrected to obtain a comprehensive Algorithm, training the agent based on the comprehensive algorithm to obtain an optimal scheduling strategy.

The present invention combines the theoretical framework of the advantageous learning value function with the SAC algorithm and improves it to realize the optimal scheduling of the comprehensive energy system with the goal of low carbon and economy. In this method, the maximum entropy mechanism of SAC makes the optimal scheduling of the comprehensive energy system more robust Stickiness, combined with the idea of advantage learning, reduces the overestimation of the value of non-optimal actions by the Q network, reduces the misselection of non-optimal actions by the agent, improves the generalization ability, and adds neural network stability to the algorithm Judgment to decide whether to start advantage learning, to prevent advantage learning from interfering with the previous iteration of neural network parameters.

Description of drawings

The drawings constituting a part of the application are used to provide further understanding of the application, and the schematic embodiments and descriptions of the application are used to explain the application, and do not constitute an improper limitation to the application. In the attached picture:

Fig. 1 is the comprehensive energy system diagram in the embodiment of the present invention;

Fig. 2 is a Markov decision process diagram in an embodiment of the present invention;

Fig. 3 is the ALSAC algorithm flowchart in the embodiment of the present invention;

Fig. 4 is an optimized scheduling diagram of an integrated energy system based on ALSAC algorithm and transfer learning in an embodiment of the present invention;

Fig. 5 is the K-Means cluster diagram of the historical data in the embodiment of the present invention;

Fig. 6 is a curve diagram of wind power generation power and electric heating load in the test scene in the embodiment of the present invention;

Fig. 7 is a comparison diagram of the convergence process of the reward function value R in the embodiment of the present invention;

Fig. 8 is a schematic diagram of the results of optimal scheduling of the heating network in the embodiment of the present invention;

FIG. 9 is a schematic diagram of an optimized scheduling result in an embodiment of the present invention;

Fig. 10 is a schematic diagram of the algorithm training process of scene 1 in the embodiment of the present invention;

Fig. 11 is a schematic diagram of wind power generation power and electric heating load curves in scene 4 in the embodiment of the present invention;

Fig. 12 is a schematic diagram of the SAC optimization result combined with transfer learning in the embodiment of the present invention;

Fig. 13 is a schematic diagram of ALSAC optimization results combined with transfer learning in an embodiment of the present invention;

Fig. 14 is a schematic diagram of an ALSAC optimization result without transfer learning in an embodiment of the present invention.

Detailed ways

It should be noted that, in the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and embodiments.

It should be noted that the steps shown in the flowcharts of the accompanying drawings may be performed in a computer system, such as a set of computer-executable instructions, and that although a logical order is shown in the flowcharts, in some cases, The steps shown or described may be performed in an order different than here.

Embodiment one

In this embodiment, a method for optimal dispatching of an integrated energy system is provided, including:

1. Composition and equipment model of gas-electricity-heat comprehensive energy system

The integrated energy system scheduling model constructed in this embodiment adopts grid-connected operation, and the given structure is shown in Fig. 1 .

The main components include:

1.1 Hydrogen energy storage model

The hydrogen production model adopts proton exchange membrane water hydrogen production equipment, and uses solid polymer water electrolysis to produce hydrogen. The hydrogen production capacity and the hydrogen storage capacity of the hydrogen storage tank are as follows:

V _HES (t) = P _HES (t) η _HES (1)

V _HSOC (t)＝V _HSOC (t-1)+V _HES (t)η _t -V _HOUT (t)η _HOUT (2)

Wherein V _HES (t) is the hydrogen volume produced by electrolysis in the t period; _PHES (t) is the electric power consumed in the t period; η _HES , η _t , η _HOUT are electrolysis efficiency, hydrogen storage tank hydrogen storage efficiency and output efficiency; V _HSOC (t) is the hydrogen storage capacity of the hydrogen storage tank during the t period; V _HOUT (t) is the hydrogen output volume of the hydrogen storage tank during the t period. The constraints on the hydrogen output of the electrolytic cell are:

V _{HES, min} < V _HES (t) < V _{HES, max} (3)

In the formula, V _HES,max and V _HES,min are the upper and lower limits of the hydrogen production of the electrolytic cell in the period t, respectively.

Use the ratio of the current hydrogen storage capacity to the maximum storage capacity to indicate the energy storage status of the hydrogen storage tank:

In the formula, SOC _h (t) is the energy storage state of the hydrogen storage tank, and V _h,max is the maximum storage capacity of the hydrogen storage tank. Hydrogen storage tank constraints:

SOC _h,min <SOC _h (t)<SOC _h,max (5)

V _{HOUT, min} < V _HOUT (t) < V _{HOUT, max} (6)

In the formula, SOC _h,max and SOC _h,min are the upper and lower limits of the hydrogen energy storage state, and V _HOUT,max and V _HOUT,min are the upper and lower limits of the hydrogen energy storage output during the period t.

The hydrogen volume V _HOUT (t) output by the hydrogen storage tank during the t period is used for daily industrial hydrogen demand and natural gas pipeline hydrogen mixing transportation:

V _HOUT (t) = V _HDE (t) + V _H,in (t) (7)

In the formula, V _HDE (t) is the volume of industrial hydrogen demand in the period t; V _H,in (t) is the volume of hydrogen mixed in natural gas pipelines in the period t.

1.2 Electric energy storage model

The electric energy storage model of this embodiment is composed of accumulators. The battery state of charge formula is as follows:

In the formula, SOC _e (t) represents the state of charge of the battery at time t, P _soc,in (t) and P _soc,out (t) represent the charging and discharging power of the battery during t period; W _soc is the maximum capacity of the battery; η _e is the charging and discharging efficiency, and Δt is the time interval. In order to prolong the life of the battery, the constraints are stipulated as follows:

SOC _e,min <SOC _e (t)<SOC _e,max (9)

0＜P _soc,in (t)＜P _soc,inmax (10)

0＜P _{soc, out} (t)＜P _{soc, outmax} (11)

In the formula, SOC _e,max and SOC _e,min are the upper and lower limits of the state of charge of the energy storage; P _soc,inmax and P _soc,outmax are the maximum charging and discharging power of the energy storage.

1.3 Combined heat and power model

The combined heat and power unit includes a gas turbine and a waste heat recovery boiler. The gas turbine generates electricity through the consumption of natural gas, and also produces flue gas with heat energy to output heat power. Gas turbine power generation:

P _GT (t) = V _GT (t) q _NG η _GT (12)

In the formula: _PGT (t) is the power generation power of the gas-fired boiler during the t period; V _GT (t) is the natural gas consumption per unit time of cogeneration during the t period; q _NG is the low calorific value of natural gas; η _GT is the power generation efficiency of the gas turbine . The power generation of the gas turbine satisfies the constraints:

P _GT,min <P _GT (t)<P _GT,max (13)

In the formula, PGT _,max and PGT _,min are the upper and lower limits of the power generation of the gas turbine during the period t. The mathematical expression of the thermal power generated by the gas turbine:

Q _GT (t) = V _GT (t)q _NG (1-η _GT ) (14)

Where Q _GT (t) is the output thermal power of the waste heat recovery boiler during the t period.

The thermal power constraint of the gas turbine is:

Q _GT,min <Q _GT (t)<Q _GT,max (15)

In the formula, Q _GT,max and Q _GT,min are the upper and lower limits of the output thermal power of the gas turbine during the period t, respectively.

The waste heat recovery boiler will collect the heat in the flue gas emitted by the gas turbine and supply it to the heat network. Its output thermal power is:

Q _HRSG (t) = Q _GT (t) η _HRSG (16)

In the formula: Q _HRSG (t) is the output thermal power of the waste heat recovery boiler during the t period; Q _GT (t) is the output thermal power of the gas turbine during the t period; η _HRSG is the heat transfer efficiency of the waste heat boiler. The upper and lower limits of the heat output power of the waste heat recovery boiler are:

Q _{HRSG, min} < Q _HRSG (t) < Q _{HRSG, max} (17)

In the formula, Q _HRSG,max and Q _HRSG,min are the upper and lower limits of the output power of the waste heat recovery boiler in the period t, respectively.

1.4 Electric boiler model

The electric boiler can convert the electric energy converted from clean energy into heat energy without natural gas combustion, which greatly reduces carbon emissions and improves the consumption of clean energy. The mathematical expression of heat emission is:

Q _EB (t) = P _EB (t)η _EB (18)

In the formula, P _EB (t) and Q _EB (t) are the electricity consumption and heating power of the electric boiler during the period t, respectively; η _EB is the electric-thermal conversion efficiency of the electric boiler. The thermal power of the electric boiler satisfies the constraints:

Q _EB,min <Q _EB (t)<Q _EBmax (19)

In the formula, Q _EBmax and Q _EB,min are the upper and lower limits of the output power of the electric heating boiler in the period t respectively.

1.5 Gas boiler model

A gas boiler is a device that uses natural gas to generate heat in an integrated energy system, and its thermal power output is:

Q _SB (t) = V _SB (t) q _NG η _SB (20)

In the formula: Q _SB (t) is the thermal rate of the gas boiler output during the t period; V _SB (t) is the natural gas consumption of the gas boiler during the t period; η _SB is the efficiency of the gas boiler. Q _SB (t) satisfies the constraints:

Q _SB,min <Q _SB (t)<Q _SBmax (21)

In the formula, _{QSBmax and QSB,min} are the upper and lower limits of the output power of the gas-fired boiler during the period t, respectively.

1.6 Heat exchange device model

The heat exchange device can convert the heat energy delivered by the waste heat recovery boiler, electric boiler and gas boiler to supply the heat load demand. The formula of its output heat power is:

Q _HE (t) = Q _HE,in (t)η _HE (22)

In the formula, Q _HE (t) is the output thermal power of the heat exchange device during the t period; Q _HE,in (t) is the thermal power input of the heating network during the t period; η _HE is the thermal energy conversion efficiency. The output heat power constraints of the heat exchange device are:

Q _HE,min <Q _HE (t)<Q _HE,max (23)

In the formula, Q _HE,max and Q _HE,min are the upper and lower limits of the output power of the heat exchange device in the period t, respectively.

1.7 Constraints

According to the energy structure composition of the integrated energy system, the constraint balance is as follows:

Grid balance equation:

L _E (t)+P _HES (t)+P _soc,in (t)+P _EB (t)＝P _GT (t)+P _soc,out (t)+P _G (t)+P _solar (t )+P _wind (t)(24)

In the formula: _PG (t), P _solar (t), and P _wind (t) are the electric power flowing into the integrated energy system from the grid ( _PG (t) is a negative value when the power generated by the integrated energy system flows into the grid), photovoltaic Generating power, wind power generating power; L _E (t) is the electric load power.

Heat network balance equation:

Q _HRSG (t)+Q _EB (t)+Q _SB (t)＝L _Q (t)/η _HE (25)

Where L _Q (t) is the heat load.

Air network balance equation:

V _HOUT (t) + V _GT (t) + V _SB (t) + V _RES (t) = V _H (t) (26)

In the formula, V _RES (t) is the residential gas consumption during the t period. V _H (t) is the output of natural gas during the t period. In order to meet the full-load operation of the gas-consuming unit, the output limit of V _H (t) in the period t of the natural gas pipeline is:

0＜V _H (t)＜V _H,max (27)

In the formula, V _H,max is the upper limit of the output gas of the natural gas pipeline in the period t.

According to the experience of existing international projects, the volume fraction of hydrogen mixed with natural gas can reach up to 20%. Under the condition of considering the thermal efficiency of gas, 12T-0 is used as the base gas for blending, and 5% hydrogen doping ratio is selected. The Wobbe number and calorific value of the mixed gas are better than other ratios, and the gas quality meets the national standard GB17820 - In 2012, the technical index of first-class natural gas high calorific value not less than 36.0MJ/m3. In this embodiment, the constraint conditions for the total amount of hydrogen transported from the hydrogen storage tank to the natural gas pipeline during the period t are:

0＜V _H,in (t)＜5.26％V _H,max (28)

2. SAC algorithm principle

2.1 Reinforcement Learning

Reinforcement learning is based on the Markov decision process (markov decision process, MDP), that is, the agent makes the action of the next environment based on the current environment information and obtains rewards, and the process of making the agent obtain the maximum reward through continuous "trial and error" .

As shown in Figure 2, an agent refers to a controller based on a certain control algorithm. The model of the Markov decision process is generally expressed as a tuple (S, A, P, R), where: S is the state set, A is the action set, P is the state transition probability, and R is the reward and punishment function.

2.2 SAC algorithm

When the model of the problem to be solved is unknown and there are various types of environmental information, resulting in a state space dimension too high, reinforcement learning will not be applicable. In order to allow reinforcement learning to deal with high-dimensional events, deep learning (deep learning, DL) is introduced, and the combination of the two becomes DRL.

The SAC algorithm is a reinforcement learning algorithm proposed by Harrnoja et al. Compared with other DRL algorithms based on policy gradients such as PPO, Actor-Critic multithreaded exploration (actor-criticalgorithm, A3C) and DDPG, the maximum entropy encouragement mechanism introduced by it is , which improves the robustness of the algorithm, and can explore better scheduling strategies in complex power environments.

2.2.1 SAC maximum entropy

Entropy is defined as the expectation of the amount of information, and it is a measure of the uncertainty of random variables. The greater the uncertainty of the event, the greater the entropy.

H(X) is the entropy value, and P( _xi ) is the event probability. An excellent DRL can explore the environment as much as possible to obtain the optimal strategy, instead of being greedy for an action with the largest reward and falling into a local optimum. When an action is repeatedly selected, the entropy will decrease. Using the maximum entropy mechanism, SAC will choose other actions, increasing the scope of exploration. In an environment state, more scheduling strategies and accompanying probabilities can be explored, increasing the system robustness.

In SAC, the reward value and policy entropy are added to the objective function, and the policy π is required to not only increase the final reward value, but also maximize the entropy. Accordingly, the objective function J(π) is constructed as follows:

为期望函数，π为策略，S_t和a_t为t时刻综合能源系统状态空间和动作空间；r(s_t,a_t)为t时刻奖励函数。(s_q,a_q)～P_π为策略π状态动作轨迹；α为熵温度项，决定熵对于奖励的影响程度。αH(π(·|s_t))为状态s_t时的熵，参考式(29)可得出其表达式：In the formula:

is the expectation function, π is the strategy, S _t and a _t are the state space and action space of the integrated energy system at time t; r(s _t , a _t ) is the reward function at time t. (s _q , a _q )～P _π is the strategy π state-action trajectory; α is the entropy temperature item, which determines the degree of influence of entropy on rewards. αH(π(·|s _t )) is the entropy of state s _t , its expression can be obtained by referring to formula (29):

In the formula, P(π(a _t |s _t )) is the probability that a _t is used as the action space when t.

2.2.2 SAC iteration method

In reinforcement learning, the value function Q(s _t , a _t ) is described as shown in formula (32), which is used for the evaluation of the strategy value of SAC; the policy update uses the Bellman operator as shown in formula (33).

In the formula, T ^π is the Bellman operator under the strategy π; γ is the discount factor of the reward; V(s _t+1 ) is the value function of the state s _t+1 , and the calculation method is:

Combined with the Bellman operator at the same time, we have

Q ^k+1 = T ^π Q ^k (35)

In the formula, Q ^k is the value function of the kth calculation. The soft policy evaluation can be iterated through formula (35), and finally Q will converge to the soft Q value function under the fixed policy π.

2.2.3 SAC Policy Update

The strategy is output as a Gaussian distribution, and the gap between the two distributions is minimized by minimizing the KL divergence.

为旧策略π_old下的值函数；

为旧策略下的分配函数，为对Q值进行归一化分布。where D _KL is the KL divergence (KL divergence); Π is the strategy set;

is the value function under the old strategy π _old ;

is the allocation function under the old strategy, and is the normalized distribution of the Q value.

2.2.4 SAC parameter update

The SAC algorithm is an Actor-Critic algorithm, Actor models the strategy, and Critic models the Q-value function. Two neural networks are used to fit the Q-value function and the strategy function respectively. The neural network parameter update strategy of the Q-value function is shown in formula (37), and the policy function parameter update strategy is shown in formula (38).

和Q_θ为更新后的函数。In the formula, θ and φ are the Q value network and strategy network parameters;

and Q _θ are the updated functions.

The action entropy is also output in the policy network, where the update of the temperature parameter α is crucial to the entropy, and its update is shown in (39):

The neuron activation function of the SAC in this embodiment selects a linear correction function (rectified linear unit, ReLU)

f(x)=max(0,x) (40)

The output layer selects the tanh function in the range [-1,1]. For the convenience of scheduling, the value of action a _t is attributed to [0,1].

3. SAC-based multi-energy system optimization scheduling scheme

3.1 State space

In the multi-energy system environment of this embodiment, the information provided by the environment to the agent generally includes: wind energy, light energy, main network time-of-use electricity price, micro-grid time-of-use electricity price, electric load, heat load, electric energy storage situation, hydrogen storage situation, time.

Then the state space is:

S(t)＝[L _E (t), L _Q (t), P _solar (t), P _wind (t), Γ _PG (t), Γ _DG (t), SOC _h (t), SOC _e (t),t] (41)

In the formula, Γ _PG (t) is the time-of-use electricity price of the power grid during the t period; Γ _DG (t) is the time-of-use electricity price of the integrated energy system during the t period.

3.2 Action space

After the agent obtains state information from the environment, it chooses an action in the action space according to its own strategy π. The power equipment model in the integrated energy system is relatively complex, and there are many types of energy storage and energy conversion equipment. In order to simplify the action space, the actions of the two energy storage equipment are transformed into two actions of ACT ₁ and ACT ₂ , and the formula (12 ) and formula (14), it can be seen that there is a coupling relationship between the electricity and heat of cogeneration, and the output power of gas-fired boilers can be obtained according to the heat network balance equation (25). The output power of the output power and the action space are as follows:

A(t)＝[P _GT (t), P _EB (t), ACT ₁ , ACT ₂ ] (42)

In the formula, ACT ₁ and ACT ₂ are the two actions of excessive and insufficient renewable energy. When the renewable energy is excessive, the priority is to meet the charging of electric energy storage, and the electrolysis of water releases hydrogen. When the renewable energy is insufficient, compare the electricity price to check whether to start the energy storage discharge.

3.3 Reward function

The reward function is a quantification of the target task, which can guide the agent to optimize towards the target. The reward function of the integrated energy system in this embodiment mainly comes from operating costs, energy sales income, carbon emissions, and strategic reward and punishment constants. The source of operating costs is the power purchase cost, gas purchase cost and maintenance cost of the integrated energy system; the income from energy sales comes from the sale of electricity, thermal energy and hydrogen energy in the integrated energy system. Considering the small scale of the integrated energy system, the cost of heat-electricity-gas network loss and the cost of starting and stopping equipment can be ignored. The operating cost C ₁ (t) in period t is:

C ₁ (t) = C _e (t) + C _f (t) + C _ME (t) (43)

In the formula, C _e (t) is the power purchase cost of the grid during the t period; C _f (t) is the gas cost during the t period; C _ME (t) is the maintenance cost during the t period. Among them, the power purchase cost C _e (t) in period t is defined as:

In the formula, _PG (t) is the purchased power within t period, and Δt is the time interval. The cost of purchasing natural gas is:

C _f (t) = c _f (V _GT (t) + V _SB (t)) (45)

where c _f is the price of natural gas; V _GT (t) and V _SB (t) are the gas consumption of cogeneration and gas boilers in the time period t; the maintenance cost is:

In the formula, C _ME (t) is the maintenance cost of the t period; C _mi is the maintenance cost coefficient of the i-th unit; P _i (t) is the output power of the unit i in the t period. Energy sales income includes comprehensive energy system electricity, thermal energy and electricity energy storage and hydrogen energy storage surplus energy sales income:

In the formula, C ₂ (t) is the energy sales revenue of the integrated energy system during the t period; L _E (t) and L _Q (t) are the power consumption of the integrated energy system’s electric load and heat load during the t period; Γ _Q (t), Γ _h (t) is the heat power and hydrogen price in time period t.

Carbon emissions come from the combustion of natural gas and the consumption of coal electricity on the main grid. According to the national "double carbon" construction goal, it is estimated that by 2060, the proportion of wind, light and other new energy power generation in my country will reach 65%. In this embodiment, 1 kilowatt-hour of electricity will emit 0.45 CO ² , and 1 m ³ of natural gas will produce 1.9 kg of CO ² . The carbon emission formula is as follows:

In the formula, V _GT (t) and V _SB (t) are the amount of natural gas used by cogeneration and gas-fired boilers in the period t. The emergence of strategy reward constants reduces the number of actions beyond the limited range during exploration, increases the number of correct actions of the strategy, and speeds up the convergence of the algorithm. The penalty constants D ₁ (t) and D ₂ (t) in the time period t are given for the supply of natural gas beyond the limit of the gas network pipeline, the imbalance of heat and power buses, and t for actions to reduce carbon emissions and increase profits Reward constant D ₃ (t) over time period. The reward constant C ₄ (t) in the time period t is:

C ₄ (t) = D ₁ (t) + D ₂ (t) + D ₃ (t) (49)

The optimal scheduling in this embodiment targets economy and carbon emissions. From the above formula, the reward function for period t can be obtained as:

R(t)=α(C ₂ (t)-C ₁ (t))-(1-α)C ₃ (t)+C ₄ (t) (50)

Since the reinforcement learning will randomly explore other actions during training, resulting in large fluctuations in R, the reward value R(t) is scaled down here, and the sliding average is used to smooth the curve of the R value, which is conducive to the convergence of the observation algorithm Condition.

3.4 Objective function

Combined with the reward function, the objective function C of the integrated energy system is obtained as follows:

4. Combination method of SAC algorithm and advantage learning

In the agent learning process of DRL, the Q value fitted by the Q value neural network is not the real value, but only an estimate of the real Q value, and DRL will only choose the action with the largest Q value in the current state. The Q value of the optimal action may be overestimated, causing DRL to choose an action that is not optimal in this state, thus affecting the final result of the algorithm.

In 1999, Baird proposed the idea of advantage learning. This idea will reduce the Q value of the non-optimal action in the reinforcement learning Q-learning, thereby widening the gap with the Q value of the optimal action and reducing the non-optimal action. The overestimation of the Q value reduces the probability of the agent choosing an action incorrectly. The state-value function of advantage learning is defined as:

where A ^* (s,a) is the advantage function under state S and action A, which is defined as follows:

A ^* (s,a)＝V ^* (s)-α(V ^* (s)-Q ^* (s,a)) (53)

In the formula, α(V ^* (s)-Q ^* (s, a)) is a correction item. When Q ^* (s, a) is the Q value of the optimal action, its value is zero, and when it is a non-optimal action When the Q value of , its value is negative, the distance between the Q values of optimal and non-optimal actions is widened.

In order to add advantage learning to the DRL using the deep neural network, the correction function is changed. Using the SAC algorithm, the characteristics of a better strategy can be quickly obtained, and the current state is input into the strategy network, and the output action is regarded as the optimal action. , bring the action into the Q-value network, and the output Q-value is regarded as the current optimal state value Q(s _t ,at ₊₁ ; θ ^- ), and the correction term is formula (54).

F(s _t ,a _t )＝α(Q(s _t ,a _t+1 ;θ ^- )-Q(s _t ,a _t ;θ ^- )) (54)

However, the above method ignores the inaccurate estimation of the Q value of the action by the Q value network at the initial stage of algorithm training. If the Q value of the non-optimal action is greater than the Q value of the optimal action, widening the Q value gap at this time will interfere with the iteration of the algorithm. convergence. In order to solve the above problems, this embodiment uses the rate of decline of the loss function loss of the Q-value network to judge whether the Q-value network has the ability to start advantage learning, and the judgment of the rate of decline is as follows:

In formula (55), Loss(t) is the average value of the two Q-value network loss function values at time t, and k is its rate of decline. When the rate of decline reaches the specified threshold, the neural network has passed the period of unstable update of the previous parameters. , start advantage learning: when F(s _t ,a _t )>0, the value of F(st _t ,a _t ) remains unchanged; when F(s _t ,a _t )<0, F(s _t ,a t ) _t ) has a value of 0.

The target Q-value formula of the ALSAC median function estimation part:

The combination process of advantage learning and SAC algorithm is shown in Figure 3 below.

5. Combination method of ALSAC algorithm and transfer learning

Compared with other DRLs, ALSAC is more robust and can maximize the stability of the integrated energy system in the new environment. However, due to the incomplete historical data during the training of the agent, it is difficult to make the best decision for the new scene. best scheduling strategy. In order to obtain the optimal scheduling strategy, accelerate the training speed, and make full use of the existing scheduling knowledge, the parameter migration of transfer learning is introduced, and a comprehensive energy optimization scheduling method based on ALSAC and transfer learning is proposed. The process is shown in Figure 4.

In the actual application of this embodiment, the existing historical data is used to train the agent for the purpose of low carbon and economy, and after the training is completed, the weight value of the neural network of the ALSAC algorithm is stored, and the weight value is the accumulated scheduling knowledge . Through K-Means, the similarity between the environment of the target task and the historical environment of the source task is compared. If the similarity is too low, the environment of the target task is identified as a new environment. When encountering a new environment, the existing scheduling knowledge is transferred to the target task, and the parameters of the deep neural network are fine-tuned through the ALSAC algorithm to obtain the optimal scheduling strategy.

6. Example analysis

6.1 Simulation parameter setting of integrated energy system

See Table 1, Table 2, and Table 3 for various energy prices, time-of-use electricity prices, and various equipment parameters in this embodiment.

Table 1

Table 2

table 3

6.2 Training scene and test scene selection

The simulation data of this embodiment uses the historical data of the European Eliagroup company from November 2020 to November 2021. The data used is 358 in total, and each piece of data has 96 time periods, each time period is 15 minutes, and the data of each time period is the time period The wind and solar power generation and the electricity and thermal load power are added together.

K-Means is used to divide historical data into four categories, which are rhombus, regular triangle, inverted triangle and pentagram as shown in Figure 5. In order to prevent data leakage, diamonds are selected from the four categories, and the 12 data points closest to the diamond cluster center (solid circle) are used as training samples to optimize the scheduling strategy. In order to verify that the optimal scheduling strategy of the integrated energy system based on the ALSAC algorithm can improve the robustness of the system, 143, 192 and 172 points with relatively low similarity to the training set were randomly selected as scenarios 1, 2, and 3 (all solid graphics). The three scene data are shown in Figure 6 for hours 0-24, 24-48, and 48-72. The three scenarios have obvious output characteristics, which are windy and sunny, less windy and less sunny, and less windy and sunny weather. In this embodiment, three consecutive extreme and changeable scenarios are used as test scenarios to test the performance of the strategy proposed in this embodiment.

6.3 Algorithm Reward Function Value Comparison

The comparison of the convergence process of the reward function value R is shown in Figure 7.

The simulation experiment of the integrated energy system in this embodiment is built using Python3.7.3 software, the computer configuration is I5-1135G7, the graphics card is IrisXe Max, and the hyperparameters of the algorithm are shown in Table 4.

ALSAC, SAC, PPO, A3C, and DDPG algorithms are trained, and the convergence process of the reward R value is shown in Figure 8.

It can be seen from Figure 7 that among the five algorithms, ALSAC has the highest reward function value R, and its optimization effect is the best. In complex scenes, actions are explored in different ways, and the best strategies for their exploration will also be different. DDPG uses OU-noise, a method with many hyperparameters, to explore actions, and needs to manually add the noise variance of actions. PPO adds a changeable penalty item in the trust domain of action selection to change the action selection, and changes the noise variance artificially added by DDPG into a trainable parameter. A3C explores the environment in a multi-threaded manner and updates policies asynchronously. The action update parameters of PPO and SAC can be changed by training, and the hyperparameters used are less, and the algorithm is more robust. However, compared with PPO of online learning, SAC has a higher utilization rate of samples and can use small samples. Data learns better scheduling strategies. After adding advantage learning to SAC, it reduces the overestimation of the Q value of the non-optimal action by the Q-value network, reduces the misselection of the non-optimal action by the agent, and improves the convergence speed of the algorithm and the final optimization effect.

Table 4

6.4 Optimizing scheduling results

The dispatching equipment of the comprehensive energy system simulation model in this embodiment is electric boiler, cogeneration, electric energy storage and hydrogen energy storage, respectively using ALSAC, SAC, PPO, DDPG and A3C algorithms to optimize dispatch in scenarios 1, 2 and 3 robustness test. Since the above scheduling devices can view all calls in the heating network, the test in this section refers to the scheduling of heating network devices.

From the ALSAC scheduling optimization in Figure 8, we can see that:

1) During the flat and peak periods of electricity prices, when the total amount of wind and solar power generation is large, under the condition of satisfying the electric load, the electric energy storage starts to charge, the electrolyzed water starts to produce hydrogen, the hydrogen storage of hydrogen storage increases, and the electric boiler starts to operate , because the thermal load and electrical load are at their peak, the combined heat and power generation starts to operate. When the total amount of wind and solar power generation is small and cannot meet the electricity and heat loads, the electric energy storage will start to contribute. After three hours of power consumption, the hydrogen energy storage will start to output, and the hydrogen energy will be converted into electric energy and heat energy, reducing carbon emissions. Two energy storage They all played the role of cutting peaks and filling valleys. The electric boiler will not operate because the total amount of wind and solar power generation is too small and the electricity price is at the peak. Cogeneration produces heat and electricity due to high electricity prices and during periods of high load.

2) During the low electricity price period, due to the low electricity price, the profits of cogeneration and electric energy storage capacity are too low during this period and do not operate. When the solar energy production capacity is not much, the electric boiler starts to supply heat by purchasing electricity from the grid under the premise of considering the electricity price and carbon emissions. If the heat supply is insufficient, the gas-fired boiler starts to produce heat.

6.4.1 Comparison of optimization results:

From the perspective of optimization results, in the optimization results of PPO, the combined heat and power generation on the second day runs at full load during the low electricity price period, which reduces profits; in the optimization results of A3C, the electric boiler runs at full load on the third day when the solar energy production capacity is the lowest , so that the electricity purchased from the main network increases, resulting in a decrease in profits and an increase in carbon emissions. At the same time, starting cogeneration during the period of low electricity prices reduces profits; in the optimization results of DDPG, the electric boiler also runs at full load on the third day, and Cogeneration operates during periods of low electricity prices. The results of the three-day optimization are shown in Table 5, where the α value of the objective function C is 0.7. It can be seen from Table 5 that the optimization effect of the SAC-based optimal scheduling strategy in dealing with the new environment is better than that of the other three DRL optimal scheduling strategies.

From the perspective of power balance, PPO at 18-20 and 42-48 hours, A3C at 18-20 and 68-70 hours and DDPG at 18-20 and 68-72 hours all have the phenomenon that the heating power of the heating equipment is greater than the heat load , power imbalance. Compared with the obvious power imbalance of the heating network above, the optimal scheduling strategy based on SAC has a good performance in terms of power balance. In summary, the optimal scheduling strategy based on SAC has better robustness in the face of new scenarios than the other three optimal scheduling strategies.

table 5

Compared with the optimal dispatching results based on ALSAC, the optimal dispatching results based on ALSAC increased the use of electric boilers during the period of low electricity prices, and reduced the use of combined heat and power generation during the period of flat electricity prices, which made the integrated energy system reduce on the premise of maximizing profits. Carbon emissions, increased objective function value.

6.4.2 Comparison between ALSAC and traditional optimization methods

The optimized scheduling result is shown in Figure 9.

Table 6

The heuristic algorithm PSO and the traditional mixed integer programming are used to schedule and plan the test scenarios 1, 2, and 3. The mixed integer programming (MIQP) uses Yalmip and the solver Gurobi to construct and solve (the MIPGap parameter is set to 0.05) . The scheduling results are shown in Figure 9 and Table 6. Compared with the ALSAC algorithm, the profit of the PSO-based scheduling method is increased by 10.03%, the carbon emission is reduced by 28.44%, and the total objective function value is reduced by 430.915%. The effect is weaker than the ALSAC algorithm, and the algorithm falls into a local optimum; compared with ALSAC, the scheduling result of MIQP reduces profits by 1.5%, reduces carbon emissions by 3.17%, and increases the objective function by 6.25%. The optimization effect is slightly better than that of ALSAC for scenarios 1 and 2 , 3 online optimization scheduling, but the solution time is 245 times higher, and the solution efficiency is not high. When the scale of the integrated energy system increases, the requirements for the algorithm solution speed are also higher, and the solution rate of DRL can meet the online optimal scheduling of the integrated energy system.

After training a few days ago, the neural network parameters of ALSAC are determined. In the actual decision-making within the day, the action A of the dispatching equipment can be directly output by collecting the data of the state S, which reduces complex calculations. The above experiments show that the optimization results of ALSAC are not much different from the results of mixed integer programming, which greatly improves the solution speed of the optimal scheduling problem of the integrated energy system.

6.5 Analysis of Optimal Scheduling Result of Integrated Energy System Based on ALSAC and Transfer Learning

In order to verify that the introduction of transfer learning parameter transfer can improve the learning efficiency of the agent and the generalization ability to deal with new scenarios. The scheduling knowledge accumulated by the 13 training samples in Section 5.2 is transferred to the target task of the new scenario (Scenario 1), and the respective deep neural network parameters are fine-tuned through the ALSAC and SAC algorithms to obtain the optimal scheduling strategy. Randomly select a new scene (183 points in Figure 5) to test the generalization ability of scene 4. The data of scene 4 is shown in Fig. The optimal scheduling strategy without transfer learning (ALSAC) is optimized for scenario 4, and the results are shown in Figure 12, Figure 13, Figure 14 and Table 7. After ALSAC is combined with transfer learning, electric boilers are turned on during periods of low electricity prices and clean energy peaks, and cogeneration concentrates most of its operating time in periods of high electricity prices, reducing carbon emissions as much as possible while maximizing profits, with profits corresponding Compared with the migration-SAC algorithm and ALSAC algorithm, the reduction is 8.39%, 6.36%, the carbon emission is reduced by 6.79%, 14.33%, the final objective function value is increased by 18.87%, 38.86%, and the generalization ability of the algorithm has been improved. In order to reflect the improvement of learning efficiency, a comparison item is added, and the ALSAC algorithm without transfer learning is also trained with the data of scene 1 as the training sample. The training process and time of the three are shown in Figure 10 and Table 7. From Figure 10 and Table 7, it can be seen that the scheduling strategy combined with advantage learning and transfer learning has significantly faster convergence speed during training, and the initial optimization effect is better, which has higher learning efficiency. Since the optimization interval of the calculation example in this embodiment is 15 minutes, which is much longer than the convergence time of ALSAC combined with transfer learning, the real-time performance of its policy update can further satisfy the online optimization scheduling of the system.

Table 7

7. Conclusion

Whether energy can be safely and efficiently dispatched in the integrated energy system is the prerequisite for the operation of the integrated energy system. The energy multi-coupling of the integrated energy system and the uncertainty of renewable energy make its energy dispatching face many challenges. In this example, aiming at the generalization ability of the optimal scheduling strategy of the integrated energy system and the learning efficiency of the agent, the optimal scheduling of the integrated energy system based on the superior flexible strategy-evaluation algorithm and transfer learning is proposed, aiming at economy and low carbon , realizing the optimal scheduling of the integrated energy system. In this embodiment, after SAC is added to the advantage learning, changeable and extreme scenarios are selected through K-Means clustering, and compared with various DRL algorithms based on policy gradients, traditional particle swarm optimization and MIQP, it is verified that ALSAC has the advantages of comprehensive energy The optimal scheduling of the system has strong generalization ability and fast convergence speed. At the same time, the introduction of transfer learning further improves the learning efficiency of the agent and the generalization ability in the face of new scenarios, and realizes the flexible and efficient scheduling of the integrated energy system.

Beneficial effects of this embodiment:

This embodiment uses the theoretical framework of the advantage learning value function combined with the SAC algorithm, and improves it, and introduces the parameter migration of transfer learning, and proposes an integrated energy system optimization scheduling strategy based on the ALSAC algorithm and transfer learning, aiming at low carbon and economy. The optimal scheduling of the integrated energy system. The maximum entropy mechanism of SAC in this method makes the optimal scheduling of the integrated energy system more robust. After combining the idea of advantage learning, it reduces the overestimation of the value of the non-optimal action by the Q network and reduces the intelligence. Misselection of non-optimal actions by the body improves the generalization ability. At the same time, a neural network stability judgment is added to the algorithm to determine whether to start dominant learning to prevent dominant learning from interfering with the previous iteration of neural network parameters. Introduce the parameter migration of transfer learning, use the correlation of K-Means to judge whether the scene is a new scene, if it is a new scene, transfer the historical scheduling knowledge to the target task of the new scene, and then use the ALSAC algorithm to adjust the parameters of the deep neural network fine-tuning to obtain the optimal scheduling strategy.

The above is only a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in this application Replacement should be covered within the protection scope of this application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims (8)

1. An integrated energy system optimization dispatching method, is characterized in that, comprises the following steps: Constructing an integrated energy system, wherein the integrated energy system operates several equipment models through grid connection, and obtains a scheduling model based on the integrated energy system and a reinforcement learning algorithm, wherein the scheduling model includes an agent and an environment; Based on advantage learning, the Q value function in the reinforcement learning algorithm is modified to obtain a comprehensive algorithm, and the agent is trained based on the comprehensive algorithm to obtain an optimal scheduling strategy. 2. The integrated energy system optimization scheduling method according to claim 1, wherein the equipment model includes: a hydrogen energy storage model, an electric energy storage model, a combined heat and power model, an electric boiler model, a gas boiler model and a Thermal model. 3. The integrated energy system optimization scheduling method according to claim 1, wherein the process of obtaining the scheduling model comprises: The constraint balance in the integrated energy system is obtained, and based on the constraint balance, a scheduling model is constructed through a reinforcement learning algorithm, wherein the constraint balance includes: power grid balance, heat network balance and gas network balance. 4. The integrated energy system optimization scheduling method according to claim 1, wherein the reinforcement learning algorithm includes: algorithm iteration and algorithm parameter update. 5. The integrated energy system optimal scheduling method according to claim 4, wherein the process of obtaining the integrated algorithm comprises: Obtain the Q value network loss function in the algorithm iteration, calculate the rate of decline of the loss function, judge the start of advantage learning based on the rate of decline, and modify the Q value function in the reinforcement learning algorithm based on the judgment result, and finally obtain the comprehensive algorithm. 6. The integrated energy system optimal scheduling method according to claim 1, characterized in that the Q-value function is a function between state parameters and action parameters in the integrated energy system at time t. 7. The integrated energy system optimal scheduling method according to claim 1, further comprising a process of combining the integrated algorithm with transfer learning: Based on the comprehensive algorithm, the scheduling knowledge is obtained, and the scheduling knowledge is migrated to the target task; based on the migration result, the scheduling strategy is fine-tuned to obtain an optimized scheduling strategy. 8. The integrated energy system optimal scheduling method according to claim 7, wherein the process of transferring the scheduling knowledge to the target task comprises: Based on the scheduling knowledge, the parameters of the deep neural network are transferred, and at the same time, the environment of the target task is judged through the k-means clustering algorithm, and the scheduling knowledge is transferred to the target task based on the judgment result.

Images