CN115912367A - A method for intelligent generation of power system operation mode based on deep reinforcement learning - Google Patents

Abstract

The invention provides an intelligent generation method of an electric power system operation mode based on deep reinforcement learning, and relates to the technical field of power grid operation. According to the method, a Markov decision process MDP is used for carrying out reinforcement learning modeling on a power grid, and an improved mapping strategy of an intelligent action and an adjustable action object is established; intelligently generating a DQN network by constructing an operation mode, inputting the current system load flow state and the target operation state into the DQN network, and outputting the action with the maximum Q value; carrying out load flow calculation iteration by using a PQ decomposition method, if lambda is greater than 1 or the load flow is not converged after 10 times of iteration, considering that pathological load flow occurs, abandoning the action, and regenerating a new action by the DQN network; if the power flow is converged, adjusting the running state of the adjustable action object according to an improved mapping strategy; continuously adjusting the action until the load level of the target operation mode is met or the maximum action adjustment times are reached; and the intelligent generation and intelligent deletion of the power grid operation mode are completed by outputting the estimated Q network parameters.

Description

A method for intelligent generation of power system operation modes based on deep reinforcement learning

Technical Field

The present invention relates to the field of power grid operation technology, and in particular to a method for intelligently generating power system operation modes based on deep reinforcement learning.

Background Art

The calculation of the operation mode of the power system can provide the safe and stable operation boundary of the power grid. It is the overall guidance plan to ensure the safe and stable operation of the power grid, and it is also the theoretical basis for dispatchers to evaluate the real-time operation status of the power grid. Since all kinds of stability analysis of the power system need to be based on the results of power flow calculation, power flow calculation is an important basic content of the calculation of the operation mode of the power grid. In recent years, due to the rapid development of social economy, large-scale access to new energy and new power systems have not only increased the scale and complexity of the power grid to an unprecedented level, but also significantly increased the typical operation mode of the power grid. The operation mode calculation work faces severe challenges.

In actual projects, the annual operation mode calculation of large power grids is mainly completed by the mode calculation personnel of the control centers at all levels, based on the power system simulation analysis software, and involves a lot of manual participation. Specifically, it is necessary to preliminarily formulate typical operation modes under various extreme conditions based on the predicted results of the load and grid structure changes in the next year, and refer to the operation experience of the previous year. Then, the safe operation boundary of the power grid is determined by combining manual adjustment of the flow with stability calculation, providing a theoretical basis for the economic dispatch of the power grid, equipment maintenance plan and other work. On the one hand, the scale of the power grid is increasing, and the operation characteristics are becoming more and more complex; on the other hand, for a long time, the operation mode calculation has been mainly based on a large amount of manual labor, with a large workload and high repetitiveness.

At present, artificial intelligence technology is leading a new round of scientific and technological revolution and industrial transformation. With the development of artificial intelligence, deep reinforcement learning helps humans obtain the general laws of data based on a large number of sample training, greatly reducing the input of manpower and material resources. The intelligent generation of power system operation mode based on deep reinforcement learning is a process of generating power grid operation mode by machines instead of humans. While generating the operation mode, it diagnoses the rationality of the current operation mode, that is, whether it converges or generates pathological current flow. With the help of deep reinforcement learning, the high-dimensional space of the current flow can be intelligently adjusted, knowledge and experience can be added in the adjustment process, the action space can be reduced, and the process of manual adjustment can be simulated more effectively. It is of great significance to reduce the burden on staff, provide operators with a basis for flow adjustment, and improve the automation level of the power system.

The Chinese patent "CN111478331A A method and system for adjusting the convergence of power system power flow" proposes a method for calculating the operation mode of the power grid based on improved deep Q learning. This patent determines the input and output dimensions of the Q neural network model according to the state space and action space; determines the mapping relationship between the action space and the start and stop state of the power system generator, and adjusts the operation state of the power system generator according to the adjustment action output by the training model; takes the target load level and the start and stop state of the power system generator as input, and the adjustment action as output, and trains the Q neural network model according to the input and output dimensions; adjusts the power system power flow to the convergence state according to the adjustment action; and meets the load requirements under different operation modes by switching the generator on and off and adjusting the power of the balancing machine. This patent meets the load requirements under different operation modes by switching the generator on and off and adjusting the power of the balancing machine. The only action object that can be adjusted is the generator. In today's new power system with large-scale new energy access, the adjustable parameters include the line commissioning state, the new energy output state, the controllable load state, and the DC state, etc. In addition, the generator state in this patent has only two types: open and closed, which cannot meet the needs of adjusting the partial output of the unit in the actual power grid operation mode calculation.

Summary of the invention

In view of the shortcomings of the prior art, the present invention provides a method for intelligently generating an operation mode of an electric power system based on deep reinforcement learning.

A method for intelligently generating an operation mode of a power system based on deep reinforcement learning, specifically comprising the following steps:

Step 1: Use Markov decision process MDP to perform reinforcement learning modeling on the power grid;

Step 1.1: Use the Markov decision process to set parameters in the process of power grid operation;

The person calculating the power grid operation mode is set as an intelligent agent, the power grid operation data and power flow calculation formula are set as the environment, the result of the interaction between the intelligent agent and the environment is the convergence of the power grid power flow calculation, and the process of interaction between the intelligent agent and the environment is represented by the Markov decision process;

The Markov decision process MDP is composed of a 5-tuple (S, A, P _r , R, γ), where S is the system environment state space, s _t is the system state at time t; A is the action space, a _t is the agent action at time t; P _r is the transition probability, P _r (s _t+1 |s _t , a _t ) is the probability of transferring to state s _t+1 after taking action a _t in state _{s t} ; R is the reward function, r _t is the reward value obtained after taking action a _t in state s _t ; γ is a discount factor (0≤γ≤1), which is used to weigh the impact of immediate reward value and future reward value on the decision-making process;

In order to quantitatively describe the guiding role of action a t at time _t on the direction of system state transfer, the concept of state-action value function is introduced, that is, the expected value of the cumulative reward obtained after executing action a _t on state s _t at time t is expressed by Q(s, a). The specific calculation method is shown in formula (1).

Q(s,a)＝E[r _t +γr _t+1 +γ ² r _t+2 +...|s _t =s,a _t =a,π],s∈S,a∈A (1 )

In the formula, the larger the γ, the greater the impact of the future reward value on Q(s,a), γ = 1 means that the future reward value and the immediate reward value have the same impact on Q(s,a), and γ = 1 means that only the immediate reward value will affect Q(s,a); π represents the action execution strategy of the agent, that is, the mapping relationship between the system state s _t and the action a _t .

Calculate the optimal strategy μ ^* to maximize the Q value of action a _t at each moment. The formula is shown in (2):

μ ^* =maxQ _μ (s,a) (2)

Where Q _μ (s, a) is the expected return of strategy μ after taking action a starting from state s;

Step 1.2: Define the expressions of the system environment state space S, action space A, and reward function R in the Markov decision model;

In the system environment state space S, the state space s _t at time t is defined as:

s _t = [p, q, s, v, L, D, l] (3)

p＝[p ₁ ,p ₂ ,..., _pm ] (4)

q＝[q ₁ ,q ₂ ,...,q _m ] (5)

s＝[s ₁ ,s ₂ ,...,s _n ] (5)

v＝[v ₁ ,v ₂ ,...,v _g ] (6)

L＝[L ₁ ,L ₂ ,...,L _h ] (7)

D＝[D ₁ ,D ₂ ,...,D _k ] (8)

l＝[l ₁ ,l ₂ ,...,l _N ] (9)

Wherein, _pi is the active power of the generator at node i; _qi is the reactive power of the generator at node i; _si is the line operation status of node i; _vi is the new energy output of node i, _Li is the controllable load output of node i; _Di is the DC output of node i; m, n, g, h, k are the total number of adjustable generator nodes, total number of lines, total number of new energy nodes, total number of controllable load nodes and total number of DC nodes excluding balancing machines respectively; _l1 , _l2 , ..., _lN together form a binary code to represent the numbers of different operation modes;

In the action space A, the action space A is discrete, and the action space A is associated with a discrete positive integer, as shown in formula (10).

A＝[{1,2,...,m},{1,2,...,n},{1,2,...,g},{1,2,...,h} ,{1,2,...,k}](10)

The numbers in set A represent the numbers of the adjustable action objects, and the adjustment action at time t is represented by a _t ;

Four indicators of the power flow adjustment problem are defined: (1) the power flow calculation converges, represented by _c1 ; (2) the output power of the balancing machine does not exceed the limit, represented by _c2 ; (3) the network loss rate is lower than the set value, which is quantified by calculating the network loss rate; (4) no pathological power flow is generated, which is quantified by the λ value of the power flow calculation iteration; therefore, the reward function R is as follows:

After executing a _t , if the power flow calculation converges and the output power of the balancing machine does not exceed the limit, R is 0, and in other cases R is -1;

Step 2: Establish an improved mapping strategy between agent actions and adjustable action objects;

为平衡机的最大/最小有功功率；K为设定的目标网损率；P_i为发电机i的有功功率；P_imax为发电机i的最大有功功率，最小调整阈值为0.05P_imax；包括以下三种情况：The improved mapping strategy is: let _PG be the sum of the active power of the current power grid generators without the balancing machine; _PL be the sum of the active power of all the current loads in the power grid;

is the maximum/minimum active power of the balancing machine; K is the set target network loss rate; _Pi is the active power of generator i; _Pimax is the maximum active power of generator i, and the minimum adjustment threshold is 0.05P _imax ; including the following three cases:

时，a_t＝i，则令P_i＝0.5P_imax，若此时P_i≥0.5P_imax，则投运P_i与P_imax的中值向上取整，直到投运到P_imax为止。该场景下判断出系统发电机总有功功率不足，需要增加发电机有功功率，来满足潮流收敛要求。(1) When

When a _t = i, let _Pi = _0.5Pimax . If _Pi ≥ _0.5Pimax , the median of _Pi and _Pimax is rounded up until _Pimax is reached. In this scenario, it is determined that the total active power of the system generator is insufficient, and the active power of the generator needs to be increased to meet the power flow convergence requirements.

时，a_t＝i，则令P_i＝0.5P_imax，若此时P_i≤0.5P_imax，则P_i与停机出力的中值向下取整，直到投运到0％，即停机为止。该场景下判断出系统发电机总有功功率过大，需要减少发电机有功功率，来满足潮流收敛要求。(2) When

When a _t = i, let _Pi = _0.5Pimax . If _Pi ≤ _0.5Pimax , the median of _Pi and shutdown output is rounded down until it reaches 0%, i.e. shutdown. In this scenario, it is judged that the total active power of the system generator is too large, and the active power of the generator needs to be reduced to meet the power flow convergence requirements.

(3) Except for (1) and (2), a _t = i, if _Pi ≥ 0.5P _imax , the median of the operating _Pi and _Pimax is rounded up until the operation reaches _Pimax ; otherwise, the median of the operating _Pi and the shutdown output is rounded down until the operation reaches 0%, i.e., the shutdown is completed;

Step 3: Build and run the DQN network intelligently;

The DQN network combines the Q-learning network with the neural network, uses the neural network to estimate the Q value function, and after the neural network calculates the value function of each power flow adjustment action, it uses ε-greedy search to select the action and selects the action with the largest Q value as output.

Among them, the DQN network introduces the estimated Q network and the target Q network, and its training process is:

Step A1: At the beginning of training, the estimated Q network and the target Q network nodes, generators, lines, and load parameters are set to be the same, and the parameter matrices are θ and θ′ respectively;

Step A2: During the training process, the estimated Q network is updated once at each time step according to the gradient descent direction of the loss function as shown in formula (13). The DQN network calculates the Q value output flow adjustment action based on the estimated Q network and the current state;

Step A3: adjusting the running state of the adjustable action object according to step A2;

Step A4: every C steps, the estimated Q network parameters θ are passed to the target Q network θ′;

Step A5: The target Q network is updated once every C time steps according to the gradient descent direction of formula (13).

The power flow adjustment action value calculated by the estimated Q network is called the predicted value, and the sum of the instant reward in the current state and the state power flow adjustment action value calculated by the target Q network is called the true value. The parameters of the estimated Q network are updated by back propagation. During training, the above update process is repeated until the power flow converges and the output of the balancing machine does not exceed the limit or the number of iterations is reached;

Step 4: Model the operation mode intelligent deletion process and build a sick power flow diagnosis model;

If the power flow calculation cannot converge, it can be divided into the following two cases: the power flow calculation has no feasible solution, that is, the power flow has no solution; or the power flow calculation has a feasible solution, but it cannot be searched, that is, an ill-conditioned power flow problem;

The pathological power flow problem includes the following two situations:

(1) The pathological power flow caused by excessive cross-section power flow can be solved by adjusting the output of the adjustable action object to distribute the unbalanced active power;

(2) For pathological power flow caused by insufficient local reactive power support, the following power flow iteration index is defined to judge:

When the PQ decomposition method does not converge in power flow calculation, the index λ is used as the criterion, as shown in formula (14):

λ＝max{|[ΔU] ⁽³⁾ /[ΔU] ⁽²⁾ |} (14)

Where, [ΔU] ⁽³⁾ is the voltage increment of the third iteration, and [ΔU] ⁽²⁾ is the voltage increment of the second iteration;

When the power flow converges normally, λ<1; when the reactive power demand of the PQ node load increases, λ increases accordingly; when the power flow becomes pathological, λ>1.

After the operation mode is generated in step 3, the rationality of the operation mode is judged, and the PQ decomposition method is used to perform power flow calculation. When λ>1 or the power flow calculation iteration does not converge within 10 times, it is considered to be a pathological power flow phenomenon. Delete the operation mode, otherwise it is considered to be reasonable and retained, completing the intelligent deletion of the operation mode;

Step 5: Repeat steps 3 to 4 and continuously adjust the action until the target operation mode load level is met or the maximum number of action adjustments is reached;

Step 6: Output the estimated Q network parameter θ to complete the intelligent generation and intelligent deletion of power grid operation mode.

The beneficial effects of adopting the above technical solution are:

The present invention provides an intelligent generation method for the operation mode of an electric power system based on deep reinforcement learning, in which a computer replaces a person to complete the adjustment process of an adjustable object, and finally outputs an available operation mode, which greatly reduces the workload of operation mode calculation personnel. The adjustable action objects in the present invention include generator status, line commissioning status, new energy output status, controllable load status, and DC status, etc., which are more in line with the needs of operation mode calculation in an actual power grid. The actual needs of adjustment of new power system components can be well met by adjusting the action space, and the design of the improved mapping strategy also speeds up the training speed of the model, thereby reducing the computing power requirements. The output adjustment threshold of the action object in the present invention is 5% of the maximum power, which is more in line with the operation mode calculation adjustment requirements in an actual power grid. Different mapping strategies are designed for action objects of different actions. The increase in the number of action spaces can be well solved by improving the mapping strategy without causing a significant increase in the running time, thereby speeding up the DQN network training process.

Compared with the prior art, the technical solution proposed in the present invention adopts an intelligent generation method of operating modes based on deep reinforcement learning. Computers replace people to complete the adjustment process of adjustable objects and finally output available operating modes, which greatly reduces the workload of personnel who calculate the operating modes. By adjusting the action space, the actual needs of adjusting new power system components can be well met. The design of improved mapping strategies also speeds up the training speed of the model, thereby reducing computing power requirements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for intelligently generating an operation mode of a power system according to an embodiment of the present invention;

FIG2 is a schematic diagram of a mapping relationship between generator nodes in an IEEE-30 node system according to an embodiment of the present invention;

FIG3 is a flow chart of an algorithm for intelligently generating a DQN network in an operation mode according to an embodiment of the present invention.

DETAILED DESCRIPTION

The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

A method for intelligently generating power system operation modes based on deep reinforcement learning, as shown in FIG1 , specifically comprises the following steps:

Step 1: Use Markov decision process MDP to perform reinforcement learning modeling on the power grid;

The formulation of the grid operation mode is essentially a process of convergence and adjustment of the power system flow. This process can be regarded as a decision-making process in which an operation mode calculation personnel calculates and adjusts the grid data to obtain the system flow.

Step 1.1: Use the Markov decision process to set parameters in the process of power grid operation;

The person who calculates the operation mode of the power grid is set as an agent (Agent refers to a computing entity that resides in a certain environment, can continuously and autonomously play a role, and has the characteristics of residence, responsiveness, sociality, initiative, etc.), the power grid operation data and power flow calculation formula are set as the environment, and the result of the interaction between the agent and the environment is the convergence of the power grid power flow calculation, and the result of the typical operation mode is obtained. The process of interaction between the agent and the environment is represented by the Markov Decision Process (MDP);

The Markov decision process MDP is composed of a 5-tuple (S, A, P _r , R, γ), where S is the system environment state space, s _t is the system state at time t; A is the action space, a _t is the agent action at time t; P _r is the transition probability, P _r (s _t+1 |s _t , a _t ) is the probability of transferring to state s _t+1 after taking action a _t in state _{s t} ; R is the reward function, r _t is the reward value obtained after taking action a _t in state s _t ; γ is a discount factor (0≤γ≤1), which is used to weigh the impact of immediate reward value and future reward value on the decision-making process;

In order to quantitatively describe the guiding role of action a t at time _t on the direction of system state transfer, the concept of state-action value function is introduced, that is, the expected value of the cumulative reward obtained after executing action a _t on state s _t at time t is expressed by Q(s, a). The specific calculation method is shown in formula (1).

Q(s,a)＝E[r _t +γr _t+1 +γ ² r _t+2 +...|s _t =s,a _t =a,π],s∈S,a∈A (1 )

In the formula, the larger the γ, the greater the impact of the future reward value on Q(s,a), γ = 1 means that the future reward value and the immediate reward value have the same impact on Q(s,a), and γ = 1 means that only the immediate reward value will affect Q(s,a); π represents the action execution strategy of the agent, that is, the mapping relationship between the system state s _t and the action a _t .

Calculate the optimal strategy μ ^* to maximize the Q value of action a _t at each moment. The formula is shown in (2):

μ ^* =maxQ _μ (s,a) (2)

Where Q _μ (s, a) is the expected return of strategy μ after taking action a starting from state s;

Step 1.2: Define the expressions of the system environment state space S, action space A, and reward function R in the Markov decision model;

In the system environment state space S, the state space s _t at time t is defined as:

s _t = [p, q, s, v, L, D, l] (3)

p＝[p ₁ ,p ₂ ,..., _pm ] (4)

q＝[q ₁ ,q ₂ ,...,q _m ] (5)

s＝[s ₁ ,s ₂ ,...,s _n ] (5)

v＝[v ₁ ,v ₂ ,...,v _g ] (6)

L＝[L ₁ ,L ₂ ,...,L _h ] (7)

D＝[D ₁ ,D ₂ ,...,D _k ] (8)

l＝[l ₁ ,l ₂ ,...,l _N ] (9)

where _pi is the active power of the generator at node i; _qi is the reactive power of the generator at node i; _s is the line operation status of node i; _vi is the output of new energy at node i, and it is considered that the output of new energy conforms to the Weibull probability distribution; _Li is the controllable load output of node i, which is considered to be a negative generator output; _Di is the DC output of node i, which is considered to be a negative current source generator output at the sending end and a positive current source generator output at the receiving end; m, n, g, h, k are the total number of adjustable generator nodes, the total number of lines, the total number of new energy nodes, the total number of controllable load nodes, and the total number of DC nodes excluding balancing machines, respectively; l 1 ,l ₂ ,...,l _N together form a binary code to represent the numbers of different operating modes. For example, if there are 16 operating modes, then N=4, l ₁ ,l 2 ,l 3 ,l 4 ＝0000 represents the first operating mode, l ₁ ,l ₂ ,l ₃ ,l 4 ＝0000 represents the first operating mode, l ₁ ,l ₂ ,l ₃ ,l ₄ ＝0000 represents the first operating mode, ₄ = 1111 indicates the 16th operating mode.

In this embodiment, in order to simplify the model, the adjustment method of the adjustable action object by the calculation personnel is combined with the actual operation mode, and the generator, controllable load and DC output are simplified to a minimum threshold of one adjustment of 5% of the maximum output, and the adjustment range can only be an integer multiple of 5%. For example, the allowed value of p _i is [0, 0.05, 0.1, ..., 1.0], and p ₁ = 0.3 represents that the output of the generator at node number 1 is adjusted to 30% of the maximum active power at this time; the line operation status is simplified to only two 1/0 states, 1 for operation and 0 for shutdown; since the output of new energy has random volatility, its output obeys the Weibull probability distribution, and its output is simplified to the expected value of the Weibull distribution function, and there are only two 1/0 states, 1 for the expected value output state of the distribution function, and 0 for shutdown.

In the action space A, due to the simplified definition of the adjustment method of the adjustable action object, there are only 21 output adjustment methods for generators, controllable loads and DC nodes, namely 0, 5%, 10%, ..., 100%, and there are only two adjustment methods, 1 and 0, for the line commissioning status and new energy nodes. Therefore, the action space A is discrete. The action space A is associated with discrete positive integers, as shown in formula (10).

A＝[{1,2,...,m},{1,2,...,n},{1,2,...,g},{1,2,...,h} ,{1,2,...,k}](10)

The numbers in set A represent the numbers of the adjustable action objects. If the number is 0, it means that the action of the object is not adjusted. According to the actual situation of different node systems, the code is selectively output. The adjustment action at time t is represented by a _t ; a _t = [1, 3, 0, 0, 4] represents the adjustment of the operating status of generator node 1, line 3 and DC node 4.

In the reward function R, according to formula (2), the immediate reward r _t will affect the calculation result of Q _μ (s, a), and then Q _μ (s, a) will affect the selection of action a _t . The design idea of the reward function is as follows: when the agent chooses an action that can make the power flow converge, the environment will give a larger reward; when the action that makes the power flow diverge or the balancing machine exceeds the limit is selected, the environment will give a corresponding penalty value. In order to obtain the maximum reward, the agent will constrain the action to meet the action change rate. Define the four indicators in the power flow adjustment problem: (1) The power flow calculation converges, represented by c ₁ ; (2) The output power of the balancing machine does not exceed the limit, represented by c ₂ ; (3) The network loss rate is lower than the set value, which is quantified by calculating the network loss rate; (4) No pathological power flow is generated, which is quantified by the λ value of the power flow calculation iteration; Therefore, the reward function R is as follows (11):

After executing a _t , if the power flow calculation converges and the output power of the balancing machine does not exceed the limit, R is 0, and in other cases R is -1; therefore, the fewer the adjustment steps, the greater the cumulative reward.

Step 2: Establish an improved mapping strategy between agent actions and adjustable action objects;

In this embodiment, IEEE30 nodes are taken as an example, and the mapping relationship of generator nodes is shown in Figure 2. The usual mapping strategy is to correspond actions a _t to adjustable generators one by one, but the power grid has many generators, and with each additional generator, the state space will grow exponentially, and most of the state flow calculations will not converge. If the traversal action state search is used, the time required will also grow exponentially. In order to improve the search efficiency, the improved mapping strategy designed is as follows:

为平衡机的最大/最小有功功率；K为设定的目标网损率；P_i为发电机i的有功功率；P_imax为发电机i的最大有功功率，最小调整阈值为0.05P_imax；包括以下三种情况：The improved mapping strategy is: let _PG be the sum of the active power of the current power grid generators without the balancing machine; _PL be the sum of the active power of all the current loads in the power grid;

is the maximum/minimum active power of the balancing machine; K is the set target network loss rate; _Pi is the active power of generator i; _Pimax is the maximum active power of generator i, and the minimum adjustment threshold is 0.05P _imax ; including the following three cases:

时，a_t＝i，则令P_i＝0.5P_imax，若此时P_i≥0.5P_imax，则投运P_i与P_imax的中值向上取整，直到投运到P_imax为止。例如：a_t＝i且当前P_i＝0.75P_imax，则调整后的P_i＝[0.5*(75％+100％)]P_imax＝0.875P_imax→0.9P_imax；该场景下判断出系统发电机总有功功率不足，需要增加发电机有功功率，来满足潮流收敛要求。(2) When

When a _t = i, let _Pi = 0.5P _imax . If _Pi ≥ 0.5P _imax at this time, the median of _Pi and _Pimax is rounded up until _Pimax is reached. For example: a _t = i and the current _Pi = 0.75P _imax , then the adjusted _Pi = [0.5*(75%+100%)]P _imax = 0.875P _imax → 0.9P _imax ; In this scenario, it is determined that the total active power of the system generator is insufficient, and the active power of the generator needs to be increased to meet the power flow convergence requirements.

时，a_t＝i，则令P_i＝0.5P_imax，若此时P_i≤0.5P_imax，则P_i与停机出力的中值向下取整，直到投运到0％，即停机为止。例如：a_t＝i且当前P_i＝0.25P_imax，则调整后的P_i＝[0.5*(25％+0％)]P_imax＝0.125P_imax→0.1P_imax。该场景下判断出系统发电机总有功功率过大，需要减少发电机有功功率，来满足潮流收敛要求。(2) When

When a _t = i, let _Pi = 0.5P _imax . If _Pi ≤ 0.5P _imax at this time, the median of _Pi and shutdown output is rounded down until it reaches 0%, that is, shutdown. For example: a _t = i and the current _Pi = 0.25P _imax , then the adjusted _Pi = [0.5*(25%+0%)]P _imax = 0.125P _imax → 0.1P _imax . In this scenario, it is judged that the total active power of the system generator is too large, and the active power of the generator needs to be reduced to meet the power flow convergence requirements.

(3) Except for (1) and (2), a _t = i, if _Pi ≥ 0.5P _imax , the median of the operating _Pi and _Pimax is rounded up until the operation reaches _Pimax ; otherwise, the median of the operating _Pi and the shutdown output is rounded down until the operation reaches 0%, i.e., the shutdown is completed;

Replace the generator node with other adjustable action objects, and obtain the mapping strategies of other action objects by analogy. The mapping strategies can be adjusted according to the actual situation of the node system.

Step 3: Build and run the DQN network intelligently;

The algorithm flow chart of the intelligent generation of the DQN network in the operation mode is shown in Figure 3. The DQN network is improved on the basis of the Q-learning network, which combines the Q-learning network with the neural network, and uses the neural network to estimate the Q value function. After the neural network calculates the value function of each flow adjustment action, the ε-greedy search is used to select the action, and the action with the largest Q value is selected as the output.

Among them, the DQN network introduces the estimated Q network and the target Q network, and its training process is:

Step A1: At the beginning of training, the estimated Q network and the target Q network nodes, generators, lines, and load parameters are set to be the same, and the parameter matrices are θ and θ′ respectively;

Step A2: During the training process, the estimated Q network is updated once at each time step according to the gradient descent direction of the loss function as shown in formula (13). The DQN network calculates the Q value output flow adjustment action based on the estimated Q network and the current state;

Step A3: adjusting the running state of the adjustable action object according to step A2;

Step A4: every C steps, the estimated Q network parameters θ are passed to the target Q network θ′;

Step A5: The target Q network is updated once every C time steps according to the gradient descent direction of formula (13).

The power flow adjustment action value calculated by the estimated Q network is called the predicted value, and the sum of the instant reward in the current state and the state power flow adjustment action value calculated by the target Q network is called the true value. The parameters of the estimated Q network are updated by back propagation. During training, the above update process is repeated until the power flow converges and the output of the balancing machine does not exceed the limit or the number of iterations is reached;

Step 4: Model the operation mode intelligent deletion process and build a sick power flow diagnosis model;

If the power flow calculation cannot converge, it can be divided into the following two situations: the power flow calculation has no feasible solution, that is, the power flow has no solution; or the power flow calculation has a feasible solution, but it cannot be searched, that is, the ill-conditioned power flow problem. The characteristics of ill-conditioned power flow are: the converged solution of the power flow seriously deviates from the initial value, the number of iterations increases and the convergence speed is slow; or the Jacobian matrix tends to be singular, so that the power flow cannot converge to the feasible solution. There are two reasons for the occurrence of ill-conditioned power flow: ① The cross-section power flow is too heavy, that is, the active power is too large; ② The local reactive power support is insufficient.

The pathological power flow problem includes the following two situations:

(1) The pathological power flow caused by excessive cross-section power flow can be solved by adjusting the output of the adjustable action object to distribute the unbalanced active power;

(2) For pathological power flow caused by insufficient local reactive power support, the following power flow iteration index is defined to judge:

When the PQ decomposition method does not converge in power flow calculation, the index λ is used as the criterion, as shown in formula (14):

λ＝max{|[ΔU] ⁽³⁾ /[ΔU] ⁽²⁾ |} (14)

Where, [ΔU] ⁽³⁾ is the voltage increment of the third iteration, and [ΔU] ⁽²⁾ is the voltage increment of the second iteration;

When the power flow converges normally, λ<1, when the reactive power demand of the PQ node load increases, λ increases accordingly, and when the power flow is pathological, λ>1. Taking the IEEE118 node system as an example, the active power demand of the 29th PQ node is 24MW, and the reactive power demand is 4MVar. The reactive power demand of this load node is gradually increased, and its relationship with λ is shown in Table 1.

Table 1 Relationship between the increase in reactive power demand at load nodes and λ

It can be seen from Table 1 that as the node reactive power demand increases, when the system becomes ill-conditioned, λ>1, so it is reasonable to use λ as an indicator to measure the degree of system ill-conditioning.

After the operation mode is generated in step 3, the rationality of the operation mode is judged, and the PQ decomposition method is used to perform power flow calculation. When λ>1 or the power flow calculation iteration does not converge within 10 times, it is considered to be a pathological power flow phenomenon. Delete the operation mode, otherwise it is considered to be reasonable and retained, completing the intelligent deletion of the operation mode;

Step 5: Repeat steps 3 to 4 and continuously adjust the action until the target operation mode load level is met or the maximum number of action adjustments is reached;

Step 6: Output the estimated Q network parameter θ to complete the intelligent generation and intelligent deletion of power grid operation mode.

The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) to form a technical solution.

Claims (7)

1. An intelligent generation method of an electric power system operation mode based on deep reinforcement learning is characterized by comprising the following steps:

step 1: performing reinforcement learning modeling on the power grid by using a Markov Decision Process (MDP);

step 2: establishing an improved mapping strategy of the intelligent agent action and the adjustable action object;

and step 3: an operation mode is established to intelligently generate a DQN network;

and 4, step 4: modeling an intelligent deletion process of an operation mode, and constructing a pathological tide diagnosis model;

if the power flow calculation can not be converged, the following two cases are divided: the load flow calculation has no feasible solution, namely the load flow has no solution; or the power flow calculation has a feasible solution but cannot be searched, namely the problem of pathological power flow is solved;

and 5: repeating the step 3 to the step 4, and continuously adjusting the action until the load level of the target operation mode is met or the maximum action adjustment times are reached;

step 6: and outputting an estimated Q network parameter theta, and finishing intelligent generation and intelligent deletion of a power grid operation mode.

2. The intelligent generation method of the operation mode of the power system based on the deep reinforcement learning according to claim 1, wherein the step 1 specifically comprises the following steps:

step 1.1: setting parameters in the power grid operation mode process by using a Markov decision process;

setting a power grid operation mode calculator as an intelligent agent, setting power grid operation data and a power flow calculation formula as an environment, wherein the result of interaction between the intelligent agent and the environment is power grid power flow calculation convergence, and the process of interaction between the intelligent agent and the environment is represented by a Markov decision process;

the Markov decision process MDP consists of 5-tuple (S, A, P) _r R, gamma), S is the system environment state space, S _t The system state at the moment t; a is an action space, a _t The action of the agent at the moment t; p _r To transition probabilities, P _r (s _t+1 |s _t ,a _t ) Is in a state s _t Taking action a _t Post transition to state s _t+1 The probability of (d); r is a reward function, R _t Is in a state s _t Take action a _t The reward value obtained later; gamma is a discount factor (gamma is more than or equal to 0 and less than or equal to 1) and is used for balancing the influence of the instant reward value and the future reward value on the decision process;

to quantitatively describe the action a at time t _t Guiding the system State transition direction, introducing the function concept of State Action State-Action value, namely, the State s is guided at the moment t _t Performing action a _t The expected value of the accumulated reward obtained later is expressed by Q (s, a), and the specific calculation method is shown as formula (1);

Q(s,a)＝E[r _t +γr _t+1 +γ ² r _t+2 +...|s _t ＝s,a _t ＝a,π],s∈S,a∈A (1)

where, larger γ means larger effect of the future bonus value on Q (s, a), γ =1 means that the future bonus value and the instant bonus value have the same effect on Q (s, a), and γ =1 means that only the instant bonus value affects Q (s, a); π represents the action execution policy of the agent, i.e. the system state s _t And action a _t The mapping relationship between the two;

calculating the optimal strategy mu ^* Make the action a at each time _t The Q value of (2) is maximum, and the formula is shown as (2):

μ ^* ＝max Q _μ (s,a) (2)

in the formula, Q _μ (s, a) is the expected return of policy μ after taking action a from state s;

step 1.2: defining an expression of a system environment state space S, an action space A and a reward function R in the Markov decision model;

s in the system environment state space, defining the state space S at the moment t _t Comprises the following steps:

s _t ＝[p,q,s,v,L,D,l] (3)

p＝[p ₁ ,p ₂ ,...,p _m ] (4)

q＝[q ₁ ,q ₂ ,...,q _m ] (5)

s＝[s ₁ ,s ₂ ,...,s _n ] (5)

v＝[v ₁ ,v ₂ ,...,v _g ] (6)

L＝[L ₁ ,L ₂ ,...,L _h ] (7)

D＝[D ₁ ,D ₂ ,...,D _k ] (8)

l＝[l ₁ ,l ₂ ,...,l _N ] (9)

in the formula, p _i Numbering the active power of the i-node generator; q. q.s _i Numbering the reactive power of the i-node generator; s is _i Numbering the line commissioning state of the node i; v. of _i New energy contribution condition, L, for numbering i nodes _i The controllable load output of the node i is numbered; d _i The direct current output is numbered as an i node; m, n, g, h and k are respectively adjustable generator sections without balancing machinesPoint total number, line total number, new energy node total number, controllable load node total number and direct current node total number; l ₁ ,l ₂ ,...,l _N The codes are combined to form a binary code which is used for representing the numbers of different operation modes;

in the action space A, the action space A is discrete, and the action space A is associated with discrete positive integers, wherein the formula is shown as (10);

A＝[{1,2,...,m},{1,2,...,n},{1,2,...,g},{1,2,...,h},{1,2,...,k}] (10)

the number in the set A represents the number of the adjustable action object, and the adjusting action a at the moment t _t To represent;

defining 4 indexes of the flow adjustment problem: (1) Convergence of load flow calculation by c ₁ Represents; (2) The output power of the balancing machine is not out of limit by c ₂ Represents; (3) Quantizing the network loss rate lower than a set value by calculating the network loss rate; (4) No pathological load flow is generated, and the lambda value is quantized through load flow calculation iteration; thus, the reward function R is as in equation (11):

execution of a _t And then, the load flow calculation is converged, and the output power of the balancing machine is not out of limit, R is 0, and R is-1 in other cases.

3. The method as claimed in claim 1, wherein the improved mapping strategy in step 2 is set as P _G The active power sum of the current power grid generator without the balancing machine is obtained; p _L The total active power of all the current loads of the power grid is obtained;

maximum/minimum active power for the balancing machine; k is the set target network loss rate; p _i Is the active power of the generator i; p _imax For maximum active power of generator i, minimum adjustment thresholdIs 0.05P _imax 。

4. The intelligent generation method of the operation mode of the power system based on the deep reinforcement learning as claimed in claim 1, wherein the DQN network in step 3 is obtained by combining a Q-learning network and a neural network, estimating a Q-value function by using the neural network, calculating a value function of each power flow adjustment action by the neural network, then selecting an action by using epsilon-greedy search, and selecting an action output with the largest Q value.

5. The intelligent generation method for the operation mode of the power system based on the deep reinforcement learning as claimed in claim 1, wherein the pathological power flow problem in the step 4 includes the following two situations;

(1) The active unbalanced power is distributed by adjusting the output of the adjustable action object according to the pathological trend caused by the overweight of the section trend, so that the problem of the pathological trend is solved;

(2) The pathological tide caused by insufficient local reactive power support is judged by defining the following tide iteration indexes:

when the flow calculation by adopting the PQ decomposition method is not converged, taking the index lambda as a criterion, as shown in a formula (14):

λ＝max{|[ΔU] ⁽³⁾ /[ΔU] ⁽²⁾ |} (14)

in the formula, [ Delta U ]] ⁽³⁾ For the third iteration voltage value increment, [ Δ U [ ]] ⁽²⁾ The voltage value increment is the second iteration value;

lambda is less than 1 when the power flow is normally converged, lambda is increased when the reactive demand of PQ node load is increased, and lambda is greater than 1 when the power flow is ill-conditioned;

and 3, after the operation mode is generated in the step 3, judging the rationality of the operation mode, carrying out flow calculation by adopting a PQ decomposition method, and when the lambda is greater than 1 or the iteration of the flow calculation is not converged within 10 times, considering the pathological flow phenomenon and deleting the operation mode, otherwise, considering the operation mode to be reasonable and reserved, and finishing the intelligent deletion of the operation mode.

6. The intelligent deep reinforcement learning-based power system operation mode generation method according to claim 3, wherein the improved mapping strategy comprises the following three conditions:

(1) When the temperature is higher than the set temperature

When a is turned on _t If i, let P _i ＝0.5P _imax If at this time P _i ≥0.5P _imax Then put into operation P _i And P _imax Is rounded up until P is delivered _imax Until the end; judging that the total active power of a system generator is insufficient in the scene, and increasing the active power of the generator to meet the requirement of power flow convergence;

(2) When the temperature is higher than the set temperature

When a is turned on _t If i, let P _i ＝0.5P _imax If at this time P _i ≤0.5P _imax Then P is _i The median value of the output force of the machine halt is rounded downwards until the input is 0 percent, and the machine halt is carried out; in the scene, the situation that the total active power of a system generator is too large is judged, and the active power of the generator needs to be reduced to meet the requirement of power flow convergence;

(3) When except (1) and (2), a _t If P is not less than = i _i ≥0.5P _imax Then put into operation P _i And P _imax Is rounded up until P is reached _imax Until the end; otherwise, put into operation P _i And rounding the median value of the output force of the shutdown downwards until the delivery reaches 0 percent, namely, the shutdown is carried out.

7. The intelligent generation method of the operation mode of the power system based on the deep reinforcement learning of claim 4, wherein the DQN network introduces an estimation Q network and a target Q network, and the training process comprises:

step A1: when training starts, setting the parameters of the estimated Q network and the target Q network node, the generator, the circuit and the load to be the same, wherein the parameter matrixes are theta and theta';

step A2: in the training process, updating each time step of the estimated Q network once according to the gradient descending direction of the loss function as the formula (13), and calculating a Q value by the DQN network according to the estimated Q network and the current state to output a load flow adjusting action;

step A3: adjusting the running state of the adjustable action object according to the step A2;

step A4: transmitting the estimated Q network parameter theta to a target Q network theta' every other step C;

step A5: updating the target Q network once every C time steps according to the gradient descending direction of the formula (13);

and updating parameters of the estimated Q network in a back propagation mode, and repeating the updating process during training until the power flow is converged and the output of the balancing machine is not out of limit or reaches the number of iteration rounds.

Images