CN114444802A - Electric vehicle charging guide optimization method based on graph neural network reinforcement learning - Google Patents

Abstract

The present invention provides an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning, comprising the following steps: step S1: initialization of a collaborative optimization model of the power-transport fusion network; step S2: updating the electric vehicle charging load; step S3: according to The epsilon-Greedy algorithm and the graph neural network reinforcement learning algorithm generate a _i,t ; Step S4: execute the charging guide behavior strategy ai _,t ; Step S5: calculate the reward function of the graph neural network reinforcement learning algorithm; The state xi _,t of the Kov decision-making process is updated; Step S7: Store the information of the current step ( xi _,t , a _i,t , ri _,t , xi _,t ) in the memory unit D ; Step S8 : determine whether the predetermined time T _end is reached; if not, execute (2) to (7); if so, output the graph neural network reinforcement learning algorithm parameters and corresponding output results. By applying the technical solution, the total cost of electric vehicle charging can be effectively reduced, and the orderly charging of the electric vehicle and the coordinated optimal scheduling of the power system can be realized.

Description

Electric vehicle charging guidance optimization method based on graph neural network reinforcement learning

technical field

The invention relates to the technical field of collaborative optimization of an electric power-transportation fusion network, in particular to an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning.

Background technique

With the large-scale operation of electric vehicles, there will be many interactions and integrations between the power system and the transportation system, forming a power-transportation fusion network. The fusion network involves multiple subjects such as electric vehicles, power systems, and transportation systems, and contains a variety of random uncertain factors. The interaction of multiple agents, the influence of multiple random factors, and the coupling relationship of multiple random factors make it more difficult to understand the interaction mechanism of the power and transportation systems and to solve the collaborative optimization of the power-transport fusion network. For example, the travel and psychological behavior and driving behavior of electric vehicle users have a certain degree of randomness, which will affect the flow distribution of the traffic system, making the traffic flow also have certain uncertainty, and further affecting the electric vehicle reaching the charging station. Time, making the charging time, queuing time and charging time of electric vehicles also have strong uncertainty. Different from traditional electric loads, electric vehicles, as a movable load, have stronger randomness and are more difficult to predict than traditional electric loads.

At present, the research on the power-transport fusion network can be divided into three research directions: 1) From the perspective of the power system, by calculating the node marginal cost price or optimizing the charging station service pricing to guide the electric vehicle to charge at the lowest cost; 2) From the perspective of the power system From the perspective of the transportation system, the optimization of the charging path is considered to minimize the charging cost; 3) the interests of the electric vehicle, the electric power and the transportation system are comprehensively considered, and the comprehensive benefit is maximized by optimizing the charging strategy of the electric vehicle and the scheduling decision of the electric power system. However, most of the existing researches belong to static optimization problems, and have not considered the coupling relationship of electric vehicles, charging stations, and power systems on a continuous time scale. The influence of correlation coupling on the collaborative optimization of power-transportation fusion network. More importantly, the existing research has not considered the impact of the interaction between electric vehicles on the synergistic optimization of the power-transportation fusion network.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning, which can effectively reduce the number of uncertainties in the power-transportation fusion network. The total cost of electric vehicle charging, realizing the orderly charging of electric vehicles and the coordinated optimal scheduling of the power system.

In order to achieve the above purpose, the present invention adopts the following technical scheme: an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning, comprising the following steps:

Step S1: initialization of the collaborative optimization model of the power-transport fusion network;

Step S2: updating the electric vehicle charging load, and optimizing the marginal cost electricity price of the node where the electric vehicle charging station is located based on the second-order cone relaxation optimization and dual theory;

Step S3: according to the epsilon-Greedy algorithm and the graph neural network reinforcement learning algorithm, the electric vehicle charging guidance behavior strategy a _i,t is generated;

Step S4: executing the charging guidance behavior strategy a _i,t , and judging and updating the state of the electric vehicle;

Step S5: Calculate the reward function of the graph neural network reinforcement learning algorithm according to the power-transport fusion environment;

Step S6: update the state x _{i, t} of the partially observed Markov decision process;

Step S7: Store the information of the current step (x _i,t ,a _i,t ,r _i,t , _xi,t ') in the memory unit D, and perform reinforcement learning on the graph neural network based on the stochastic gradient descent method The algorithm weights are updated; among them, x _i,t , represents the current state of the reinforcement learning of the graph neural network; a _i,t represents the electric vehicle behavior strategy; ri _,t represents the reward function value of the reinforcement learning of the graph neural network; _xi,t 'represents the next state of the reinforcement learning of the graph neural network;

Step S8: determine whether the predetermined time T _end is reached; if not, execute (2) to (7); if so, output the parameters of the graph neural network reinforcement learning algorithm and the corresponding output result.

In a preferred embodiment, initializing the collaborative optimization model of the power-transport fusion network includes the following steps:

Step 21: Determine the topology and parameters of the power network and the transportation network, including power system nodes, lines, initial voltages, and optimized upper and lower limits, and the transportation network includes traffic nodes, road parameters, capacity, and maximum travel speed;

Step 22: Neural network parameter initialization, including neural network weight initialization and hyperparameter settings, such as learning rate α, discount factor γ, batch size B and memory unit D capacity;

Step 23: Consider each electric vehicle in the study area as an agent, and regard it as a node n∈N, and regard the connection between electric vehicles as an edge e∈E, so as to form a graph network structure G=( N, E), and initialize the current state x _{i, t} and the adjacency matrix A of each electric vehicle i.

In a preferred embodiment, the steps of updating the electric vehicle charging load and optimizing the marginal cost electricity price of the node where the electric vehicle charging station is located based on second-order cone relaxation optimization and dual theory include:

Step 31: Update the charging load of electric vehicles: Calculate the charging load of each charging station according to the number of electric vehicles in the charging station and the charging power, obtain the charging load of each station and add the basic load of the node to obtain the final power consumption of the node load;

Step 32: Establish the optimal power flow model of the distribution network based on the branch power flow model:

min f(p,q,P,Q,V,I) (1)

和

表示发电机有功和无功出力，即注入到节点j的有功功率和无功功率；

和

表示风机注入到节点j的有功功率和无功功率；Q_js表示从节点j流向节点s的支路无功功率；r_ij和x_ij表示从节点i到节点j的支路电阻和电抗；I_ij表示从节点i到节点j的支路电流；π(j)表示与节点j相连的支路集合；

和

表示连接在节点j上的有功负荷和无功负荷；V_i表示节点i的电压幅值；V_j表示节点j的电压幅值；z_ij表示连接节点i和节点j的支路阻抗，满足z_ij＝r_ij+jx_ij；

表示连接节点i和节点j的支路电流最大值；V _j和

表示节点j的最小和最大电压；

表示连接到节点j的风机最大有功出力；

表示连接到节点j的风机的功率因素；In the formula, _{EN and EL represent the distribution network nodes and line sets, respectively; P ij and} Q _ij represent the branch active power and reactive power flowing from node i to node j; _{P jk} represent the flow from node j to node k. branch active power;

and

Represents the active and reactive output of the generator, that is, the active power and reactive power injected into node j;

and

represents the active power and reactive power injected by the fan into node j; Q _js represents the branch reactive power flowing from node j to node s; r _ij and x _ij represent the branch resistance and reactance from node i to node j; I _ij represents the branch current from node i to node j; π(j) represents the set of branches connected to node j;

and

represents the active load and reactive load connected to node j; V _i represents the voltage amplitude of node i; V _j represents the voltage amplitude of node j; z _ij represents the branch impedance connecting node i and node j, satisfying z _ij =r _ij +jx _ij ;

represents the maximum value of the branch current connecting node i and node j; V _j and

represents the minimum and maximum voltage of node j;

Represents the maximum active power output of the fan connected to node j;

represents the power factor of the fan connected to node j;

包括基础负荷

和电动汽车充电负荷

即The load of distribution network node j

including base load

and electric vehicle charging load

which is

According to the actual demand of the distribution network, the objective function min f(p,q,P,Q,V,I) can be finally defined as:

表示注入节点i发电机的有功出力；a_i和b_i分别表示发电机的二次煤耗和一次煤耗系数；

和

分别从主网中购买电量的电价和有功功率；In the formula,

represents the active power output of the generator injected into node i; a _i and b _i represent the secondary coal consumption and primary coal consumption coefficient of the generator, respectively;

and

The electricity price and active power of electricity purchased from the main network, respectively;

Step 33: Convert the above nonlinear distribution network optimal power flow model into a second-order cone relaxation programming model:

以及支路电压幅值

并对式进行二阶锥松弛(SOCR)转换，可以得到以下模型：Since BFM-OPF is a nonlinear programming model, let the branch current amplitude

and the branch voltage amplitude

And the second-order cone relaxation (SOCR) transformation of the formula, the following model can be obtained:

where ||·|| ₂ represents the second-order cone operation; the above formula - constitutes the basic form of the optimal power flow of the distribution network after relaxation;

Step 34: Use the Gurobi solver to solve the original problem and dual variables of the above model, and obtain the marginal cost electricity price λ _k of the node where the charging station is located.

In a preferred embodiment, the epsilon-Greedy algorithm includes the following steps:

Step 41: Generate a random number u, and judge the size of it and the decay factor ξ of the epsilon-Greedy algorithm;

Step 42: If u<ξ, generate a behavior a _i,t for each electric vehicle in the current state in a random manner, and this behavior represents the electric vehicle charging path strategy in the patent;

a _i,t = randint(N _action ) (19)

where N _action represents the number of EV behavior decisions;

Step 43: If u≥ξ, generate a behavior a _i,t for each electric vehicle i under the current state x _i,t and the adjacency matrix A according to the experience of the graph neural network reinforcement learning algorithm, that is,

In the formula, θ _t represents the parameters of the graph neural network reinforcement learning algorithm; argmax() represents the parameter operation corresponding to the maximum value; x _{i, t} represents the state of the i-th electric vehicle at time t, which is mainly determined by time t. The state x _i,t of the i-th electric vehicle is composed of the electric vehicle state EV _i,t , the neighboring traffic road information Ro _i,t , the neighboring electric vehicle state Ne _i,t and the information CS _t of each charging station, namely

x _i,t =[EV _i,t ,Ro _i,t ,Ne _i,t ,CS _t ] (21)

道路编号

电动汽车行驶速度v_i,t和剩余电量SOC_i,t；近邻交通道路信息状态Ro_i,t包括与电动汽车i所在下一节点

相连的下一条道路的起始节点

末节点

道路长度

以及道路上的电动车数量

近邻电动汽车状态Ne_i,t包括各近邻电动汽车k的状态，如与第i辆电动汽车临近的第k辆电动汽车下一节点

其所在的道路编号

电动汽车行驶速度v_i,k,t和剩余电量SOC_i,k,t；充电站信息CS_t包括各充电站的充电电价p_c,t和电动汽车数量

In the formula, the state EV _i,t of the ith electric vehicle includes the next node when the electric vehicle goes to the charging station

road number

Electric vehicle driving speed v _i,t and remaining power SOC _i,t ; neighbor traffic road information state Ro _i,t includes the next node where electric vehicle i is located

The starting node of the next connected road

end node

road length

and the number of electric vehicles on the road

Neighboring electric vehicle state Ne _i,t includes the state of each neighboring electric vehicle k, such as the next node of the k-th electric vehicle adjacent to the i-th electric vehicle

the road number it is on

Electric vehicle running speed v _i,k,t and remaining power SOC _i,k,t ; charging station information CS _t includes charging electricity price p _c,t of each charging station and the number of electric vehicles

The neural network structure of the graph neural network reinforcement learning algorithm includes an input layer of one layer, and a fully connected layer of one layer performs feature extraction x _{i, t} ' on the input state x _{i, t} , and then the proposed feature x _i , t ' is extracted. _t ' and the adjacency matrix A are input into the two-layer graph neural network for feature extraction, and finally a fully connected layer is connected to output the electric vehicle charging path strategy a _{i, t} ; wherein, the graph neural network adopts is the graph attention network.

In a preferred embodiment, the reward function ri _,t of the graph neural network reinforcement learning algorithm is shown in the formula:

和

分别表示在时间t时第i辆电动汽车前往第k个充电站时的行驶时间、充电等待时间和充电所需时间；λ_k,t表示在时间t时充电站k所在节点的边际成本电价；SOC_i,k,t表示在时间t时第i辆电动汽车达到充电站k时的剩余电量SOC_i,k,t；

表示第i辆电动汽车电池额定容量；In the formula, node _cur and node _tar represent the current node where the electric vehicle is located and any charging station node that the electric vehicle will go to, step represents the number of steps the electric vehicle has traveled; penalty represents a large penalty factor; w _i represents the i -th The unit time cost of an electric vehicle;

and

λ _{k, t} represent the marginal cost electricity price of the node where the charging station k is located at time t; SOC _i,k,t represents the remaining power SOC _{i,k,t when the ith electric vehicle reaches the charging station k at time t} ;

Indicates the rated capacity of the i-th electric vehicle battery;

和充电时间

以及充电时电费来计算；It can be seen from the formula that the reward function ri _,t is a piecewise function; if the i-th electric vehicle does not arrive at the charging station node _cur ≠node _tar and the current number of steps of the electric vehicle to the charging station is within the given maximum charging step step<N _step in the number, then its reward function ri _,t = 0; if the number of steps taken by the i-th electric vehicle to the charging station is greater than or equal to the given maximum number of charging steps step≥N _step , it indicates that this charging time If the behavior exploration fails, a large negative reward ri _,t =-penalty is given to it; if the ith electric vehicle reaches the charging station node _cur = node _tar and the current number of steps of the electric vehicle to the charging station is within the given value step<N _step within the maximum number of charging steps, at this time, the reward function is based on the driving time of the electric vehicle

and charging time

And the electricity charge when charging is calculated;

The travel time t _a,t of the i-th electric vehicle on road segment a is calculated according to the U.S. Federal Highway Administration (BPR) function, that is,

分别表示路段a的容量上限和t时刻电动汽车自由通行时间；由此可以得到第i辆电动汽车前往充电站k所需时间

即In the formula, n _{a, t} represent the number of electric vehicles on road segment a at time t; c _a and

respectively represent the upper limit of the capacity of road section a and the free passage time of electric vehicles at time t; from this, the time required for the i-th electric vehicle to go to charging station k can be obtained.

which is

可以通过式得到；In addition, the charging waiting time of the i-th electric vehicle

can be obtained by the formula;

表示电动汽车电池的额定容量；η表示充电功率因素，P^charging表示电动汽车充电的额定功率。In the formula, SOC _t represents the remaining power of the electric vehicle;

Represents the rated capacity of the electric vehicle battery; η represents the charging power factor, and P ^charging represents the rated power of the electric vehicle charging.

In a preferred embodiment, the stochastic gradient descent-based method for updating the weights of the graph neural network reinforcement learning algorithm includes:

Step 61: Randomly extract a certain number of samples from the memory unit D;

Step 62: Construct the loss function as shown in the formula, and update the weights of the graph neural network reinforcement learning algorithm according to the stochastic gradient descent method under the sampled sample as shown in the formula;

表示在目标图神经网络强化学习算法参数θ′_t下的状态-动作值；In the formula, x, a, x' and a' are the current state, action and state and action at the next moment, respectively; r represents the immediate reward of the reinforcement learning of the graph neural network; θ _t represents the reinforcement learning of the graph neural network at the current moment t Algorithm parameters; 0≤γ≤1 represents the discount factor, which reflects the influence of the future Q value on the current action;

represents the state-action value under the target graph neural network reinforcement learning algorithm parameter θ′ _t ;

表示对θ_t进行求导操作；α表示学习速率；In the formula, θ _t represents the parameters of the reinforcement learning algorithm of the graph neural network at the current time t;

Represents the derivation operation on θ _t ; α represents the learning rate;

Step 63: Update the target graph neural network reinforcement learning parameter θ′ _t according to the current graph neural network reinforcement learning parameter θ _t every time a certain number of steps pass.

Compared with the prior art, the present invention has the following beneficial effects:

The invention provides an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning. Based on graph theory, the mutual influence relationship between electric vehicles is converted into a dynamic network graph structure, and a graph neural network based on attention mechanism is proposed. Network reinforcement learning is used to deal with irregular non-European structure data, in order to study the communication and cooperation between multi-agents, and to explore the mutual influence between electric vehicles. Based on the active distribution network considering the output of renewable energy, the optimal power flow of the distribution network is solved through the second-order cone optimization and dual optimization theory, and the marginal cost price of the distribution network node is obtained, so as to study the power-transportation integrated network. Collaborative optimization. The proposed optimization method of electric vehicle charging guidance based on the reinforcement learning of graph neural network can effectively reduce the total cost of electric vehicle charging and realize the electric vehicle charging under the condition of considering various uncertain factors of the power-transport fusion network. Orderly charging and coordinated optimal scheduling of the power system.

Description of drawings

FIG. 1 is a flow chart of an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to a preferred embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

It should be noted that the terms used herein are for the purpose of describing particular embodiments only and are not intended to limit exemplary embodiments in accordance with the present application; as used herein, unless the context clearly dictates otherwise, the singular forms are also intended to include Plural forms, furthermore, should also be understood that when the terms "comprising" and/or "comprising" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and/or combinations thereof.

As shown in FIG. 1, it is an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning of the present invention, which includes the following steps:

S11: initialization of the collaborative optimization model of the power-transport fusion network;

S12: Update the electric vehicle charging load, and optimize the marginal cost electricity price of the node where the electric vehicle charging station is located based on the second-order cone relaxation optimization and dual theory;

S13: Generate an electric vehicle charging guidance behavior strategy a _i,t according to the epsilon-Greedy algorithm and the graph neural network reinforcement learning algorithm;

S14: Execute the charging guidance behavior strategy a _i,t , and judge and update the state of the electric vehicle;

S15: Calculate the reward function of the graph neural network reinforcement learning algorithm according to the power-transport fusion environment;

S16: The state x _{i, t} of the partially observed Markov decision process is updated;

S17: Store the information of the current step (x _i,t ,a _i,t ,r _i,t , _xi,t ') in the memory unit D, and strengthen the learning algorithm of the graph neural network based on the method of stochastic gradient descent weights are updated;

S18: Determine whether the predetermined time T _end is reached. If not, execute (2) to (7); if so, output the parameters of the graph neural network reinforcement learning algorithm and the corresponding output results.

specific:

1. Initialization of the collaborative optimization model of the power-transport fusion network. The main steps include the determination of power network and traffic network topology and parameters, including power system nodes, lines, initial voltages, and optimized upper and lower limits. The traffic network includes traffic nodes, road parameters, capacity, and maximum driving speed.

Neural network parameter initialization, including neural network weight initialization and hyperparameter settings, such as learning rate α, discount factor γ, batch size B and memory cell capacity size D;

Each electric vehicle in the study area is regarded as an agent, and it is regarded as a node n∈N, and the connection between electric vehicles is regarded as an edge e∈E, so as to form a graph network structure G=(N,E ), and initialize the current state _xi,t and adjacency matrix A for each electric vehicle i.

2. Update the electric vehicle charging load, and optimize the marginal cost electricity price of the node where the electric vehicle charging station is located based on the second-order cone relaxation optimization and dual theory. It mainly includes the following steps:

Step 21: Update the electric vehicle charging load: Calculate the charging load of each charging station according to the number of electric vehicles in the charging station and the charging power, and then add the base load of the node to obtain the final power consumption of the node after obtaining the charging load of each station. load;

Step 22: Establish the optimal power flow model of the distribution network based on the branch power flow model:

min f(p,q,P,Q,V,I) (1)

和

表示发电机有功和无功出力，即注入到节点j的有功功率和无功功率；

和

表示风机注入到节点j的有功功率和无功功率；r_ij和x_ij表示从节点i到节点j的支路电阻和电抗；I_ij表示从节点i到节点j的支路电流；π(j)表示与节点j相连的支路集合；

和

表示连接在节点j上的有功负荷和无功负荷；V_i表示节点i的电压幅值；z_ij表示连接节点i和节点j的支路阻抗，满足z_ij＝r_ij+jx_ij；

表示连接节点i和节点j的支路电流最大值；V _j和

表示节点j的最小和最大电压；

表示连接到节点j的风机最大有功出力；

表示连接到节点j的风机的功率因素。In the formula, EN and _{EL represent the distribution network nodes and line sets, respectively; P ij and Q ij represent the branch} active power and reactive power flowing from node i to node j;

and

Represents the active and reactive output of the generator, that is, the active power and reactive power injected into node j;

and

Represents the active power and reactive power injected by the fan into node j; r _ij and x _ij represent the branch resistance and reactance from node i to node j; I _ij represents the branch current from node i to node j; π(j ) represents the set of branches connected to node j;

and

represents the active load and reactive load connected to node j; V _i represents the voltage amplitude of node i; z _ij represents the branch impedance connecting node i and node j, satisfying zi _ij =r _ij +jx _ij ;

represents the maximum value of the branch current connecting node i and node j; V _j and

represents the minimum and maximum voltage of node j;

Represents the maximum active power output of the fan connected to node j;

represents the power factor of the fan connected to node j.

包括基础负荷

和电动汽车充电负荷

即The load of distribution network node j

including base load

and electric vehicle charging load

which is

According to the actual demand of the distribution network, the objective function min f(p,q,P,Q,V,I) can be finally defined as:

和

分别从主网中购买电量的电价和有功功率。In the formula, a _i and b _i represent the secondary coal consumption and primary coal consumption coefficient of the generator, respectively;

and

The electricity price and active power of electricity purchased from the main network, respectively.

Step 23: Convert the above nonlinear distribution network optimal power flow model into a second-order cone relaxation programming model:

以及

并对式进行二阶锥松弛(SOCR)转换，可以得到以下模型：Since BFM-OPF is a nonlinear programming model, let

as well as

And the second-order cone relaxation (SOCR) transformation of the formula, the following model can be obtained:

In the formula ||·|| ₂ represents the second-order cone operation; the above formula - constitutes the basic form of the optimal power flow of the distribution network after relaxation.

Step 24: Use the Gurobi solver to solve the original problem and dual variables of the above model, and obtain the marginal cost electricity price λ _k of the node where the charging station is located.

3. Generate electric vehicle charging guidance behavior strategy a _i,t according to epsilon-Greedy algorithm and graph neural network reinforcement learning algorithm. It mainly includes the following steps:

Step 31: Generate a random number u, and determine its size with the decay factor ξ of the epsilon-Greedy algorithm.

Step 32: If u<ξ, generate a behavior a _i,t for each electric vehicle in the current state in a random manner, which represents the electric vehicle charging path strategy in the patent;

a _i,t = randint(N _action ) (19)

where N _action represents the number of EV behavior decisions.

Step 33: If u≥ξ, according to the experience of the graph neural network reinforcement learning algorithm, a behavior a _i,t is generated for each electric vehicle i under the current state x _i,t and the adjacency matrix A, that is,

In the formula, θ _t represents the parameters of the graph neural network reinforcement learning algorithm; argmax() represents the parameter operation corresponding to the maximum value; x _{i, t} represents the state of the i-th electric vehicle at time t, which is mainly determined by time t. The state x _i,t of the i-th electric vehicle is composed of the electric vehicle state EV _i,t , the neighboring traffic road information Ro _i,t , the neighboring electric vehicle state Ne _i,t and the information CS _t of each charging station, namely

x _i,t =[EV _i,t ,Ro _i,t ,Ne _i,t ,CS _t ] (21)

道路编号

电动汽车行驶速度v_i,t和剩余电量SOC_i,t；近邻交通道路信息状态Ro_i,t包括与电动汽车i所在下一节点

相连的下一条道路的起始节点

末节点

道路长度

以及道路上的电动车数量

近邻电动汽车状态Ne_i,t包括各近邻电动汽车k的状态，如与第i辆电动汽车临近的第k辆电动汽车下一节点

其所在的道路编号

电动汽车行驶速度v_i,k,t和剩余电量SOC_i,k,t；充电站信息CS_t包括各充电站的充电电价p_c,t和电动汽车数量

In the formula, the state EV _i,t of the ith electric vehicle includes the next node when the electric vehicle goes to the charging station

road number

Electric vehicle driving speed v _i,t and remaining power SOC _i,t ; neighbor traffic road information state Ro _i,t includes the next node where electric vehicle i is located

The starting node of the next connected road

end node

road length

and the number of electric vehicles on the road

Neighboring electric vehicle state Ne _i,t includes the state of each neighboring electric vehicle k, such as the next node of the k-th electric vehicle adjacent to the i-th electric vehicle

the road number it is on

Electric vehicle running speed v _i,k,t and remaining power SOC _i,k,t ; charging station information CS _t includes charging electricity price p _c,t of each charging station and the number of electric vehicles

The neural network structure of the graph neural network reinforcement learning algorithm includes an input layer of one layer, and a fully connected layer of one layer performs feature extraction x _{i, t} ' on the input state x _{i, t} , and then the proposed feature x _i , t ' is extracted. _t ' and the adjacency matrix A are input into a two-layer graph neural network for feature extraction, and finally a fully connected layer is connected to output the electric vehicle charging path strategy a _i,t . Among them, the graph neural network described in this patent adopts the graph attention network.

Fourth, implement the charging guidance behavior strategy a _i,t , and judge and update the state of the electric vehicle. There are three states of electric vehicles: decision state, running state and charging state. If the electric vehicle arrives at the intersection node _cur = node _next and the intersection is not a charging station node node _cur ≠node _tar , at this time the electric vehicle is in a decision-making state, the electric vehicle executes the charging guidance behavior strategy a _i,t , and updates the road state such as the electric vehicle Quantity, ideal driving speed, update the information of the electric vehicle status such as the road location, driving speed and distance; if the electric vehicle does not reach the intersection node _cur ≠ node _next , the electric vehicle is in the running state at this time, that is, the electric vehicle is in the running state according to the previous step. The charging guidance strategy a _{i, t-1} continues to drive forward along the current road, and updates the electric vehicle position information, speed information and SOC status at this time; if the electric vehicle is located at the node position of the charging station node _cur = node _tar , At this time, the electric vehicle is in the charging state. If the current number of electric vehicles is greater than the number of charging piles in the charging station, the electric vehicle needs to wait in line for charging. If there are available charging piles in the charging station, the electric vehicle will be charged immediately. And update EV charging waiting time, charging time and EV SOC status.

充电等待时间

充电时间

以及充电时电费来计算，具体计算表达式如所示。5. Calculate the reward function of the graph neural network reinforcement learning algorithm according to the power-transport fusion environment. Specifically, the reward function r _i,t is a piecewise function: if the ith electric vehicle does not arrive at the charging station node _cur ≠ node _tar and the current number of steps of the electric vehicle to the charging station is within the given maximum number of charging steps <N _step , the reward function ri _,t = 0 at this time; if the number of steps taken by the i-th electric vehicle to the charging station is greater than or equal to the given maximum number of charging steps step≥N _step , it indicates that the exploration of the charging behavior has failed. , at this time give it a large negative reward ri _,t =-penalty; if the i-th electric vehicle arrives at the charging station node _cur = node _tar and the current number of steps of the electric vehicle to the charging station is within the given maximum charging step Step<N _step in the number, at this time, its reward function is based on the driving time of the electric vehicle

Charging waiting time

charging time

And the electricity cost during charging is calculated, and the specific calculation expression is as shown.

充电等待时间

充电时间

计算表达式如-所示。travel time

Charging waiting time

charging time

The calculation expression is as shown in -.

The transit time of the i-th electric vehicle on road segment a is calculated according to the U.S. Federal Highway Administration (BPR) function, that is,

分别表示路段a的容量上限和t时刻电动汽车自由通行时间。由此可以得到第i辆电动汽车前往充电站k所需时间

即In the formula, n _{a, t} represent the number of electric vehicles on road segment a at time t; c _a and

respectively represent the upper limit of the capacity of road segment a and the free passage time of electric vehicles at time t. From this, the time required for the i-th electric vehicle to go to the charging station k can be obtained.

which is

可以通过式得到。In addition, the charging waiting time of the i-th electric vehicle

can be obtained by formula.

表示电动汽车电池额定容量；η表示充电功率因素，P^charging表示电动汽车充电的额定功率。In the formula, SOC _t represents the remaining power of the electric vehicle;

Represents the rated capacity of the electric vehicle battery; η represents the charging power factor, and P ^charging represents the rated power of the electric vehicle charging.

6. Partially observe the update of the state x _i,t of the Markov decision process, including the update of the electric vehicle state EV _i,t , the neighbor traffic road information Ro _i,t , the neighbor electric vehicle state Ne _i,t and the charging station information CS _t .

7. Store the information of the current step ( _xi,t ,ai _,t ,ri _,t , _xi,t ') in the memory unit D, and strengthen the learning algorithm of the graph neural network based on the method of stochastic gradient descent Weights are updated. It mainly includes the following steps:

Step 71: Randomly extract a certain number of samples from the memory unit D;

Step 72: Construct the loss function as shown in the formula, and update the weights of the graph neural network reinforcement learning algorithm according to the stochastic gradient descent method under the sampled sample as shown in the formula;

表示在目标图神经网络强化学习算法参数θ′_t下的状态-动作值。In the formula, x, a, x' and a' are the current state, action and state and action at the next moment, respectively; θ _t represents the graph neural network reinforcement learning algorithm parameter at the current moment t; 0≤γ≤1 represents the discount factor , which reflects the impact of the future Q value on the current action;

Represents the state-action value under the target graph neural network reinforcement learning algorithm parameter θ′ _t .

表示对θ_t进行求导操作；α表示学习速率。In the formula, θ _t represents the parameters of the reinforcement learning algorithm of the graph neural network at the current time t;

represents the derivation operation on θ _t ; α represents the learning rate.

Step 73: Update the target graph neural network reinforcement learning parameter θ′ _t according to the current graph neural network reinforcement learning parameter θ _t every time a certain number of steps pass.

8. Determine whether the predetermined time T _end is reached. If not, execute (2) to (7); if so, output the parameters of the graph neural network reinforcement learning algorithm and the corresponding output results.

The invention is an electric vehicle charging guidance optimization method based on graph neural network reinforcement learning. Based on graph theory, the mutual influence relationship between electric vehicles is converted into a dynamic network graph structure, and a graph neural network enhancement based on attention mechanism is proposed. Learn to deal with irregular non-European structure data, in order to study the communication and cooperation between multi-agents, and explore the mutual influence between electric vehicles. Based on the active distribution network considering the output of renewable energy, the optimal power flow of the distribution network is solved through the second-order cone optimization and dual optimization theory, and the marginal cost price of the distribution network node is obtained, so as to study the power-transportation integrated network. Collaborative optimization. The proposed optimization method of electric vehicle charging guidance based on the reinforcement learning of graph neural network can effectively reduce the total cost of electric vehicle charging and realize the electric vehicle charging under the condition of considering various uncertain factors of the power-transport fusion network. Orderly charging and coordinated optimal scheduling of the power system.

The above-mentioned embodiments only represent several embodiments of the present invention, and the descriptions thereof are specific and detailed, but should not be construed as a limitation on the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention should be subject to the appended claims.

Claims (6)

1. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning, is characterized in that, comprises the following steps: Step S1: initialization of the collaborative optimization model of the power-transport fusion network; Step S2: updating the electric vehicle charging load, and optimizing the marginal cost electricity price of the node where the electric vehicle charging station is located based on the second-order cone relaxation optimization and dual theory; Step S3: according to the epsilon-Greedy algorithm and the graph neural network reinforcement learning algorithm, the electric vehicle charging guidance behavior strategy a _i,t is generated; Step S4: executing the charging guidance behavior strategy a _i,t , and judging and updating the state of the electric vehicle; Step S5: Calculate the reward function ri _,t of the graph neural network reinforcement learning algorithm according to the power-traffic fusion environment; Step S6: update the state x _{i, t} of the partially observed Markov decision process; Step S7: Store the information of the current step (x _i,t ,a _i,t ,r _i,t , _xi,t ') in the memory unit D, and perform reinforcement learning on the graph neural network based on the stochastic gradient descent method The algorithm weights are updated; among them, x _i,t , represents the current state of the reinforcement learning of the graph neural network; a _i,t represents the electric vehicle behavior strategy; ri _,t represents the reward function value of the reinforcement learning of the graph neural network; _xi,t 'represents the next state of the reinforcement learning of the graph neural network; Step S8: determine whether the predetermined time T _end is reached; if not, execute (2) to (7); if so, output the parameters of the graph neural network reinforcement learning algorithm and the corresponding output result. 2. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to claim 1, characterized in that, initializing the power-transportation fusion network collaborative optimization model, comprising the following steps: Step 21: Determine the topology and parameters of the power network and the transportation network, including power system nodes, lines, initial voltages, and optimized upper and lower limits, and the transportation network includes traffic nodes, road parameters, capacity, and maximum travel speed; Step 22: Neural network parameter initialization, including neural network weight initialization and hyperparameter settings, such as learning rate α, discount factor γ, batch size B and memory unit D capacity; Step 23: Consider each electric vehicle in the study area as an agent, and regard it as a node n∈N, and regard the connection between electric vehicles as an edge e∈E, so as to form a graph network structure G=( N, E), and initialize the current state x _{i, t} and the adjacency matrix A of each electric vehicle i. 3. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to claim 1, wherein the electric vehicle charging load is updated and the node where the electric vehicle charging station is located based on second-order cone relaxation optimization and dual theory The steps of optimizing the marginal cost of electricity price include: Step 31: Update the charging load of electric vehicles: Calculate the charging load of each charging station according to the number of electric vehicles in the charging station and the charging power, obtain the charging load of each station and add the basic load of the node to obtain the final power consumption of the node load; Step 32: Establish the optimal power flow model of the distribution network based on the branch power flow model: minf(p,q,P,Q,V,I) (1) s.t.

In the formula, _{EN and EL represent the distribution network nodes and line sets, respectively; P ij and} Q _ij represent the branch active power and reactive power flowing from node i to node j; _{P jk} represent the flow from node j to node k. branch active power;

and

Represents the active and reactive output of the generator, that is, the active power and reactive power injected into node j;

and

Represents the active power and reactive power injected by the fan into node j; Q _js represents the branch reactive power flowing from node j to node s; r _ij and x _ij represent the branch resistance and reactance from node i to node j; I _ij represents the branch current from node i to node j; π(j) represents the set of branches connected to node j;

and

represents the active load and reactive load connected to node j; V _i represents the voltage amplitude of node i; V _j represents the voltage amplitude of node j; z _ij represents the branch impedance connecting node i and node j, satisfying z _ij =r _ij +jx _ij ;

represents the maximum value of the branch current connecting node i and node j; V _j and

represents the minimum and maximum voltage of node j;

Represents the maximum active power output of the fan connected to node j;

represents the power factor of the fan connected to node j; The load of distribution network node j

including base load

and electric vehicle charging load

which is

According to the actual demand of the distribution network, the objective function minf(p,q,P,Q,V,I) can be finally defined as:

In the formula, a _i and b _i represent the secondary coal consumption and primary coal consumption coefficient of the generator, respectively;

Indicates the active power output injected into the generator of node i;

and

The electricity price and active power of electricity purchased from the main network, respectively; Step 33: Convert the above nonlinear distribution network optimal power flow model into a second-order cone relaxation programming model: Since BFM-OPF is a nonlinear programming model, let the branch current amplitude

and the branch voltage amplitude

And the second-order cone relaxation (SOCR) transformation of the formula, the following model can be obtained:

s.t.

where ||·|| ₂ represents the second-order cone operation; the above formula - constitutes the basic form of the optimal power flow of the distribution network after relaxation; Step 34: Use the Gurobi solver to solve the original problem and dual variables of the above model, and obtain the marginal cost electricity price λ _k of the node where the charging station is located. 4. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to claim 3, is characterized in that, described epsilon-Greedy algorithm comprises the following steps: Step 41: Generate a random number u, and judge the size of it and the decay factor ξ of the epsilon-Greedy algorithm; Step 42: If u<ξ, generate a behavior a _i,t for each electric vehicle in the current state in a random manner, and this behavior represents the electric vehicle charging path strategy in the patent; a _i,t = randint(N _action ) (19) where N _action represents the number of EV behavior decisions; Step 43: If u≥ξ, generate a behavior a _i,t for each electric vehicle i under the current state x _i,t and the adjacency matrix A according to the experience of the graph neural network reinforcement learning algorithm, that is,

In the formula, θ _t represents the parameters of the graph neural network reinforcement learning algorithm; argmax() represents the parameter operation corresponding to the maximum value; x _{i, t} represents the state of the i-th electric vehicle at time t, which is mainly determined by time t. The state x _i,t of the i-th electric vehicle is composed of the electric vehicle state EV _i,t , the neighboring traffic road information Ro _i,t , the neighboring electric vehicle state Ne _i,t and the information CS _t of each charging station, namely x _i,t =[EV _i,t ,Ro _i,t ,Ne _i,t ,CS _t ] (21)

In the formula, the state EV _i,t of the ith electric vehicle includes the next node when the electric vehicle goes to the charging station

road number

Electric vehicle running speed v _i,t and remaining power SOC _i,t ; neighbor traffic road information state Ro _i,t includes the next node where electric vehicle i is located

The starting node of the next connected road

end node

road length

and the number of electric vehicles on the road

Neighboring electric vehicle state Ne _i,t includes the state of each neighboring electric vehicle k, such as the next node of the k-th electric vehicle adjacent to the i-th electric vehicle

the road number it is on

Electric vehicle travel speed v _i,k,t and remaining power SOC _i,k,t ; charging station information CS _t includes charging electricity price p _{c, t} of each charging station and the number of electric vehicles

The neural network structure of the graph neural network reinforcement learning algorithm includes an input layer of one layer, and a fully connected layer of one layer performs feature extraction x _{i, t} ' on the input state x _{i, t} , and then the proposed feature x _i , t ' is extracted. _t ' and the adjacency matrix A are input into the two-layer graph neural network for feature extraction, and finally a fully connected layer is connected to output the electric vehicle charging path strategy a _{i, t} ; wherein, the graph neural network adopts is the graph attention network. 5. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to claim 2, is characterized in that, the reward function r _i,t of described graph neural network reinforcement learning algorithm is as shown in the formula:

In the formula, node _cur and node _tar represent the current node where the electric vehicle is located and any charging station node that the electric vehicle will go to, step represents the number of steps the electric vehicle has traveled; penalty represents a large penalty factor; w _i represents the i -th The unit time cost of an electric vehicle;

and

λ _{k, t} represent the marginal cost electricity price of the node where charging station k is located at time t; SOC _i,k,t represents the remaining power SOC _{i,k,t when the i-th electric vehicle reaches the charging station k at time t} ;

Indicates the rated capacity of the i-th electric vehicle battery; It can be seen from the formula that the reward function ri _,t is a piecewise function; if the ith electric vehicle does not reach the charging station node _cur ≠node _tar and the current number of steps of the electric vehicle to the charging station is within the given maximum charging step step<N _step in the number, then its reward function ri _,t = 0; if the number of steps taken by the i-th electric vehicle to the charging station is greater than or equal to the given maximum number of charging steps step≥N _step , it indicates that this charging time If the behavior exploration fails, a large negative reward ri _,t =-penalty is given to it; if the ith electric vehicle reaches the charging station node _cur = node _tar and the current number of steps of the electric vehicle to the charging station is within the given value step<N _step within the maximum number of charging steps, at this time, the reward function is based on the driving time of the electric vehicle

and charging time

And the electricity charge when charging is calculated; The transit time t _a,t of the i-th electric vehicle on the road section a is calculated according to the function of the Federal Highway Administration (BPR), that is,

In the formula, n _{a, t} represent the number of electric vehicles on road segment a at time t; c _a and

respectively represent the upper limit of the capacity of the road section a and the free passage time of the electric vehicle at time t; from this, the time required for the i-th electric vehicle to go to the charging station k can be obtained.

which is

In addition, the charging waiting time of the i-th electric vehicle

can be obtained by the formula;

In the formula, SOC _t represents the remaining power of the electric vehicle;

Represents the rated capacity of the electric vehicle battery; η represents the charging power factor, and ^Pcharging represents the rated power of the electric vehicle charging. 6. The electric vehicle charging guidance optimization method based on graph neural network reinforcement learning according to claim 1, wherein the method based on stochastic gradient descent updates the weight of the graph neural network reinforcement learning algorithm comprising: Step 61: Randomly extract a certain number of samples from the memory unit D; Step 62: Construct the loss function as shown in the formula, and update the weights of the graph neural network reinforcement learning algorithm according to the stochastic gradient descent method under the sampled sample as shown in the formula;

In the formula, x, a, x' and a' are the current state, action and state and action at the next moment, respectively; r represents the immediate reward of the reinforcement learning of the graph neural network; θ _t represents the reinforcement learning of the graph neural network at the current moment t Algorithm parameters; 0≤γ≤1 represents the discount factor, which reflects the influence of the future Q value on the current action;

represents the state-action value under the target graph neural network reinforcement learning algorithm parameter θ′ _t ;

In the formula, θ _t represents the parameters of the reinforcement learning algorithm of the graph neural network at the current time t;

Represents the derivation operation on θ _t ; α represents the learning rate; Step 63: After a certain number of steps, update the target graph neural network reinforcement learning parameter θ′ _t according to the current graph neural network reinforcement learning parameter θ _t .

Images