CN114613169A - Traffic signal lamp control method based on double experience pools DQN - Google Patents

Abstract

The invention discloses a traffic signal light control method based on double experience pool DQN, comprising: 1. establishing a traffic signal light control main network and a target value network based on a DQN algorithm; Establish the state value s _t ; 3. Input s _t into the main network, select the action at _t with the maximum value of Q; 4. Execute at and calculate the reward r _t and the state s _{t +1} ; (s _t , at _t , r _t , s _t+1 ) are stored in the first experience pool; 5. If the reward r _t is greater than the average reward of historical experience

Store (s _t , at , r _t , s _{t +1} ) in the second experience pool; 6. Generate a random number P, select the first experience pool with probability 1-P, and select the second experience pool with probability P, Randomly sample from the selected experience pool, and train the parameters of the main network by minimizing the loss function; S7, update the parameters of the target value network regularly; update s _t according to the current road condition information, and jump to step 3 to continue execution. This method can make the algorithm converge quickly, and the obtained traffic light control strategy can be quickly optimized.

Description

A traffic signal control method based on double empirical pool DQN

technical field

The invention belongs to the technical field of traffic signal light control, and in particular relates to a traffic signal light control method based on double-experience pool deep Q-learning.

Background technique

There has been a lot of research on the regulation of traffic lights based on deep Q-learning algorithm (DQN). This method does not need labeled test data, but builds training data by establishing an experience pool. The strategy obtained at the beginning of the algorithm is poor. With the continuous update of the experience pool and the continuation of training, the obtained strategy is gradually optimized. Getting better and better. Therefore, how to make the algorithm converge quickly, that is, the rapid optimization of the strategy, is an important factor affecting the overall execution effect of the method.

SUMMARY OF THE INVENTION

Purpose of the invention: The present invention provides a traffic signal light control method based on the double experience pool DQN, which can make the algorithm converge quickly, and the obtained traffic signal light control strategy can be quickly optimized.

Technical scheme: the present invention adopts the following technical scheme:

A traffic signal control method based on double experience pool DQN, comprising the steps of:

S1. Establish a traffic light control main network and a target value network based on the DQN algorithm; the traffic light control main network and the target value network have the same structure, the input is a state value, and the output is a Q for performing various actions under the input state value value maximum value, and the action corresponding to the Q value maximum value; the state space of the main network and the target value network is the vector formed by the number of vehicles in each lane of the traffic intersection, and the action space is the current phase of all traffic lights at the traffic intersection. The vector formed by the control operation of , and the reward function is the difference between the number of vehicles in all incoming lanes and the number of vehicles in outgoing lanes at the traffic intersection;

S2. Randomly initialize the parameter θ of the main network, initialize the parameter θ′ of the target value network to θ, initialize the time step t=0, collect the road condition information of the traffic intersection, establish the initial state value s _t , and initialize

S3. Input s _t into the main network, and select the action a _t that makes Q(s _t , a; θ) take the maximum value as the control operation for the traffic lights at the current time, namely: a _t =argmax _a Q(s _t , a; θ), where Q(s _t , a; θ) represents the Q value output by the main network according to the state s _t action a under the parameter θ;

S4. Execute action at and calculate reward _rt and state s _{t +1} ; store (s _t , at , r _t , s _{t +1} ) in the first experience pool;

如果

将(s_t,a_t,r_t,s_t+1)存储到第二经验池中；S5. Calculate the average reward of current historical experience when t>0

if

store (s _t , at , r _t , s _{t +1} ) into the second experience pool;

S6. Generate a random number P in the (p ₁ , p ₂ ) interval, select the first experience pool with 1-P as the probability, select the second experience pool with P as the probability, and randomly sample B records from the selected experience pool , the parameter θ of the main network is trained by minimizing the loss function; p ₁ , p ₂ are the preset lower and upper limits of the interval, 0<p ₁ <p ₂ <1;

The loss function is:

where (s _i , a _i , r _i , s _i+1 ) are the records randomly sampled in the selected experience pool, γ is the discount factor, max _a′ Q′(s _i+1 , a′, θ′) Represents the maximum Q value output by the target value network in the input state s _i+1 , max _a Q(s _i , a, θ) represents the maximum Q value output by the main network in the input state s _i ;

S7. Let t increase by one, if mod(t, C) is 0, update the parameter θ' of the target value network to the parameter θ of the main network; mod is the remainder operation, and C is the preset parameter update time step; The current road condition information is updated s _t , and jumps to step S3 to continue the execution.

Further, the step S6 adopts the gradient descent method to minimize the loss function to obtain the parameters of the main network.

Further, when the traffic intersection is an intersection, the state values in the state space of the main network and the target value network are [n ₁ ,m ₁ ,n ₂ ,m ₂ ,n ₃ ,m ₃ ,n ₄ ,m ₄ ], where n _j is the number of vehicles in the j-th incoming lane in the intersection, m _j is the number of vehicles in the j-th outgoing lane; j=1,2,3,4.

Further, the action values in the action space of the main network and the target value network have three values, respectively: ac ₁ : the current phase duration plus T seconds; ac ₂ : the current phase duration minus T seconds; ac ₃ : the current phase duration The phase duration does not change. In the present invention, T is 5 seconds.

Further, in the present invention, the interval lower limit p ₁ =0.7 of the random number P is generated, and the interval upper limit p ₂ =0.9.

Further, the reward function value is:

Further, the first experience pool and the second experience pool both use queues with fixed capacity to store records.

Further, the step S5 calculates the average reward of current historical experience

Beneficial effects: The traffic signal control method disclosed in the present invention adopts a combination of dual experience pools and DQN, wherein the dual experience pool mechanism can make the network parameter training converge quickly, and the obtained traffic signal control strategy can be quickly optimized, so as to better realize the control of traffic lights. Intelligent regulation.

Description of drawings

FIG. 1 is a flowchart of a traffic signal light control method disclosed in the present invention;

2 is a schematic diagram of an intersection in an embodiment;

FIG. 3 is a schematic diagram of the network architecture of the present invention.

Detailed ways

The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments.

The invention discloses a traffic signal light control method based on double experience pool DQN, as shown in FIG. 1 , including steps:

S1. Establish a traffic light control main network and a target value network based on the DQN algorithm; the traffic light control main network and the target value network have the same structure, the input is a state value, and the output is a Q for performing various actions under the input state value value; the state space of the main network and the target value network is the vector formed by the number of vehicles in each lane of the traffic intersection, the action space is the vector formed by the control operations on the phases of all current traffic lights at the traffic intersection, and the reward function is all traffic intersections. The difference between the number of vehicles in the incoming lane and the number of vehicles in the outgoing lane;

即进车道上车辆数量与出车道上车辆数量之差。网络和目标值网络的动作空间中的动作值有三种取值，分别为：ac₁：当前相位时长加T秒；ac₂：当前相位时长减T秒；ac₃：当前相位时长不变，即按照预设的交通信号灯相位变化来改变当前相位的状态。When the traffic intersection is an intersection, as shown in intersection A in Figure 2, its four intersections have lanes entering and leaving the intersection. Lanes away from the intersection, the state values in the state space of the main network and the target value network are [n ₁ ,m ₁ ,n ₂ ,m ₂ ,n ₃ ,m ₃ ,n ₄ ,m ₄ ], where n _j is The number of vehicles in the j-th incoming lane in the intersection, m _j is the number of vehicles in the j-th outgoing lane; j=1,2,3,4. This data can be captured by sensors or cameras placed on the road in all directions. The reward function is:

That is, the difference between the number of vehicles in the incoming lane and the number of vehicles in the outgoing lane. The action value in the action space of the network and the target value network has three values, namely: ac ₁ : the current phase duration plus T seconds; ac ₂ : the current phase duration minus T seconds; ac ₃ : the current phase duration is unchanged, that is Change the state of the current phase according to the preset traffic light phase change.

S2. Randomly initialize the parameter θ of the main network, initialize the parameter θ′ of the target value network to θ, initialize the time step t=0, collect the road condition information of the traffic intersection, establish the initial state value s _t , and initialize

S3. Input s _t into the main network, and select the action a _t that makes Q(s _t , a; θ) take the maximum value as the control operation for the traffic lights at the current time, namely: a _t =argmax _a Q(s _t , a; θ), where Q(s _t , a; θ) represents the Q value output by the main network according to the state s _t action a under the parameter θ;

S4. Execute action at and calculate reward _rt and state s _{t +1} ; store (s _t , at , r _t , s _{t +1} ) in the first experience pool;

如果

将(s_t,a_t,r_t,s_t+1)存储到第二经验池中；S5. Calculate the average reward of current historical experience when t>0

if

store (s _t , at , r _t , s _{t +1} ) into the second experience pool;

即根据上一时间步的平均奖励

和当前时间步数t和奖励r_t来计算。Current historical experience average reward

i.e. according to the average reward of the previous time step

and the current time step t and reward r _t to calculate.

In the present invention, both the first experience pool and the second experience pool use a queue with a fixed capacity to store records. When the queue is full, the record at the head of the queue is deleted, and the new record is stored at the end of the queue to update the experience pool.

S6. Generate a random number P in the (p ₁ , p ₂ ) interval, select the first experience pool with 1-P as the probability, and select the second experience pool with P as the probability; randomly sample B records from the selected experience pool , the parameter θ of the main network is trained by minimizing the loss function; p ₁ , p ₂ are the preset lower and upper limits of the interval, 0<p ₁ <p ₂ <1. In this embodiment, p ₁ =0.7, p ₂ =0.9, that is, the probability of selecting the second experience pool is greater than that of the first experience pool. Since the record reward in the second experience pool is larger, its performance is better than that of the first experience pool, and the training with the records in the second experience pool can speed up the convergence speed. At the same time, the selection of the first experience pool with a lower probability (1-P) is reserved in order to reduce the probability of the network entering overfitting.

The loss function is:

where (s _i , a _i , r _i , s _i+1 ) are the records randomly sampled in the selected experience pool, γ is the discount factor, max _a′ Q′(s _i+1 , a′, θ′) Represents the maximum Q value output by the target value network in the input state s _i+1 , max _a Q(s _i , a, θ) represents the maximum Q value output by the main network in the input state s _i ;

In this embodiment, the gradient descent method is used to minimize the loss function to obtain the parameters of the main network. As shown in FIG. 3 , it is a schematic diagram of the network architecture of the present invention.

S7. Let t increase by one, if mod(t, C) is 0, update the parameter θ' of the target value network to the parameter θ of the main network; mod is the remainder operation, and C is the preset parameter update time step; The duration between time t-1 and time t and the value of C can control the frequency of updating the target value network parameters; update s _t according to the current road condition information, and jump to step S3 to continue execution.

The form of double experience pools adopted in the present invention accelerates the convergence speed of the network during DQN training, thereby better relieving traffic congestion and promoting the development of intelligent transportation and deep reinforcement learning.

Claims (9)

1. a traffic signal control method based on double experience pool DQN, is characterized in that, comprises the steps: S1. Establish a traffic light control main network and a target value network based on the DQN algorithm; the traffic light control main network and the target value network have the same structure, the input is a state value, and the output is a Q for performing various actions under the input state value value maximum value, and the action corresponding to the Q value maximum value; the state space of the main network and the target value network is the vector formed by the number of vehicles in each lane of the traffic intersection, and the action space is the current phase of all traffic lights at the traffic intersection. The vector formed by the control operation of , and the reward function is the difference between the number of vehicles in all incoming lanes and the number of vehicles in outgoing lanes at the traffic intersection; S2. Randomly initialize the parameter θ of the main network, initialize the parameter θ′ of the target value network to θ, initialize the time step t=0, collect the road condition information of the traffic intersection, establish the initial state value s _t , and initialize

S3. Input s _t into the main network, and select the action a _t that makes Q(s _t , a; θ) take the maximum value as the control operation for the traffic lights at the current time, namely: a _t =argmax _a Q(s _t , a; θ), where Q(s _t , a; θ) represents the Q value output by the main network according to the state s _t action a under the parameter θ; S4. Execute action at and calculate reward _rt and state s _{t +1} ; store (s _t , at , r _t , s _{t +1} ) in the first experience pool; S5. Calculate the average reward of current historical experience when t>0

if

store (s _t , at , r _t , s _{t +1} ) into the second experience pool; S6. Generate a random number P in the (p ₁ , p ₂ ) interval, select the first experience pool with 1-P as the probability, select the second experience pool with P as the probability, and randomly sample B records from the selected experience pool , the parameter θ of the main network is trained by minimizing the loss function; p ₁ , p ₂ are the preset lower and upper limits of the interval, 0<p ₁ <p ₂ <1; The loss function is:

where (s _i , a _i , r _i , s _i+1 ) are the records randomly sampled in the selected experience pool, γ is the discount factor, max _a′ Q′(s _i+1 , a′, θ′) Represents the maximum Q value output by the target value network in the input state s _i+1 , max _a Q(s _i , a, θ) represents the maximum Q value output by the main network in the input state s _i ; S7. Let t increase by one, if mod(t, C) is 0, update the parameter θ' of the target value network to the parameter θ of the main network; mod is the remainder operation, and C is the preset parameter update time step; The current road condition information is updated s _t , and jumps to step S3 to continue the execution. 2 . The traffic signal light control method based on the double empirical pool DQN according to claim 1 , wherein, in the step S6 , a gradient descent method is used to minimize the loss function to obtain the parameters of the main network. 3 . 3. The traffic signal light control method based on double experience pool DQN according to claim 1, is characterized in that, when the traffic intersection is an intersection, the state value in the state space of the main network and the target value network is [n ₁ ,m ₁ ,n ₂ ,m ₂ ,n ₃ ,m ₃ ,n ₄ ,m ₄ ], where n _j is the number of vehicles in the j-th incoming lane at the intersection, and m _j is the vehicle in the j-th outgoing lane Quantity; j=1,2,3,4. 4. the traffic light control method based on double experience pool DQN according to claim 1 is characterized in that, the action value in the action space of described main network and target value network has three kinds of values, respectively: ac ₁ : Current phase duration plus T seconds; ac ₂ : Current phase duration minus T seconds; ac ₃ : Current phase duration unchanged. 5 . The traffic signal control method based on the double empirical pool DQN according to claim 1 , wherein p ₁ =0.7, p ₂ =0.9. 6 . 6. the traffic light control method based on double experience pool DQN according to claim 3, is characterized in that, reward function value is:

7 . The traffic light control method based on dual experience pools DQN according to claim 1 , wherein the first experience pool and the second experience pool both use queues with fixed capacity to store records. 8 . 8. The traffic light control method based on double experience pool DQN according to claim 1, wherein the step S5 calculates the average reward of current historical experience

9 . The traffic light control method based on double empirical pool DQN according to claim 4 , wherein T is 5 seconds. 10 .

Images