CN114802248A - Automatic driving vehicle lane change decision making system and method based on deep reinforcement learning - Google Patents

Abstract

The invention relates to the technical field of automatic driving control, and discloses a deep reinforcement learning-based decision-making system and method for changing lanes of an automatic driving vehicle. The lane strategy module uses the collected data information of the target vehicle and the collected operation data of the interfering vehicles near the target vehicle to analyze to obtain the current safe and automatic lane changing strategy of the target vehicle, so as to realize the fast and safe lane change. The invention has the beneficial effects of improving the accuracy of the automatic lane changing strategy of the automatic driving vehicle, ensuring the safety of the vehicle during the lane changing process, and reducing road congestion and the occurrence of collision accidents.

Description

Lane-changing decision-making system and method for autonomous vehicles based on deep reinforcement learning

technical field

The invention relates to the technical field of automatic driving control, in particular to a lane change decision system and method for an automatic driving vehicle based on deep reinforcement learning.

Background technique

In recent years, the world has paid special attention to autonomous driving, which is considered to be an important technology to alleviate traffic congestion, reduce traffic accidents and environmental pollution. At present, some autonomous driving has undergone large-scale road tests, such as Google Autopilot and Apple Autopilot. . According to research, in the current traffic accidents, more than 30% of the road accidents are caused by unreasonable lane-changing behavior. Therefore, the research on the lane-changing assistance technology in the intelligent assisted driving technology is particularly important. All rule-based algorithms are faced with insufficient data, so that the model cannot fully cope with the problem of infinite scenarios during the lane changing process of autonomous driving vehicles, resulting in the failure of lane changing or affecting the safety during the lane changing process.

In the prior art, there is a lane change decision control method for an automatic driving vehicle based on hierarchical reinforcement learning, which belongs to the technical field of automatic driving control. It solves the problem of poor safety/low efficiency in the existing automatic driving process. The invention uses the speed in the actual driving scene of the automatic driving vehicle and the relative position and relative speed information of the vehicle in the surrounding environment to establish a decision-making neural network with three hidden layers, and uses the lane-changing safety reward function to adjust the decision-making neural network. The network is trained to fit the Q evaluation function to obtain the action with the largest Q evaluation; using the speed in the actual driving scene of the autonomous vehicle and the relative position information of the surrounding environment vehicle and the reward function corresponding to the car-following or lane-changing action, establish The acceleration decision-making model of deep Q-learning obtains lane-changing or car-following acceleration information. When changing lanes, a 5th-degree polynomial curve is used to generate a reference lane-changing trajectory. The present invention is suitable for automatic driving lane change decision and control.

Although this scheme can perform simulated training for lane changing and determine the reward function for the scene and surrounding environment to ensure the training results of automatic driving lane changing, it still has insufficient data and lane changing scenarios for specific lane changing scenarios. The limited problem eventually leads to the low accuracy and safety of its lane changing.

SUMMARY OF THE INVENTION

The present invention aims to provide a lane-changing decision system and method for an autonomous driving vehicle based on deep reinforcement learning, so as to improve the accuracy of the lane-changing strategy of the autonomous driving vehicle and ensure the safety of lane-changing.

In order to achieve the above purpose, the present invention adopts the following technical solutions: a deep reinforcement learning-based automatic driving vehicle lane change decision system, comprising a processor module, and a data acquisition module, a data analysis module and a lane change strategy module respectively connected with the processor module ;

a data collection module, used for collecting data information of the target vehicle and collecting operation data of interfering vehicles near the target vehicle, and then forming a first data set and sending the first data set to the data analysis module;

a data analysis module, used for analyzing and processing the first data set, and obtaining the lane-changing scene and lane-changing data of the autonomous driving vehicle;

a lane-changing strategy module, configured to generate a first lane-changing strategy according to the obtained lane-changing scene and lane-changing data, and send the first lane-changing strategy to the processor module;

The processor module includes a data storage unit and a lane change execution unit, the data storage unit is used to store a first lane change strategy; the lane change execution unit is used to obtain a rule-based algorithm according to the first lane change policy. The lane change trajectory executes the model and controls the autonomous vehicle for lane changes.

The principle and advantages of this scheme are: in practical application, the deep reinforcement learning method is used to train and try the lane-changing model based on the rule-based lane-changing model, and the Actor-Critic algorithm is used to continuously optimize the lane-changing strategy of the autonomous vehicle. , so that the self-driving car can accurately deal with the problem of infinite scenarios in the lane-changing process, thereby improving the accuracy of the automatic lane-changing strategy of the self-driving vehicle, ensuring the safety of the vehicle during the lane-changing process, and reducing road congestion and collision accidents. occur. Compared with the prior art, the advantage of the present invention lies in that the established deep learning model can conduct a more comprehensive and accurate test on the lane changing strategy of the automatic driving technology, guide the automatic driving vehicle to complete the lane change quickly and safely, and the obtained lane changing trajectory can be It is suitable for scenarios where autonomous vehicles change lanes, so that vehicles can make correct and reasonable responses to new traffic scenarios under the condition of limited data volume to ensure driving safety.

Preferably, as an improvement, the first data set includes surrounding vehicle information at the current moment, surrounding road information at the current moment, surrounding road information at the next moment, surrounding vehicle information at the next moment, and vehicle information of the vehicle.

Beneficial effects: By collecting the surrounding vehicle information and road information, the data to be referenced for lane changing can be accurately provided, so as to make a lane-changing safety judgment, avoid collision safety accidents between the target vehicle and surrounding interfering vehicles during the lane-changing process, and can also extremely To a great extent, the accuracy of the lane-changing strategy of this lane-changing model is improved.

Preferably, as an improvement, the first data set is analyzed and processed by using a preset analysis algorithm to perform infinite scene exploration and analysis on the limited first data set, and before obtaining the corresponding lane changing scene, analyze the analysis process. Perform deep reinforcement learning.

Beneficial effect: analyzing and processing the collected data through this process can effectively overcome the defect of insufficient data volume at present, and can analyze the lane changing model and the lane changing strategy in infinite scenarios on the limited data volume, which can greatly Improve the diversity of lane-changing scenarios and the accuracy of lane-changing strategies, and provide reliable guarantees for the automatic and safe lane-changing behavior of autonomous vehicles, thereby ensuring driving safety.

Preferably, as an improvement, the preset analysis algorithm is the Actor-Critic algorithm; the deep reinforcement learning is to use the Markov decision process to describe the analysis process to form a six-dimensional tuple M=(S, A, P, r, ρ, γ), where S is the state space, the state space is the set of all states; A is the action space, the action space is the set of all actions; P is the state transition probability; r is the state transition The reward function of the process; γ is the discount coefficient in the state transition process.

Beneficial effect: By using the Actor-Critic algorithm to strengthen the interaction between the learning data and the environment, and learning is not one-sided to obtain the optimal strategy for single-step decision-making, but the pursuit of long-term cumulative rewards obtained by interacting with the environment, so that the replacement of self-driving vehicles can be achieved. The lane strategy training results are more accurate and ensure the safety of lane changing.

式中，v为车辆实时速度，v_min为车辆训练过程中采用的最小速度，v_max为车辆训练过程中采用的最大速度，a为对于换道过程中速度奖励值，b是对车辆发生碰撞的碰撞惩罚值，collision为仿真环境对于车辆发生碰撞的反馈结果。Preferably, as an improvement, the reward function is

In the formula, v is the real-time speed of the vehicle, v _min is the minimum speed used in the vehicle training process, v _max is the maximum speed used in the vehicle training process, a is the speed reward value for the lane-changing process, and b is the collision with the vehicle. The collision penalty value, collision is the feedback result of the simulation environment for the collision of the vehicle.

Beneficial effects: In the course of training the lane-changing model, the reward function is used to accumulate rewards in the training process, and the parameter data of the lane-changing model is corrected, thereby improving the accuracy of the lane-changing model.

Preferably, as an improvement, the data analysis module uses a deep reinforcement learning method to train and try the lane-changing model based on the rule-based lane-changing model, and finally verifies the model.

Beneficial effects: For the lane-changing model obtained after multiple reinforcement learning, in order to ensure the accuracy, the method of deep reinforcement learning is used to train and try, so as to complete the correction of the lane-changing model and ensure the correctness of the parameters of the lane-changing model. Self-driving vehicles provide more precise lane-changing strategy services.

其中θ_i为规划步长起点的航向角，

为终点横向坐标，x_n为车辆n的纵向位置，y_n为车辆n的横向位置。Preferably, as an improvement, the lane-changing strategy module uses a rule-based trajectory planning algorithm to assist calculation when generating the lane-changing strategy, and the rule-based trajectory planning algorithm is expressed as

where θ _i is the heading angle of the starting point of the planning step,

is the lateral coordinate of the end point, x _n is the longitudinal position of the vehicle n, and y _n is the lateral position of the vehicle n.

Beneficial effect: The lane-changing trajectory is planned in this way, so as to ensure the safety of the self-driving vehicle during the lane-changing process, without colliding with other vehicles, and can effectively reduce the lane-changing time of the vehicle, improve the traffic efficiency of road vehicles, and effectively slow down road congestion.

其中a_t，r_t，s_t+1，a_t+1，r_t+1，...～π表示轨迹来自策略π与环境的交互。Preferably, as an improvement, when using Markov decision process to describe the analysis process, the state value function is defined as follows:

where at , r _t , s _{t +1} , at _t+1 , r _t+1 , ... ∼ π represent the trajectory from the interaction of policy π with the environment.

Beneficial effects: By continuously updating road environment data, new lane-changing decision-making data is provided, and new data is continuously updated, thereby improving the real-time and accuracy of the lane-changing model and ensuring the lane-changing safety of autonomous vehicles.

The present invention also provides a lane-changing decision method for an autonomous driving vehicle based on deep reinforcement learning, comprising the following steps:

Step S1, collecting data information of the target vehicle and operation data of interfering vehicles near the target vehicle;

Step S2, using the Actor-Critic algorithm to analyze and process the collected data, and combining with the Markov decision process to describe the reinforcement learning problem, to obtain the lane-changing scene and lane-changing data of the autonomous driving vehicle;

Step S3, using the rule-based trajectory planning algorithm to calculate the lane-changing trajectory and using the reward function to correct the test process, and finally obtain the lane-changing strategy;

In step S4, the output result of the model is verified by using the automatic driving simulation environment highway_env and the dynamics-based simulation software CarSim.

Beneficial effects: the method is used to realize the test of the automatic lane-changing model of the self-driving vehicle and the generation of the lane-changing strategy, so as to ensure the safety of the lane-changing of the vehicle, improve the efficiency of the lane-changing, reduce the road congestion, and at the same time, it can greatly improve the Road traffic safety.

Preferably, as an improvement, the output result of this model is verified as: using the automatic driving simulation environment to test the vehicle lane changing strategy using the highway scene, and using the two statistics of mean absolute error and mean absolute relative error to test the model Perform error statistics.

Beneficial effects: By verifying the output results of the model, the decision-making accuracy of the model for the lane-changing strategy can be ensured to the greatest extent, thereby avoiding the collision between lane-changing vehicles and surrounding vehicles, and improving the safety and efficiency of lane-changing. Improve the efficiency of road traffic and reduce congestion.

Description of drawings

FIG. 1 is a system schematic diagram of Embodiment 1 of a lane change decision system for an autonomous driving vehicle based on deep reinforcement learning of the present invention.

FIG. 2 is a schematic diagram of an LSTM neural network according to Embodiment 1 of the deep reinforcement learning-based lane-changing decision-making system for automatic driving vehicles of the present invention.

FIG. 3 is a schematic diagram of the structure of an LSTM neural network according to Embodiment 1 of the deep reinforcement learning-based lane-changing decision-making system for automatic driving vehicles of the present invention.

FIG. 4 is a schematic flowchart of Embodiment 1 of a method for lane changing decision-making of an autonomous driving vehicle based on deep reinforcement learning of the present invention.

FIG. 5 is a schematic diagram of a change in revenue of Embodiment 1 of the method for decision-making on lane changing of an autonomous driving vehicle based on deep reinforcement learning of the present invention.

Detailed ways

The following is further described in detail by specific embodiments:

The symbols in the drawings in the description include: processor module 1 , data acquisition module 2 , data analysis module 3 , lane change strategy module 4 , data storage unit 5 , and lane change execution unit 6 .

Example 1:

This embodiment is basically as shown in FIG. 1 : a lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning includes a processor module 1 , and a data acquisition module 2 , a data analysis module 3 and a change-over module respectively connected to the processor module 1 . Road strategy module 4;

The data collection module 2 is used to collect the data information of the target vehicle and the operation data of the interfering vehicles near the target vehicle, and then form a first data set and send the first data set to the data analysis module 3;

The data analysis module 3 is used for analyzing and processing the first data set, and obtaining the lane-changing scene and lane-changing data of the autonomous driving vehicle;

A lane-changing strategy module 4 is used to generate a first lane-changing strategy according to the obtained lane-changing scene and lane-changing data, and send the first lane-changing strategy to the processor module 1;

The processor module 1 includes a data storage unit 5 and a lane change execution unit 6, the data storage unit 5 is used to store the first lane change strategy; The lane-changing trajectory of the model executes the model and controls the autonomous vehicle to make a lane change.

The data collected by the data collection module 2 includes the information of surrounding vehicles at the current moment, the surrounding road information at the current moment, the surrounding road information at the next moment, the surrounding vehicle information at the next moment and the vehicle information of the vehicle;

As shown in Figure 2, the Actor-Critic algorithm is used to process the first data set, and the Markov decision process is used to describe the reinforcement learning problem, forming a six-dimensional tuple M=(S, A, P, r, ρ, γ), where S is the state space, that is, the set of all states; A is the action space, that is, the set of all actions; P is the state transition probability; r is the reward function of the state transition process; γ is the discount coefficient in the state transition process . Under the condition of Markov decision process M and policy π, the state value function is defined as follows,

Among them, at _t , r _t , s _t+1 , at _t+1 , r _t+1 , ... ~ π represent the trajectory from the interaction between the policy π and the environment, the above formula indicates that starting from the state st = s, the agent The expected cumulative reward for interacting with the environment using policy π, similarly, the state action value function can also be defined:

Indicates that starting from state st, after performing action at, the agent uses the policy π to interact with the expected cumulative reward, and the state value function and the state action value function can be converted to each other. When the policy π is a probabilistic strategy,

For any probabilistic policy π, we have the following Bellman expectation equation,

When the agent adopts the strategy π, performs the action at from the state st to the state st+1 and obtains the reward rt, it directly performs the following dynamic programming update,

V _π (S _t )=V _π (S _t )+α(r _t +γV(S _t+1 )-V(S _t ))

Let r _t +γV(s _t+1 )-V(s _t )=TD-error

After obtaining, the neural network will perform backpropagation. Actor is based on the policy gradient, and the policy is parameterized as a neural network, which is represented by θ. The direction of θ iteration is to maximize the expectation of the periodic reward. The objective function is expressed as:

Among them, τ represents a sampling period, π _θ (τ) represents the probability of sequence occurrence, and the gradient of J(θ) can be obtained:

but:

Finally update the neural network parameters:

As shown in Figure 3, the LSTM neural network is mainly composed of input layer, hidden layer, and output layer neurons. The hidden layer neurons mainly have three gate structures and one state: forget gate, input gate, output gate, and cell state. . For the training and correction of the lane-changing strategy in the later stage, it is first necessary to decide which old data needs to be discarded from the cell state when new data is passed into the long-term and short-term memory network, and this part is determined by the forget gate, which is a sigmoid function. layer,

where W _f is the weight matrix of the forget gate, h _t-1 is the cell state at time t-1, x _t is the environmental input data, and b _f is the bias term of the forget gate.

Then it goes through a sigmoid function layer, that is, the input gate will decide which values need to be updated, and then a tanh function layer will create a vector as a candidate value to add to the cell state:

是准备用以更新的数据矩阵，W_c是准备用以更新的数据的权重矩阵。where b _i is the bias term of the input gate,

is the data matrix to be updated and W _c is the weight matrix of the data to be updated.

Then update the cell state at the previous moment, first remove the information we decided on the forget gate from the cell state, and then add the candidate value calculated by the input gate at the ratio of the update of each state value:

Finally, the part to be output is decided. The output is appropriately processed on the basis of the cell state, that is, a sigmoid function layer is used to determine which parts need to be updated, and then it will be processed by a tanh function, and the sigmoid in the forget gate will be processed. The outputs of the layers are multiplied to determine the output:

O _t =σ(W _o [h _t-1 , x _t ]+b _o )

In the formula, W _o is the weight matrix of the output gate, and b _o is the bias term of the output gate.

The cell state before improvement is:

s _t =tanh(W _g [h _t-1 , x _t ]+b _g )·σ(W _i [h _t-1 , x _t ]+b _i )+s _t-1 ·σ(W _f [h _t-1 , x _t ]b _f ))

The output is:

h _t =tanh(s _t )·σ(W _o [h _t-1 , x _t ]+b _o )

Then the error between the value of the pth output and the true value is:

A cubic polynomial with a simpler form is selected as the rule-based trajectory planning algorithm, and its expression is as follows:

为终点横向坐标，xn为车辆n的纵向位置，yn为车辆n的横向位置。In the formula, θ _i is the heading angle of the starting point of the planning step,

is the lateral coordinate of the end point, xn is the longitudinal position of the vehicle n, and yn is the lateral position of the vehicle n.

As shown in FIG. 4 , the present invention also provides a deep reinforcement learning-based decision-making method for automatic driving vehicle lane change applied in the above system, comprising the following steps:

Step S1, collecting data information of the target vehicle and operation data of interfering vehicles near the target vehicle;

Step S2, using the Actor-Critic algorithm to analyze and process the collected data, and combining with the Markov decision process to describe the reinforcement learning problem, to obtain the lane-changing scene and lane-changing data of the autonomous driving vehicle;

Step S3, using the rule-based trajectory planning algorithm to calculate the lane-changing trajectory and using the reward function to correct the test process, and finally obtain the lane-changing strategy;

In step S4, a rule-based lane-changing trajectory execution model is obtained according to the lane-changing strategy, and the output results of the model are verified by using the automatic driving simulation environment highway_env and the dynamics-based simulation software CarSim.

In the reinforcement learning process, the vehicle lane changing strategy is tested using the highway scene in the autonomous driving simulation environment, and the error statistics of the model are carried out with the two statistics of mean absolute error and mean absolute relative error.

In the formula, N represents the number of test data samples, d _{r, i} represents the nominal value of the i-th vehicle, and d _{s, i} represents the predicted value of the i-th vehicle.

As shown in Fig. 5, the income of this model changes with the increase of training times. It can be seen that the training income increases rapidly with the increase of training times. When the training exceeds 2000 times, the income value is stable and tends to converge.

Using this system, the vehicle lane changing strategy can be updated based on the current actual data, so as to provide automatic lane changing strategies and methods for autonomous vehicles, so as to ensure the smooth progress of the lane changing process and avoid collisions with surrounding vehicles. , improving the safety and efficiency of lane changing, further ensuring the smoothness of road traffic and reducing urban traffic congestion.

The specific implementation process of this embodiment is as follows:

The first step is to use the data collection module 2 to collect the data information of the target vehicle and the operation data of the interfering vehicles near the target vehicle, including the surrounding vehicle information at the current moment, the surrounding road information at the current moment, the surrounding road information at the next moment, and the surrounding road information at the next moment. The vehicle information and the vehicle information of the vehicle are then formed into a first data set and sent to the data analysis module 3 .

In the second step, after the data analysis module 3 receives the first data set, it analyzes and processes the first data set, uses the Actor-Critic algorithm to process the first data set, and uses the Markov decision process to describe the reinforcement learning problem, A six-dimensional tuple M=(S, A, P, r, ρ, γ) is formed, and then the lane-changing scene and lane-changing data of the autonomous vehicle are obtained.

In the third step, the lane-changing strategy module 4 generates a first lane-changing strategy according to the obtained lane-changing scene and lane-changing data, and sends the first lane-changing strategy to the processor module 1, and the data storage unit 5 of the processor module 1 receives the The first lane-changing strategy is stored, and the lane-changing execution unit 6 obtains a rule-based lane-changing trajectory execution model according to the first lane-changing strategy, and controls the automatic driving vehicle to change lanes.

The fourth step is to train and try the lane-changing trajectory execution model, and finally use the automatic driving simulation environment highway_env and the dynamics-based simulation software CarSim to verify the output of this model, and use the automatic driving simulation environment to use the highway scene to change the vehicle. The lane strategy is tested, and the two statistics of mean absolute error and mean absolute relative error are used to perform error statistics on the model. The resulting lane-changing trajectory and speed change smoothly in the dynamic simulation, and can keep a small error with the target trajectory. It is tracked under conditions, and the vehicle has good driving stability.

In recent years, the world has paid special attention to autonomous driving, which is considered to be an important technology to alleviate traffic congestion, reduce traffic accidents and environmental pollution. At present, some autonomous driving has undergone large-scale road tests, such as Google Autopilot and Apple Autopilot. . According to research, in the current traffic accidents, more than 30% of the road accidents are caused by unreasonable lane-changing behavior. Therefore, the research on the lane-changing assistance technology in the intelligent assisted driving technology is particularly important. All rule-based algorithms are faced with insufficient data, so that the model cannot fully cope with the problem of infinite scenarios during the lane changing process of autonomous driving vehicles, resulting in the failure of lane changing or affecting the safety during the lane changing process.

In this scheme, a deep reinforcement learning model is established, and the shortcomings of the current mainstream lane changing models are comprehensively considered, so a rule-based training model is introduced, and the problem of how the neural network can adapt to and learn more driving skills under limited data is raised. In order to solve the problem, the collected data information of the target vehicle and the operation data of the interfering vehicles near the target vehicle are used for statistical analysis and processing, the Actor-Critic algorithm is used to process the first data set, and the Markov decision process is used. To describe the reinforcement learning problem, form a six-dimensional tuple M=(S, A, P, r, ρ, γ), and use the deep reinforcement learning method to train and try the lane changing model, and finally use the autonomous driving simulation environment highway_env and the dynamics-based simulation software CarSim to verify the output results of this model. Compared with the past, this model has more choices of lane-changing scenarios and lane-changing strategy data, and the lane-changing strategy provided by this scheme is more efficient than before. Therefore, compared with the previous model, the travel time of lane changing in this scheme is less. It is found that the travel time is reduced by more than 50%, and the probability of collision is reduced by more than 75%, thus effectively ensuring the lane changing strategy of this model. Authenticity and accuracy, guide the autonomous vehicle to quickly and safely complete the lane change, and the obtained lane change trajectory can be applied to the scene of the autonomous vehicle changing lane, so that the vehicle can make decisions on new traffic scenarios under the condition of limited data volume. A reasonable response not only provides a large amount of data for the backup lane changing strategy, but also feeds back the monitoring data of the surrounding environment during the lane changing process to the model in real time, so as to conduct timely interference and tracking of the lane changing strategy during the lane changing process. It can improve the safety of lane changing of autonomous vehicles, effectively reduce road traffic congestion and accident rate, and ensure the safety of drivers and passengers.

Embodiment 2:

This embodiment is basically the same as the first embodiment, the difference is that the system also includes a display module. During the process of analyzing and establishing the lane-changing model, the display module is used to display the lane-changing trajectory and vehicle operation data of the autonomous driving vehicle in real time, which can be more intuitive. Accurately know the specific situation of the vehicle lane change.

The specific implementation process of this embodiment is the same as that of the first embodiment, and the difference is:

The fourth step is to train and try the lane-changing trajectory execution model, and finally use the automatic driving simulation environment highway_env and the dynamics-based simulation software CarSim to verify the output of this model, and use the automatic driving simulation environment to use the highway scene to change the vehicle. Test the lane strategy, and use the two statistics of mean absolute error and mean absolute relative error to conduct error statistics on the model, and finally obtain an accurate automatic driving lane change strategy. During the entire training and verification process, the display module is used to display the automatic driving vehicle in real time. lane change trajectory and vehicle operation data.

Provides the function of displaying lane-changing trajectory and lane-changing data, so that the lane-changing test of autonomous vehicles is more accurate, and the operator can understand the entire test process more intuitively, which is convenient for correcting unqualified places and improving lane-changing. Test efficiency of the test.

Embodiment three:

This embodiment is basically the same as the first embodiment, the difference is that: the system further includes a lane-changing trajectory correction module, which is used for automatically changing lanes when the current lane-changing trajectory cannot be safely completed due to changes in road condition information during the lane-changing process. The lane-changing trajectory of the driving vehicle is corrected, so that the self-driving vehicle can successfully complete the lane-changing, improving the safety guarantee of lane-changing, and at the same time, it can also guarantee the driving safety to a great extent.

The specific implementation process of this embodiment is the same as that of the first embodiment, and the difference is:

In the third step, the lane-changing strategy module 4 generates a first lane-changing strategy according to the obtained lane-changing scene and lane-changing data, and sends the first lane-changing strategy to the processor module 1, and the data storage unit 5 of the processor module 1 receives the And store the first lane-changing strategy, and the lane-changing execution unit 6 obtains a rule-based lane-changing trajectory execution model according to the first lane-changing strategy, and controls the autonomous driving vehicle to change lanes; The situation causes the road condition information to change, so that when the lane change cannot be completed safely according to the current lane change trajectory, the lane change trajectory correction module corrects the lane change trajectory of the automatic driving vehicle in real time, so that the automatic driving vehicle can successfully complete the lane change.

Considering the irregular changes of the road environment around the autonomous vehicle and the emergency braking or sudden acceleration of the surrounding vehicles, when the autonomous vehicle changes lanes according to the current lane-changing strategy, if the road information changes, the lane-changing trajectory will be used for correction. The module intervenes to correct the lane change trajectory in real time, so as to ensure the safe and smooth progress of the lane change.

The above are only examples of the present invention, and common knowledge such as well-known specific technical solutions and/or characteristics in the solutions are not described too much here. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the technical solution of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the implementation of the present invention. effect and the applicability of the patent. The scope of protection claimed in this application should be based on the content of the claims, and the descriptions of the specific implementation manners in the description can be used to interpret the content of the claims.

Claims (10)

1. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning, is characterized in that: comprising a processor module, and a data acquisition module, a data analysis module and a lane-changing strategy module connected with the processor module respectively; The data collection module is used to collect the data information of the target vehicle and the operation data of the interfering vehicles near the target vehicle, and then form a first data set and send the first data set to the data analysis module; The data analysis module is used to analyze and process the first data set, and obtain the lane-changing scene and lane-changing data of the autonomous driving vehicle; The lane-changing strategy module is configured to generate a first lane-changing strategy according to the obtained lane-changing scene and lane-changing data, and send the first lane-changing strategy to the processor module; The processor module includes a data storage unit and a lane change execution unit, the data storage unit is used to store a first lane change strategy; the lane change execution unit is used to obtain a base The regular lane-changing trajectory executes the model and controls the autonomous vehicle for lane changes. 2 . The lane-changing decision system for autonomous driving vehicles based on deep reinforcement learning according to claim 1 , wherein the first data set includes information on surrounding vehicles at the current moment, surrounding road information at the current moment, and surrounding roads at the next moment. 3 . information, surrounding vehicle information at the next moment, and own vehicle information. 3. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 1, wherein the analyzing and processing the first data set is to use a preset analysis algorithm to analyze the limited first data The collection is used for infinite scene exploration and analysis, and deep reinforcement learning is performed on the analysis process before the corresponding lane changing scene is obtained. 4. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 3, wherein: the preset analysis algorithm is an Actor-Critic algorithm; the deep reinforcement learning is, using Markov The decision-making process describes the analysis process and forms a six-dimensional tuple M=(S, A, P, r, ρ, γ), where S is the state space, which is the set of all states; A is the action space, The action space is the set of all actions; P is the state transition probability; r is the reward function of the state transition process; γ is the discount coefficient in the state transition process. 5. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 4, wherein the reward function is:

In the formula, v is the real-time speed of the vehicle, v _min is the minimum speed used in the vehicle training process, v _max is the maximum speed used in the vehicle training process, a is the speed reward value for the lane-changing process, and b is the collision with the vehicle. The collision penalty value, collision is the feedback result of the simulation environment for the collision of the vehicle. 6. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 1, wherein the data analysis module uses a deep reinforcement learning method to perform a lane-changing model on the basis of the rule-based lane-changing model. Train and try, and finally validate the model. 7. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 1, wherein the lane-changing strategy module utilizes a rule-based trajectory planning algorithm to assist calculation when generating a lane-changing strategy, so that The expression of the rule-based trajectory planning algorithm is

where θ _i is the heading angle of the starting point of the planning step,

is the lateral coordinate of the end point, x _n is the longitudinal position of the vehicle n, and y _n is the lateral position of the vehicle n. 8. The lane-changing decision-making system for autonomous driving vehicles based on deep reinforcement learning according to claim 3, characterized in that: when the analysis process is described using the Markov decision-making process, the state value function is defined as follows,

where a _t , r _t , s _t+1 , at _t+1 , r _t+1 , ... ∼ π represent the trajectory from the interaction of policy π with the environment. 9. A lane-changing decision-making method for autonomous driving vehicles based on deep reinforcement learning, characterized in that it comprises the following steps: Step S1, collecting data information of the target vehicle and operation data of interfering vehicles near the target vehicle; Step S2, using the Actor-Critic algorithm to analyze and process the collected data, and combining with the Markov decision process to describe the reinforcement learning problem, to obtain the lane-changing scene and lane-changing data of the autonomous driving vehicle; Step S3, using the rule-based trajectory planning algorithm to calculate the lane-changing trajectory and using the reward function to correct the test process, and finally obtain the lane-changing strategy; In step S4, a rule-based lane-changing trajectory execution model is obtained according to the lane-changing strategy, and the output results of the model are verified by using the automatic driving simulation environment highway_env and the dynamics-based simulation software CarSim. 10. The deep reinforcement learning-based decision-making method for changing lanes of an autonomous vehicle according to claim 9, wherein the verification of the output result of the model is that the vehicle is changed lanes using a highway scene in an autonomous driving simulation environment. The strategy is tested and the model is errored using two statistics, Mean Absolute Error and Mean Absolute Relative Error.

Images