CN114228690B - Automatic driving vehicle roll control method based on DDPG and iterative control - Google Patents

Abstract

The invention discloses an automatic driving vehicle roll control method based on DDPG and iterative control, which trains a DDPG algorithm on running maps of the automatic driving vehicle in different scenes, and the automatic driving vehicle generates real-time vehicle states through interaction with map environments in different scenes to determine the action behaviors of the vehicle; initializing an action space when action training is carried out, generating state space information by an online strategy network in an actor network, carrying out action output, and adding an action noise to obtain the action space with exploratory property; based on LSTM historic memory and road planning attributes, generating a predicted path of the state of the automatic driving vehicle, adopting a DDPG algorithm to realize tracking control of the path track under the normal running working condition and the extreme running working condition of the automatic driving vehicle, and adopting an iterative control method to realize compensation control of the automatic driving vehicle. The invention avoids the instability problem of the reinforcement learning algorithm under the extreme road environment driving condition of the vehicle, and improves the driving safety and the robustness of the vehicle.

Description

A roll control method for autonomous driving vehicles based on DDPG and iterative control

Technical Field

The present invention belongs to the field of intelligent vehicle control, and in particular relates to a roll control method for an autonomous driving vehicle based on DDPG and iterative control.

Background Art

With the development of artificial intelligence technology, the current autonomous driving technology has made great progress. At present, it focuses on closed campus scenes, such as closed campus scenes, logistics industrial park scenes, etc., especially in the port road environment with structured road characteristics and fewer pedestrians and vehicles, and its application is more common. Autonomous driving vehicles use environmental perception, navigation, map positioning, decision-making, motion planning and trajectory tracking control to realize the intelligence of vehicles. However, when autonomous driving vehicles are in complex weather and complex driving environments such as cross-sea bridges, the harsh climate environment will affect the bridge road conditions, causing the vehicle to turn or skid and roll over. For example, rain, snow, wind, etc. in the climate environment will cause changes in road adhesion coefficients and tire slippage, thereby changing path tracking, lane keeping and vehicle control accuracy. In addition, the bridge road environment may vibrate due to the influence of windy weather, causing vehicle roll, which leads to uncontrollable situations. Therefore, the road vibration characteristics, road angles and aerodynamic characteristics are also considered in the overall control design. When the vehicle has uncertain characteristics such as side slip caused by slippery roads, vehicle yaw characteristics caused by bridge vibration, and vehicle roll dynamics caused by high-speed vehicle performance, control technology is a complex task. Therefore, it is an important key technology to study the driving safety and stability of the roll control of autonomous vehicles in harsh environments. Reinforcement learning is the application scope of artificial intelligence technology. Intelligent agents can explore unknown dynamic environments, try different actions and interact with dynamic environments without any precise vehicle models and given surrounding environments. Reinforcement learning can learn unknown environments and realize complex vehicle dynamics through actions and states of interaction with the environment. It is a new implementation method to adapt to dynamic road environment cognition and complex vehicle dynamics performance. Therefore, the use of reinforcement learning algorithms to realize the vehicle control of the driving environment of autonomous vehicles on cross-sea bridge road conditions will help to achieve the safety of intelligent vehicles and realize the large-scale industrial development of autonomous vehicles.

Summary of the invention

Purpose of the invention: The present invention provides a roll control method for an autonomous driving vehicle based on DDPG and iterative control, which enables the vehicle to drive safely in any cross-sea bridge intersection environment of different complexity levels, and can improve the intelligence level of the vehicle through the instantaneous roll angle.

Technical solution: The present invention provides a roll control method for an autonomous driving vehicle based on DDPG and iterative control, which specifically includes the following steps:

(1) Installing laser radar, vision, millimeter wave radar, ultrasonic radar sensors, positioning systems and inertial navigation systems on autonomous vehicles;

(2) Use visual sensors, positioning systems, and inertial navigation systems to realize the position and map of the vehicle in different scenarios, so as to generate driving maps of autonomous vehicles in different scenarios and realize the environment required for the vehicle's driving trajectory;

(3) Control the steering wheel, accelerator, and pedals respectively, drive on the cross-sea bridge, obtain the corresponding driving trajectories in rainy and snowy days, strong winds and bad weather, and sunny days, and construct a data set;

(4) The DDPG algorithm is trained on driving maps of autonomous vehicles in different scenarios, which is used to monitor the driving status of cross-sea bridges under different levels of complex road conditions in bad weather. Autonomous vehicles interact with map environments in different scenarios to generate real-time vehicle status and determine vehicle action behaviors. During action training, the action space is initialized, the online policy network in the actor network generates state space information, performs action output, and adds an action noise to obtain an exploratory action space.

(5) Based on the LSTM historical memory and road planning attributes, a path for predicting the state of the autonomous driving vehicle is generated. The DDPG algorithm is used to achieve tracking control of the path trajectory of the autonomous driving vehicle under normal driving conditions and extreme driving conditions, and an iterative control method is used to achieve compensation control of the autonomous driving vehicle.

Furthermore, in step (1), the laser radar sensor is used to detect dynamic and static obstacles on the road, including pedestrians, motorcycles and various vehicles, as well as drivable road areas; the visual sensor is used to sense lane lines, pedestrian and vehicle detection, as well as to perform positioning and synchronize map creation; the millimeter wave radar sensor is used to detect the distance between vehicles and pedestrians and moving vehicles; the ultrasonic radar is used to detect the distance between close-range vehicles; and the visual sensor, positioning system and inertial navigation system are used to implement vehicle positioning technology.

Furthermore, the different scenarios described in step (2) include five scenarios: road conditions of a cross-sea bridge in rainy and snowy weather, road conditions of a cross-sea bridge in strong wind and severe weather, road conditions when the bridge vibrates on a sunny day, road conditions of a single vehicle in frequent and changeable weather, and road conditions of multiple vehicles in frequent and changeable weather.

Furthermore, the data set in step (3) includes vehicle speed, driving trajectory, vehicle position, heading angle, slip angle, yaw angular velocity, and roll angle.

Furthermore, the network design of the DDPG algorithm in step (4) is as follows:

Construct an actor network, with vehicle state and environment state as input, and output a vector consisting of steering angle, throttle and brake signals, which correspond to the three neurons in the output layer of the actor strategy network respectively. Set the activation function of throttle and brake to Sigmoid, and the activation function of steering action value to Tanh. The structure of the hidden layer is as follows: the first layer has a convolution size of 7*7, a filter size of 48, a step size of 4, and a total of 200 neurons; the second layer has a convolution size of 5*5, a filter size of 16, a step size of 2, and an activation function of ReLu, with a total of 400 neurons; the third layer adds 100 neurons to the LSTM layer; the fourth layer is a fully connected layer with 128 units; the fifth layer is a fully connected layer with a total of 128 units; the critic network input is the state and action space, after two hidden layers, 200 neurons in the first layer and 400 neurons in the second layer, spliced with the activation function ReLu, and finally the Q value is obtained; define h _i ∈(S _tT ,S _t-T+1 ,…,S _t ), where S _tT and S _t represent the state information at the current moment and the current moment respectively, then the encoded state is: s = f( _hi ; β), then the strategy of the actor network after the change is defined as: a = μ( _hi /β, ^γπ ) + η.

Furthermore, the tracking control of the path trajectory of the autonomous driving vehicle under normal driving conditions in step (5) is implemented as follows:

Normal driving conditions, i.e., road conditions when the bridge vibrates on a sunny day, consider the vehicle's roll, sideslip, and yaw dynamics characteristics, establish a vehicle dynamics model, set vehicle state constraints, determine the lateral stability range, maximum steering angle range, and allowable vehicle control range to prevent roll, so as to reduce the vehicle's lateral offset error:

ω _z-min ≤ω _z ≤ω _z-max ,ω _x-min ≤ω _x ≤ω _x-max ,u _x-min ≤u _x ≤u _x-max ,e _rx-min ≤e _r ≤e _{rx- max}

Where ω _z is the yaw rate; ω _x is the vehicle roll angle; u _x is the steering angle; _er is the lateral tracking offset error;

According to the road state information predicted by LSTM, an objective function considering steering angle, tire adhesion coefficient, roll angle error and path tracking error is constructed. Taking into full account the dynamic constraint of the maximum allowable error of the vehicle, the physical constraint of the vehicle determined by the lateral tracking offset error is determined to reduce the error of tracking and controlling the vehicle:

Where w ₁ ,w ₂ ,w ₃ ,w ₄ are parameter variables; μ _r is the road adhesion coefficient;

Iterative control algorithm is used to realize the compensation control of vehicle roll, and reference vehicle state, reference control input and reference output values are set to ensure the tracking function under vehicle physical constraints and road constraints under multiple constraints, increase the anti-interference of vehicle driving and reduce the model error rate;

According to the vehicle driving conditions, the state space, action space and reward function required by the DDPG algorithm are constructed; the action space mainly includes the steering wheel angle, throttle and brake signals, and the state space includes the vehicle lateral tracking error and its rate of change, the vehicle roll angle error and its rate of change, and the yaw angular velocity error and its rate of change; the reward function is constructed in the case of vibration of the cross-sea bridge, the actual trajectory of the vehicle will change, and the road will have a certain inclination, so the reward function is equal to the cumulative product of the discount factor and the change in speed.

Furthermore, the process of implementing the tracking control of the path trajectory of the autonomous driving vehicle under extreme driving conditions in step (5) is as follows:

Under extreme driving conditions, i.e., bad weather and other factors that affect road conditions, the vehicle is prone to slipperiness and vibration, which will cause the driving trajectory to change, affecting the actual running speed of the vehicle, causing the actual speed to deviate from the planned speed. Therefore, the change in the actual speed is set as follows:

Where, v _ref is the reference speed; v _act is the actual speed; α is the influencing factor;

When the error between the actual speed and the reference speed is within 2 km/h, it can be assumed that the actual speed is equal to the reference speed; when the error between the actual speed and the reference speed is in the range of [2 5] km/h, the actual speed is equal to the reference speed and the difference between the two speeds; where α is the speed change factor; when the error between the actual speed and the reference speed is greater than 5 km/h, the vehicle needs to brake instantly to ensure the driving safety of the vehicle;

At time t, the set of vehicle actions and states is expressed as follows: {v ₁ ,…,v _i …v _n }, i = 1,…,n. Bezier curves are used to implement path planning to generate collision-free predicted trajectories. To determine the accuracy of vehicle speed planning, a small sample of vehicle states and actions is taken from the experience buffer of the DDPG algorithm as reference values: {v _r1 ,…,v _ri …v _rn }, i = 1,…,n, and the error between the two is calculated as:

Where, _vi is the vehicle speed; L is the vehicle speed error rate; _vri is the reference vehicle speed obtained from the experience buffer;

Under extreme driving conditions, given the desired state reference trajectory, S _r , the output error is e _k (t) = S _r (t) - S (t), the learning law: uk ₊₁ (t) = L ( _uk (t), e _k (t)), and the compensation control action a _k is obtained; under normal driving, the DDPG algorithm is used to realize vehicle control, and its output action is a _π = μ ( _hi / β, γ ^π ) + η, then the total action control a = a _k + a _π , the reference formula is as follows:

In order to determine the accuracy of path planning, the actual running trajectory of the vehicle is calculated. The trajectory error when the roll angle changes between the actual running trajectory of the vehicle and the reference trajectory is as follows:

是实际的轨迹与参考轨迹最大的角度差，μ是路面附着系数；σ是车辆产生横向侧滑时的偏离角度，Φ是车辆侧倾角度，d垂向振动距离，χ是桥梁垂向振动时的偏离角度，即，桥面产生的垂向倾角；Where, r _act is the actual vehicle trajectory after vibration; r _ref is the reference trajectory;

is the maximum angle difference between the actual trajectory and the reference trajectory, μ is the road adhesion coefficient; σ is the deviation angle when the vehicle produces lateral sideslip, Φ is the vehicle roll angle, d is the vertical vibration distance, and χ is the deviation angle when the bridge vibrates vertically, that is, the vertical inclination angle of the bridge deck;

The vertical inclination angle χ generated by bridge vibration can be maximized as:

According to the high-precision map designed by the visual sensor, the optimal road driving area is found, and in the optimal road driving area, the reference vehicle state, control input and parameter output values are designed. At the same time, the constraints of multiple states are designed, and the iterative learning control algorithm is used to realize the vehicle's roll control;

The high-precision map designed by the visual sensor and the predicted state of LSTM find the extreme road driving area and design the range of layered uncertainty state parameters and action parameters. When the vehicle is driving in the extreme road driving area, DDPG and iterative control algorithm are used to realize the roll control of the vehicle. The iterative control algorithm plays a compensation role and constructs the reward function according to the constraint range of the vehicle state:

R＝v·(R ₁ +R ₂ +…R ₆ )

Among them, _R1 and _R2 represent the lateral distance error and its rate of change respectively; _R3 and _R4 represent the lateral angular velocity and its rate of change; _R5 and _R6 represent the roll angle and its rate of change; x is the angle value; v is the vehicle speed; _ey is the lateral distance, and k _i and k _j are reward factors respectively.

Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention designs a comprehensive control method for the roll of an autonomous driving vehicle based on a reinforcement learning algorithm (DDPG), and controls the roll of the vehicle in a complex road environment through reinforcement learning, so that the autonomous driving vehicle can achieve exploratory driving of the vehicle in complex road conditions and extreme weather through exploration and utilization methods; 2. Iterate learning control is performed for extreme driving conditions to compensate for the DDPG algorithm, thereby achieving a comprehensive control effect of the vehicle and ensuring the ultimate safe driving of the vehicle.

BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a schematic diagram of the DDPG network architecture;

Fig. 2 is a schematic diagram of the integrated control of vehicle roll;

FIG3 is a flow chart of comprehensive control based on the roll of an autonomous driving vehicle.

DETAILED DESCRIPTION

The present invention will be further described in detail below in conjunction with the accompanying drawings.

The present invention proposes a roll control method for an autonomous driving vehicle based on DDPG and iterative control, which specifically includes the following steps:

Step 1: Install lidar, vision, millimeter-wave radar, ultrasonic radar sensors, positioning system and inertial navigation system on the autonomous vehicle.

The present invention is aimed at the road conditions of cross-sea bridges, and its goal is to control the vehicle to travel safely at medium and low speeds (5-80KM/H), and to achieve a high level of intelligence of the vehicle through instantaneous tilt vehicle control behavior. To achieve the above goals, the present invention installs multiple laser radars, machine vision, millimeter wave radars and ultrasonic radar sensors on the autonomous driving vehicle, and installs positioning systems and inertial navigation systems (IMU) and other equipment. Among them, the laser radar sensor is used to detect dynamic and static obstacles on the road, including pedestrians, motorcycles and various vehicles, as well as drivable road areas; the machine vision sensor is used to sense lane lines, pedestrian and vehicle detection, as well as positioning and synchronous map creation; the millimeter wave radar sensor is used to detect the distance between the vehicle and pedestrians and moving vehicles; the ultrasonic radar is used to detect the distance between close-range vehicles; the positioning system and the inertial navigation system (IMU) are used to implement vehicle positioning technology.

Step 2: Use visual sensors, positioning systems, and inertial navigation systems to realize the vehicle’s position and map in different scenarios, so as to generate driving maps for autonomous vehicles in different scenarios and realize the environment required for the vehicle’s driving trajectory.

Use visual sensors, positioning systems and inertial navigation systems (IMU) to realize the position and map of the vehicle in severe weather conditions such as sunny days, rain, snow, fog and strong winds, so as to generate driving maps for autonomous driving vehicles in five scenarios: road conditions of cross-sea bridges in rainy and snowy weather, road conditions of cross-sea bridges in severe weather with strong winds, road conditions when the bridge vibrates on sunny days, driving conditions of a single vehicle in frequent and changeable weather, and driving conditions of multiple vehicles in frequent and changeable weather, to realize the environment required for the vehicle's driving trajectory.

Step 3: Control the steering wheel, accelerator and pedals respectively, drive on the cross-sea bridge, obtain the corresponding driving trajectories in rainy and snowy days, strong wind and bad weather, and sunny days, and build a data set.

A number of experienced drivers drove on the cross-sea bridge in five scenarios by controlling the steering wheel, accelerator and pedals, and recorded the corresponding driving trajectories to build the corresponding data set. The data set includes: vehicle speed, driving trajectory, vehicle position, heading angle, slip angle, yaw rate, and roll angle, providing necessary reference data for training data sets and evaluating vehicle controllability.

Step 4: The DDPG algorithm is trained on the driving map of the autonomous vehicle in different scenarios, which is used for the driving status of the cross-sea bridge under different levels of complex road conditions in bad weather; the autonomous vehicle generates real-time vehicle status and determines the vehicle's action behavior by interacting with the map environment in different scenarios; when performing action training, the action space is initialized, the online policy network in the actor network generates state space information, performs action output, and adds an action noise to obtain an exploratory action space.

As shown in Figure 1, the actor network is constructed, with the vehicle state and environment state as input, and the output is a vector composed of steering angle, throttle and brake, which correspond to the three neurons of the output layer of the actor strategy network respectively. The activation function of the throttle and brake is set to Sigmoid, and the activation function of the steering action value is Tanh. The structure of the hidden layer is: the first layer is a convolution size of 7*7, a filter size of 48, a step size of 4, and a total of 200 neurons; the second layer is a convolution size of 5*5, a filter size of 16, a step size of 2, and an activation function of ReLu function, a total of 400 neurons; the third layer adds 100 neurons to the LSTM layer; the fourth layer is a fully connected layer of 128 units; the fifth layer is a fully connected layer, a total of 128 units; the critic network input is the state and action space, after two hidden layers, 200 neurons in the first layer and 400 neurons in the second layer, spliced with the activation function ReLu, and finally the Q value is obtained; define h _i ∈(S _tT ,S _t-T+1 ,…,S _t ), where S _tT and S _t represent the state information at the current moment and the current moment respectively, then the encoded state is: s = f( _hi ; β), then the strategy of the actor network after the change is defined as: a = μ( _hi /β, ^γπ ) + η.

和油门信号

考虑到车辆行驶环境较为复杂，当道路环境较为复杂时，车辆是变速行驶，制动信号是当车辆在极端行驶条件下产生的动作，以防止车辆由于制动和路面湿滑，引起车辆的侧倾和侧翻运动，此时的动作空间应该设置为方向盘转角、制动信号和油门信号三类；当车辆在正常行驶工况下，假定车辆为匀速行驶，为防止车辆由于路面湿滑而产生侧倾现象，此时设置动作空间为方向盘转角和油门信号，并且根据这两种不同的行驶工况设置三种动作的约束范围，以确保车辆在可行驶的道路区域内可控行驶。Design the vehicle's action space, which includes the vehicle's steering wheel angle δ, brake signal

and throttle signal

Taking into account the complex driving environment of the vehicle, when the road environment is complex, the vehicle is driving at variable speeds, and the brake signal is the action generated when the vehicle is under extreme driving conditions to prevent the vehicle from rolling and tipping over due to braking and slippery road conditions. At this time, the action space should be set to three categories: steering wheel angle, brake signal and throttle signal; when the vehicle is under normal driving conditions, it is assumed that the vehicle is driving at a constant speed. In order to prevent the vehicle from tipping over due to slippery road conditions, the action space is set to steering wheel angle and throttle signal, and the constraint ranges of the three actions are set according to these two different driving conditions to ensure that the vehicle can be controlled within the drivable road area.

The vehicle acquires state data by exploring and utilizing the environment. The data states generated usually include the lateral distance and its rate of change and the vehicle roll angle and its rate of change. These state data are usually included in the experience buffer. When iterative learning is used to achieve vehicle control, it is necessary to design reference vehicle states, vehicle trajectories and other parameters. These reference states and trajectories can be obtained from the experience buffer, and the reference trajectory will be adjusted and changed with the complexity of different road environments.

In five different scenarios, namely, the road conditions of the cross-sea bridge in rainy and snowy weather, the road conditions of the cross-sea bridge in strong wind and bad weather, the road conditions when the bridge vibrates on sunny days, the road conditions of a single vehicle in frequent and changeable weather, and the road conditions of multiple vehicles in frequent and changeable weather, different trajectories are generated as reference paths, and the error between the actual planned path of the vehicle and the reference trajectory is compared. In addition, the reference trajectory also needs to add various constraints that meet the vehicle dynamics characteristics, and be modified and adjusted as the actual path set, which can be expressed as:

In the formula, σ is the path influence factor, p _ref is the reference trajectory, and p _{act is} the actual trajectory. That is, when the vehicle is driving in different road environments, the obtained driving trajectory needs to be appropriately modified and adjusted to conform to the vehicle dynamics characteristics of the vehicle during automatic driving. Only then can it be used as the driving trajectory of the autonomous driving vehicle.

Step 5: As shown in Figure 2, based on the LSTM historical memory and road planning attributes, a path for predicting the state of the autonomous driving vehicle is generated, and the DDPG algorithm is used to achieve tracking control of the path trajectory of the autonomous driving vehicle under normal driving conditions and extreme driving conditions, and an iterative control method is used to achieve compensation control of the autonomous driving vehicle.

Under normal driving conditions, that is, the road conditions when the bridge vibrates on a sunny day in the five scenarios, in order to ensure the safety and stability of the vehicle, it is necessary to consider the vehicle's roll, sideslip and yaw dynamic characteristics when modeling vehicle dynamics, and set vehicle state constraints to determine the lateral stability range, maximum steering angle range and allowable vehicle control range to prevent roll, so as to reduce the vehicle's lateral offset error:

ω _z-min ≤ω _z ≤ω _z-max ,ω _x-min ≤ω _x ≤ω _x-max ,u _x-min ≤u _x ≤u _x-max ,e _rx-min ≤e _r ≤e _{rx- max}

Where Ψ＝ _{[vyωzωxφ ]} ^T is the state vector, _u1 is the control input, _u2 is the _auxiliary control input; _ωz is the yaw rate; _ωx is the vehicle roll angle; _ux is the steering angle; and _er is the lateral tracking offset error.

According to the road state information predicted by LSTM, an objective function considering steering angle, tire adhesion coefficient, roll angle error and path tracking error is constructed. Taking into full account the dynamic constraint of the maximum allowable error of the vehicle, the physical constraint of the vehicle determined by the lateral tracking offset error is determined to reduce the error of tracking and controlling the vehicle:

Where w ₁ ,w ₂ ,w ₃ ,w ₄ are parameter variables; μ _r is the road adhesion coefficient

As shown in Figure 3, an iterative control algorithm is used to achieve compensation control of vehicle roll, and reference vehicle state, reference control input and reference output values are set to ensure the tracking function under vehicle physical constraints and road constraints under multiple constraints, increase anti-interference when the vehicle is driving and reduce the model error rate. First, the DDPG algorithm is used to train the network model, and interactive training is performed between the vehicle and the dynamic cross-sea bridge road conditions to ensure that the training completes the task. If the task is completed, the trained action is saved. If the training task is not completed well, the iterative control algorithm is used to compensate the parameters of the output action space, and finally the training task is completed, and better actions are achieved, and finally better roll control of the autonomous driving vehicle is achieved.

Under extreme driving conditions, namely, the road conditions of the cross-sea bridge in rainy and snowy weather in five scenarios, the road conditions of the cross-sea bridge in strong wind and severe weather, the driving conditions of a single vehicle in frequent and changeable weather, and the driving conditions of multiple vehicles in frequent and changeable weather; due to the influence of severe weather, the road environment is uncertain, which interferes with the normal driving of vehicles.

Under extreme driving conditions, the vehicle is prone to slippery and vibrating, which will cause the driving trajectory to change, affecting the actual running speed of the vehicle and causing the actual speed to deviate from the planned speed. Therefore, the change in the actual speed can be set as follows:

Where, v _ref is the reference vehicle speed; v _act is the actual vehicle speed; α is the influencing factor.

When the error between the actual vehicle speed and the reference speed is within 2 km/h, it can be assumed that the actual vehicle speed is equal to the reference speed; when the error between the actual vehicle speed and the reference speed is in the range of [2 5] km/h, the actual vehicle speed is equal to the reference speed and the difference between the two speeds, where α is the speed change factor; when the error between the actual vehicle speed and the reference speed is greater than 5 km/h, the vehicle needs to brake instantly to ensure the vehicle's driving safety.

After the path prediction of the autonomous driving vehicle passes through the LSTM historical memory state, the predicted speed path is generated using the road planning attributes. At time t, the vehicle's action and state set are expressed as follows: {v ₁ ,…,v _i …v _n }, i = 1,…, n. Bezier curves are used to implement path planning to generate collision-free predicted trajectories. To judge the accuracy of the speed planning, a small sample of vehicle states and actions is taken from the experience buffer of the DDPG algorithm as a reference value: {v _r1 ,…,v _ri …v _rn }, i = 1,…, n, and the error between the two is calculated as:

Where, _vi is the vehicle speed; L is the vehicle speed error rate; _vri is the reference vehicle speed obtained from the experience buffer.

Under extreme conditions, given the desired state reference trajectory, S _r , the output error is e _k (t) = S _r (t) - S (t), the learning law: _{uk + 1} (t) = L ( _uk (t), e _k (t)), and the compensation control action a _k is obtained; under normal driving, the DDPG algorithm is used to realize vehicle control, and its output action is a _π = μ ( _hi / β, γ ^π ) + η, then the total action control a = a _k + a _π , the reference formula is as follows:

In order to judge the accuracy of path planning, the actual operation of the vehicle is calculated. The trajectory error when the roll angle changes between the actual running trajectory of the vehicle and the reference trajectory is as follows: The vehicle rolls, which is mainly reflected in two situations. The first movement is when the bridge vibrates, causing the autonomous driving vehicle to deviate from the planned path. At this time, the vehicle is prone to roll motion and generates a roll angle; therefore, the trajectory error when the roll angle changes between the actual running trajectory of the vehicle and the reference trajectory can be expressed as:

是实际的轨迹与参考轨迹最大的角度差，σ是车辆产生横向侧滑时的偏离角度，Φ是车辆侧倾角度，d垂向振动距离，χ是桥梁垂向振动时的偏离角度，即，桥面产生的垂向倾角。Where, r _act is the actual vehicle trajectory after vibration; r _ref is the reference trajectory;

is the maximum angle difference between the actual trajectory and the reference trajectory, σ is the deviation angle when the vehicle produces lateral sideslip, Φ is the vehicle roll angle, d is the vertical vibration distance, and χ is the deviation angle when the bridge vibrates vertically, that is, the vertical inclination angle of the bridge deck.

The vertical inclination angle χ generated by bridge vibration can be maximized as:

The vehicle rolls. The second type of movement is caused by the slippery road surface due to bad weather, which causes the road adhesion coefficient to change, causing the vehicle to roll, slide and roll over. Therefore, the trajectory error of the vehicle's actual running trajectory and the reference trajectory that produces a roll angle change can be expressed as:

是实际的轨迹与参考轨迹最大的角度差，μ是路面附着系数；σ是车辆产生横向侧滑时的偏离角度，Φ是车辆侧倾角度，d垂向振动距离，χ是桥梁垂向振动时的偏离角度，即，桥面产生的垂向倾角，μ是道路附着系数，其范围是[01]。Where, r _act is the actual vehicle trajectory after vibration; r _ref is the reference trajectory;

is the maximum angle difference between the actual trajectory and the reference trajectory, μ is the road adhesion coefficient; σ is the deviation angle when the vehicle produces lateral sideslip, Φ is the vehicle roll angle, d is the vertical vibration distance, χ is the deviation angle when the bridge vibrates vertically, that is, the vertical inclination angle generated by the bridge deck, μ is the road adhesion coefficient, and its range is [01].

According to the high-precision map designed by the visual sensor, the optimal road driving area is found, and in the optimal road driving area, the reference vehicle state, control input and parameter output values are designed. At the same time, the constraints of multiple types of states are designed, and the iterative learning control algorithm is used to realize the roll control of the vehicle. At this time, the DDPG algorithm plays a compensatory role to realize the safe driving of the vehicle in the controllable road area. According to the high-precision map designed by the visual sensor and the predicted state of LSTM, the extreme road driving area is found, and the range of layered uncertainty state parameters and action parameters is designed. When the vehicle is driving in the extreme road driving area, DDPG and iterative control algorithm are used to realize the roll control of the vehicle. The iterative control algorithm plays a compensatory role; when the vehicle is under extreme working conditions, it is necessary to increase the constraints of the vehicle state, and construct the reward function as follows:

R＝v·(R ₁ +R ₂ +…R ₆ )

Among them, _R1 and _R2 represent the lateral distance error and its rate of change respectively; _R3 and _R4 represent the lateral angular velocity and its rate of change; x is the angle value; v is the vehicle speed; _R5 and _R6 represent the roll angle and its rate of change; _ey is the lateral distance, and k _i and k _j are reward factors respectively.

The above descriptions are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the protection scope of the present invention. All equivalent methods or changes that do not deviate from the technical creation of the present invention should be included in the protection scope of the present invention.

Claims (5)

1. An automatic driving vehicle roll control method based on DDPG and iterative control, which is characterized by comprising the following steps:

(1) Installing a laser radar, a vision, a millimeter wave radar, an ultrasonic radar sensor, a positioning system and an inertial navigation system on an autopilot vehicle;

(2) The method comprises the steps of using a visual sensor, a positioning system and an inertial navigation system to realize positions and maps of vehicles in different scenes respectively so as to generate automatic driving vehicle running maps in different scenes and realize the environment required by vehicle running tracks;

(3) Respectively controlling a steering wheel, an accelerator and a pedal, driving on a cross-sea bridge, acquiring corresponding driving tracks in rainy and snowy days, strong wind and bad weather and sunny days, and constructing a data set;

(4) The DDPG algorithm is trained on the running map of the automatic driving vehicle under different scenes and is used for running states of the cross-sea bridge under different complex road condition grades in severe weather; the automatic driving vehicle generates real-time vehicle states through interaction with map environments in different scenes, and determines action behaviors of the vehicle; initializing an action space when action training is carried out, generating state space information by an online strategy network in an actor network, carrying out action output, and adding an action noise to obtain the action space with exploratory property;

(5) Based on LSTM historic memory and road planning attributes, generating a predicted path of the state of the automatic driving vehicle, adopting a DDPG algorithm to realize tracking control of the path track of the automatic driving vehicle under normal driving road conditions and extreme driving road conditions, and adopting an iterative control method to realize compensation control of the automatic driving vehicle;

the network design of the DDPG algorithm in the step (4) is as follows:

an actor network is constructed, the vehicle state and the environment state are taken as input, the output is a vector formed by steering angle, accelerator and brake signals, the vector corresponds to 3 neurons of an actor strategy network output layer respectively, the activation function of the accelerator and the brake is set to be Sigmoid, the activation function of the steering action value is Tanh, and the hidden layer has the structure that: the first layer is a convolution size 7*7, a filter size 48, a step size 4, and a total of 200 neurons; the second layer is a convolution size 5*5, a filter size 16, a step size 2, an activation function ReLu function, 400 neurons total; the third layer adds 100 neurons to the LSTM layer; the fourth layer is a 128-unit fully connected layer; the fifth layer is a fully connected layer, totaling 128 units; the input of the critic network is a state and action space, and the state and action space are spliced with an activation function ReLu through two hidden layers, namely 200 neurons in the first layer and 400 neurons in the second layer, so that a Q value is finally obtained; definition of h _i ∈(S _t-T ,S _t-T+1 ,…,S _t ) Wherein S is _t-T And S is _t State information respectively representing the current time and the current time, the encoded state is: s=f (h _i The method comprises the steps of carrying out a first treatment on the surface of the β), then the policy of the changed actor network is defined as action a=μ (h _i /β,γ ^π )+η；

The implementation process of the tracking control of the path track of the automatic driving vehicle under the normal driving road condition in the step (5) is as follows:

the method comprises the steps of establishing a vehicle dynamics model by considering roll, sideslip and yaw dynamics characteristics of a vehicle under normal running road conditions, namely road conditions when a bridge vibrates on a sunny day, setting vehicle state constraint conditions, and determining a lateral stability range, a maximum steering angle range and a range of allowable vehicle control for preventing roll so as to reduce a lateral deviation error of the vehicle:

ω _z-min ≤ω _z ≤ω _z-max ,ω _x-min ≤ω _x ≤ω _x-max ，u _x-min ≤u _x ≤u _x-max ，e _r-x-min ≤e _r ≤e _r-x-max

wherein omega is _z Is yaw rate; omega _x Is the roll angle of the vehicle; u (u) _x Is the steering angle; e, e _r Is a lateral tracking offset error;

according to LSTM predicted road state information, constructing an objective function considering steering angle, tire attachment coefficient, roll angle error and path tracking error, and determining physical constraint of the vehicle determined by transverse tracking offset error under the condition of full consideration of the dynamics constraint condition of maximum allowable error of the vehicle so as to reduce the error of tracking control of the vehicle:

wherein w is ₁ ,w ₂ ,w ₃ ,w ₄ Parameter variables respectively; mu (mu) _r Is the road adhesion coefficient;

adopting an iterative control algorithm to realize compensation control of vehicle roll, and setting a reference vehicle state, a reference control input and a reference output value to ensure a tracking function under the conditions of vehicle physical constraint and road constraint under the condition of multiple constraints, thereby improving the anti-interference performance of the vehicle during running and reducing a model error rate;

according to the running condition of the vehicle, constructing a state space, an action space and a reward function required by a DDPG algorithm; the motion space mainly comprises steering wheel rotation angle, throttle and braking signals, and the state space comprises vehicle transverse tracking error and change rate thereof, vehicle side-tipping angle error and change rate thereof, and yaw rate error and change rate thereof; construction of the bonus function in case of vibration of the cross-sea bridge, the actual track of the vehicle is changed, the road is inclined at a certain angle, and thus the bonus function is equal to the cumulative multiplication of the discount factor and the change of speed.

2. The DDPG and iterative control-based roll control method for an autonomous vehicle of claim 1, wherein the lidar sensor of step (1) is used to detect dynamic and static obstacles on roads, including pedestrians, motorcycles, and various vehicles, and movable road areas; the visual sensor is used for sensing lane lines, pedestrians and vehicles and performing positioning and synchronous map creation; the millimeter wave radar sensor is used for detecting the distance between the vehicle and the pedestrian and the distance between the vehicle and the traveling vehicle; the ultrasonic radar is used for detecting the distance between the short-distance vehicles; the vision sensor, the positioning system and the inertial navigation system are used for realizing the vehicle positioning technology.

3. The DDPG and iterative control-based automatic driving vehicle roll control method of claim 1, wherein the different scenes in the step (2) comprise five scenes including a cross-sea bridge road condition in rainy and snowy weather, a cross-sea bridge road condition in strong wind and severe weather, a road condition when a bridge vibrates on a sunny day, a driving road condition of a single vehicle in frequent variable weather, and a driving road condition of multiple vehicles in frequent variable weather.

4. The DDPG and iterative control-based automatic driving vehicle roll control method of claim 1, wherein the dataset of step (3) comprises vehicle speed, travel trajectory, vehicle position, heading angle, slip angle, yaw rate, roll angle.

5. The DDPG and iterative control-based roll control method for an autonomous vehicle according to claim 1, wherein the implementation of the step (5) for implementing the tracking control of the path trajectory of the autonomous vehicle under the extreme driving road condition is as follows:

under extreme driving road conditions, namely under severe weather and other factors affecting road conditions, the phenomenon that the vehicle easily generates wet skid and vibration is considered, the driving track is changed, the actual running speed of the vehicle is affected, and the actual vehicle speed and the planned vehicle speed are deviated, so that the change of the actual vehicle speed is set as follows:

in the formula, v _ref Is the reference vehicle speed; v _act Is the actual vehicle speed; alpha is an influencing factor;

when the error between the actual vehicle speed and the reference vehicle speed is within 2KM/H, the actual vehicle speed is equal to the reference vehicle speed; when the error between the actual vehicle speed and the reference vehicle speed is in the [2 ] KM/H interval, the vehicle speed of the actual vehicle speed is equal to the reference vehicle speed and the difference between the two vehicle speeds; wherein α is a speed change factor; when the error between the actual vehicle speed and the reference vehicle speed is greater than 5KM/H, the vehicle needs to be braked instantaneously at the moment, so that the running safety of the vehicle is ensured;

at time t, the vehicle's set of actions and states are expressed as follows: { v ₁ ,…,v _i …v _n I=1, …, n, path planning is implemented using bezier curves to produce collision-free predicted trajectories; in order to judge the accuracy of vehicle speed planning, the vehicle state and action of a small sample are taken out from an experience buffer of a DDPG algorithm as reference values: { v _r1 ,…,v _ri …v _rn I=1, …, n, and the error of both is calculated as:

in the formula, v _i Is the vehicle speed; l is the vehicle speed error rate; v _ri The reference vehicle speed is obtained from the experience buffer;

under the extreme driving condition, the expected state reference track is given as S _r The output error is e _k (t)＝S _r (t) -S (t), learning law:u _k+1 (t)＝L(u _k (t),e _k (t)) to obtain the compensation control action a _k The method comprises the steps of carrying out a first treatment on the surface of the Under normal running, the DDPG algorithm is adopted to realize vehicle control, and the output action is a _π ＝μ(h _i /β,γ ^π ) +η, total action control a=a _k +a _π The reference formula is as follows:

in order to judge the accuracy of path planning, the track error when the actual running track and the reference track of the vehicle generate the change of the roll angle is calculated as follows:

wherein r is _act Is the actual vibrating vehicle track; r is (r) _ref Is a reference track;

the angle difference between the actual track and the reference track is the largest, and mu is the road adhesion coefficient; sigma is the deviation angle when the vehicle generates lateral sideslip, phi is the vehicle side-tipping angle, d is the vertical vibration distance, χ is the deviation angle when the bridge vibrates vertically, namely, the vertical dip angle generated by the bridge deck;

vertical inclination χ maximize that bridge vibration produced sets up to:

searching an optimal road running area according to a high-precision map designed by the visual sensor, designing a reference vehicle state, a control input value and a parameter output value in the optimal road running area, designing constraint conditions of multiple types of states, and realizing roll control of the vehicle by adopting an iterative learning control algorithm;

the prediction states of the high-precision map and the LSTM designed by the visual sensor find a limited road driving area, and design a range of layered uncertainty state parameters and action parameters, when a vehicle is driven in the limited road driving area, the DDPG and an iterative control algorithm are adopted to realize the roll control of the vehicle, the iterative control algorithm plays a role in compensation, and a reward function is constructed according to the constraint range of the vehicle state, wherein the reward function is as follows:

R＝v·(R ₁ +R ₂ +…R ₆ )

wherein R is ₁ And R is ₂ Each representing a lateral distance error and a rate of change thereof; r is R ₃ And R is ₄ Representing the transverse angular velocity and its rate of change; r is R ₅ And R is ₆ Representing the roll angle and its rate of change; x is an angle value; v is the vehicle speed; e, e _y Is the lateral distance, k _i ，k _j The bonus factors, respectively.

Images