KR102479484B1 - System and Method for Improving Traffic for Autonomous Vehicles at Non Signalized Intersections - Google Patents

Abstract

The present invention relates to an apparatus and method for improved passage of autonomous vehicles at unsignalized intersections, which enables efficient passage by utilizing the safety theory of responsibility sensitivity and a partial observation Markov decision procedure at unsignalized intersections. State initialization unit that initializes the learning state of the decision (Partial Observability Markov decision process; POMDP) algorithm; derives optimal behavior by applying Responsibility-Sensitive Safety (RSS) to the POMDP model for optimizing autonomous vehicle operation An optimal action derivation unit that performs; An action execution unit that executes the optimal behavior derived from the optimal behavior derivation unit; A state observation unit that receives data observed from the vision sensor and radar sensor of the autonomous vehicle and observes the driving state; Safety, non-safety , failure, compensation determination unit that corrects the autonomous vehicle behavior based on the RSS algorithm for autonomous vehicles and the adaptive model predictive control system for human-driven vehicles (Adaptive Model Predictive Control System) according to recognition of target compensation;

Description

System and Method for Improving Traffic for Autonomous Vehicles at Non Signalized Intersections

The present invention relates to autonomous vehicle traffic control, and specifically, for improved passage of autonomous vehicles at unsignalized intersections that enables efficient passage by utilizing the safety theory of responsibility sensitivity and a partially observed Markov decision procedure at unsignalized intersections. It relates to an apparatus and method.

An autonomous vehicle is a vehicle equipped with a technology for recognizing a lane and automatically steering using a camera or a front object detection sensor. Autonomous vehicles measure the lane width, the lateral position of the vehicle on the lane, the distance to both lanes, the shape of the lane, and the radius of curvature of the road based on image processing of the camera or front object detection. The driving trajectory of the vehicle is estimated using location and road information, and the lane is changed according to the estimated driving trajectory.

An autonomous vehicle automatically controls the throttle valve, brake, and transmission of the vehicle through the location and distance of the preceding vehicle detected by a camera mounted on the front of the vehicle or a front object detection sensor to perform appropriate acceleration and deceleration, You can also ask them to keep an appropriate distance from the vehicle.

However, when such an autonomous vehicle passes through an intersection, it starts after stopping according to the traffic signal of the traffic light and detects the movement of the preceding vehicle before starting. can

In particular, when the driving environment is grasped using information input from a sensor, such as an autonomous vehicle, driving at an unsignalized intersection becomes a much more difficult task than driving on a general road.

Researches are being conducted for efficient driving at unsignalized intersections by identifying the driving environment of autonomous vehicles, but in non-signalized intersection traffic due to platooning of autonomous vehicles in mixed traffic conditions (mixed autonomous vehicles and human drivers) There are still many challenges to be addressed.

Therefore, there is a demand for the development of new technologies to improve traffic at non-signalized intersections and secure safety in accordance with platooning of autonomous vehicles.

Republic of Korea Patent Publication No. 10-2020-0071406 Republic of Korea Patent Publication No. 10-2020-0058613 Republic of Korea Patent Publication No. 10-2018-0065196

The present invention is to solve the problems of the prior art autonomous vehicle traffic control technology, and to enable efficient passage of autonomous vehicles at unsignalized intersections by utilizing the responsibility-sensitivity safety theory and partially observed Markov decision procedure at unsignalized intersections. Its purpose is to provide a device and method for improved traffic.

The present invention establishes a model that learns autonomous driving behavior in consideration of traffic safety assurance and delay time of autonomous vehicles between multiple human driver vehicles at unsignalized intersections, thereby enabling efficient autonomous vehicle accident prevention. Its purpose is to provide a device and method for improved passage of autonomous vehicles at signalized intersections.

The present invention is a method of learning through information within the range that an autonomous vehicle can observe, such as in a real situation, and maximizes the reward of reinforcement learning for actions by using a Markov decision-making model (Partial Observability MDP, POMDP), which is reinforcement learning. Its purpose is to provide a device and method for improved passage of autonomous vehicles at unsignalized intersections.

The present invention utilizes Matlab's Automated Driving Toolbox to utilize radar and vision sensor data, and optimizes it with a Responsibility-Sensitive Safety (RSS)-based reinforcement learning-autonomous driving system framework to improve the performance of autonomous vehicles. Its purpose is to provide a device and method for improved passage of autonomous vehicles at unsignalized intersections where the system predicts the behavior of other vehicles and allows them to operate.

The present invention includes a Partial Observability Markov decision process (POMDP) process that enables the learning subject to make a decision based on partial environment observation, which determines behavior and reinforces learning about behavior. Its purpose is to provide a device and method for improved passage of autonomous vehicles at unsignalized intersections that can maximize the compensation of

In the present invention, in order to simulate the actual autonomous driving environment in a simulation environment, the basis for learning and behavioral decisions is based on data obtained through autonomous vehicle sensors (partial observation) rather than all environments (entire observation) of the simulation. The purpose is to provide a device and method for improved passage of autonomous vehicles at non-signalized intersections that determine and maximize the reward of reinforcement learning for actions.

The present invention uses a Responsibility-Sensitive Safety (RSS) to maintain a safe distance between an autonomous vehicle and a human driver at an unsignalized intersection, so that the autonomous vehicle model can generate dangerous situations depending on the distance. Its purpose is to provide a device and method for improved passage of autonomous vehicles at non-signalized intersections that can respond appropriately when needed.

The present invention applies an Adaptive Model Predictive Control System for a human driver vehicle, and grasps the relative distance and relative speed with the nearest human driver in front obtained through a sensor of an autonomous vehicle in simulation, In the control variable, the self-driving vehicle autonomously maintains a certain distance from the vehicle in front and provides a device and method for improved passage of autonomous vehicles at non-signalized intersections so that they can operate in response to the behavior of human drivers. It has a purpose.

Other objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned above will be clearly understood by those skilled in the art from the description below.

In order to achieve the above object, the device for improved passage of autonomous vehicles at non-signalized intersections according to the present invention initializes the learning state of the Partial Observability Markov decision process (POMDP) algorithm. Unit; Optimal action derivation unit that derives optimal behavior by applying Responsibility-Sensitive Safety (RSS) to the POMDP model for autonomous vehicle (AV) operation optimization; Optimal behavior derived from the optimal behavior derivation unit Action execution unit to execute; State observation unit to receive data observed from the vision sensor and radar sensor of the autonomous vehicle (AV) and observe the driving state; RSS algorithm for autonomous vehicles according to safety, non-safety, failure, and target compensation recognition and a compensation determination unit for modifying behavior of the autonomous vehicle in response to a human driver vehicle based on an adaptive model predictive control system.

A method for improved passage of an autonomous vehicle at an unsignalized intersection according to the present invention for achieving another object includes a state initialization step of initializing a learning state of a Partial Observability Markov decision process (POMDP) algorithm; The optimal action derivation step of deriving the optimal action by applying the Responsibility-Sensitive Safety (RSS) to the POMDP model for autonomous vehicle (AV) operation optimization; Action execution step; State observation step of receiving data observed from the vision sensor and radar sensor of the autonomous vehicle (AV) and observing the driving state; and a reward determination step of correcting behavior of an autonomous vehicle in response to a human driver vehicle based on an adaptive model predictive control system.

As described above, the device and method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention have the following effects.

First, it enables efficient traffic at unsignalized intersections by utilizing the safety theory of responsibility sensitivity and the partially observed Markov decision procedure.

Second, it is possible to efficiently prevent accidents of autonomous vehicles by establishing a model that learns autonomous driving behavior by considering traffic safety guarantee and delay time of autonomous vehicles between multiple human driver vehicles at unsignalized intersections.

Third, it is a method of learning through information within the range that autonomous vehicles can observe, such as in real situations, and the reward of reinforcement learning for actions can be maximized by using the reinforcement learning Markov decision-making model (Partial Observability MDP, POMDP). let it be

Fourth, by using Matlab's Automated Driving Toolbox, radar and vision sensor data are utilized, and the autonomous vehicle system is optimized with a Responsibility-Sensitive Safety (RSS)-based reinforcement learning-autonomous driving system framework. It predicts the behavior of other vehicles and enables them to operate.

Fifth, it is possible to determine behavior and use reinforcement learning for behavior, including the Partial Observability Markov decision process (POMDP) process that enables the learning subject to make decisions based on partial environment observations. to maximize the reward.

Sixth, Responsibility-Sensitive Safety (RSS) is used to maintain a safe distance between an autonomous vehicle and a human driver at unsignalized intersections, when the autonomous vehicle model can cause dangerous situations depending on the distance. enable you to respond appropriately.

Seventh, by applying the Adaptive Model Predictive Control System to the control of the human driver vehicle, the relative distance and relative speed with the nearest human driver vehicle in front obtained from the sensor of the autonomous vehicle in simulation and, in the control variables, the self-driving vehicle autonomously maintains a certain distance from the vehicle in front so that it can operate in response to the behavior of the human driver.

Figure 1a is a configuration diagram showing the rules of the ACC system
1B is a block diagram of a device for improved passage of autonomous vehicles at an unsignalized intersection according to the present invention.
2a and 2b are block diagrams for explaining a Partial Observability Markov decision process (POMDP) process.
3 is an operational flow chart showing a method for improved passage of an autonomous vehicle at an unsignalized intersection according to the present invention.
Figure 4 is a pseudo code format diagram showing RSS algorithm optimization through the POMDP framework
5a and 5b are configuration diagrams of simulation experiments for performance evaluation of the RSS-based POMDP model.
6a and 6b are graphs of simulation results using the output profile of the first experiment.
7a and 7b are graphs of simulation results using the output profile of the second experiment.
8 is a performance comparison graph between a model according to the present invention and a previous adaptive MPC model
9 is a graph of simulation results according to autonomous vehicle speed in the model according to the present invention.
10 is a graph of simulation results through acceleration of an autonomous vehicle in a model according to the present invention

Hereinafter, a preferred embodiment of a device and method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention will be described in detail.

Features and advantages of the apparatus and method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention will become clear through detailed descriptions of each embodiment below.

1A is a block diagram showing rules of an ACC system, and FIG. 1B is a block diagram of a device for improved passage of autonomous vehicles at an unsignalized intersection according to the present invention.

An apparatus and method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention enables efficient passage at unsignalized intersections by utilizing the safety theory of responsibility sensitivity and the partially observed Markov decision procedure.

To this end, the present invention may include a configuration for constructing a model for learning autonomous driving behavior in consideration of traffic safety guarantee, delay time, etc. of an autonomous vehicle between multiple human driver vehicles at an unsignalized intersection.

The present invention is a method of learning through information within a range that an autonomous vehicle can observe, such as in a real situation, and maximizes the reward of reinforcement learning for actions by using a Markov decision-making model (Partial Observability MDP, POMDP), which is reinforcement learning. configuration can be included.

The present invention utilizes Matlab's Automated Driving Toolbox to utilize radar and vision sensor data, and optimizes it with a Responsibility-Sensitive Safety (RSS)-based reinforcement learning-autonomous driving system framework to improve the performance of autonomous vehicles. It may include a configuration that allows the system to drive while predicting the behavior of other vehicles.

The present invention includes a Partial Observability Markov decision process (POMDP) process that enables the learning subject to make a decision based on partial environment observation, which determines behavior and reinforces learning about behavior. It may include a configuration that compensates for

The present invention uses a Responsibility-Sensitive Safety (RSS) to maintain a safe distance between an autonomous vehicle and a human driver at an unsignalized intersection, so that the autonomous vehicle model can generate dangerous situations depending on the distance. When appropriate, it may include a corresponding configuration.

In the present invention, the Adaptive Model Predictive Control System is applied to a human driver vehicle, and the relative distance and relative speed with the nearest human driver vehicle in front obtained from the sensor of the autonomous vehicle in simulation are grasped. In the control variable, the self-driving vehicle may include a configuration in which the autonomous vehicle operates in response to the human driver's behavior in a manner of autonomously maintaining a certain distance from the preceding vehicle.

An adaptive model predictive control system will be described as follows.

MPC (Model Predictive Control) is designed to estimate the future behavior of a human driver vehicle and use real-time optimization to control appropriate behavior based on the predicted content. do.

In the present invention, the detection object of the MPC grasps the relative distance and relative speed with the nearest human driver in front obtained from the sensor of the autonomous vehicle in simulation, and the autonomous vehicle autonomously maintains a certain distance from the vehicle in front in the control variable. It operates in response to the behavior of the human driver in a manner that maintains

And the ACC system consists of lower and upper level controllers, the upper level controller is the fusion of relative speed and relative distance, and the lower level controller adjusts the brake system to achieve the best acceleration.

In order to calculate the optimal acceleration, the relationship between the ego vehicle and the lead vehicle must be established. ACC controls the longitudinal acceleration of an autonomous vehicle by autonomously maintaining the relative position and relative speed of the vehicle in front. The controller estimates relative speed and distance through vehicle-to-vehicle (V2V) communication based on real-time measurements from on-board sensors (such as radar and vision sensors).

The safety distance is defined as

는 ACC 시스템의 안전 거리,

는 자율주행차량(ego vehicle)의 실제 속도,

는 원하는 정지 거리,

은 차량 사이의 이동 시간을 나타낸다.here,

is the safety distance of the ACC system,

is the actual speed of the ego vehicle,

is the desired stopping distance,

represents the travel time between vehicles.

The driving decision function of the ACC system is as follows.

는 인간 운전 차량이 너무 가깝고, 안전 거리가 복구될 때까지 자율주행차량이 감속함을 의미한다.(공간 제어)

means that the human-driven vehicle is too close, and the autonomous vehicle decelerates until the safe distance is restored (space control).

는 인간 운전 차량이 너무 멀다는 것을 의미하며, 자율주행 차량은 설정 속도에 도달할 때까지 평상시와 같이 움직인다.(속도 제어)

means that the human-driven vehicle is too far away, and the autonomous vehicle moves as usual until it reaches the set speed (speed control).

On longitudinal with sensor fusion, the track-leading vehicle detects objects in the same lane ahead of the autonomous vehicle and in other lanes within the sensor's detection range, between the autonomous vehicle and the guided vehicle (the closest human-driven vehicle in front of the autonomous vehicle). Find the relative distance and relative speed of

1A shows the rules of the ACC system and shows the relationship between the safety distance and the relative distance related to the driving decision function of the ACC system (eg, interval control and speed control).

In the ACC system, the acceleration of an ACC-equipped vehicle at discrete time is presented as follows.

는 ACC 장착 차량의 가속,

는 샘플링 기간이며,

는 하위 레벨 컨트롤러의 유한 대역폭에 해당하는 시간 지연이며,

는 가속에 관한 제어 변수 매트릭스를 나타낸다.here,

is the acceleration of the ACC-equipped vehicle,

is the sampling period,

is the time delay corresponding to the finite bandwidth of the lower-level controller,

represents a control variable matrix related to acceleration.

The MPC algorithm is designed to estimate future behavior and determine the appropriate control action in the predictive field using online optimization.

MPC considers the interaction between output and input, similar to how feedback control algorithms work. This model selects the most suitable acceleration for AV.

MPC has been introduced through some autonomous control applications such as traction control issues and lane keeping assist systems.

In the MPC algorithm, the predictive and current state parameters that can be measured at sampling time k are shown as follows.

는 예측 상태 행렬,

는 현재 상태 행렬이며, A와 B는 다음과 같이 상태 전환 행렬을 나타낸다.here,

is the predicted state matrix,

is the current state matrix, and A and B represent state transition matrices as follows.

, 상대 속도

및 자기 속도

를 포함한 입력 데이터는 다음과 같다Relative distance of longitudinal vehicle dynamics in continuous time separated by discrete time

, relative speed

and magnetic speed

The input data including

는 상대 가속도이고,

는 시간 t에서 자율 주행 차량의 가속도이다.here,

is the relative acceleration,

is the acceleration of the autonomous vehicle at time t.

The adaptive MPC system becomes more powerful with full throttle or braking capabilities, and the host vehicle can respond instantly when the host vehicle suddenly changes lanes or brakes. Accordingly, adaptive MPC systems focus on safety, control-following vehicles, smooth driving and fuel economy.

The constraints of velocity (v), acceleration (a), braking (u), and jerk (j) integrated into hard constraints similar to those of the ACC system can be expressed as follows.

According to the input parameters of sensor fusion, simulation time, longitudinal speed of the automated vehicle and road information, the tracked lead vehicle with sensor fusion first detects an object in front of the autonomous vehicle and then passes it to the multi-object tracker.

The state of the detection entity is estimated and fused by the Kalman filter algorithm.

And Responsibility-Sensitive Safety (RSS) to formulate rules for safe automated vehicles based on human concepts related to all driving scenarios defined by tuples (safe distance, hazardous situations and appropriate responses). has been introduced

Following a few simple rules, such as maintaining a safe following distance, the RSS algorithm proposes safety guarantees for the AV in response to its surroundings. For example, an AV evaluates and determines liability when a collision between an AV and a human-driven vehicle occurs.

It is used to evaluate liability in the event of a collision between an autonomous vehicle and a human driver, and in the present invention, the RSS algorithm is used to maintain a safe distance between an autonomous vehicle and a human driver at unsignalized intersections. Be able to respond appropriately when a situation may arise.

Automated vehicles (AVs) must apply RSS algorithms to mixed traffic and maintain safe distances (e.g. safe vertical distance and safe horizontal distance) to avoid dangerous situations.

Also, if an AV can avoid a traffic accident with another car, they must slow down or change lanes with minimal acceleration.

)는 다음과 같이 표시된 AV의 반응 시간(

) 및 제동 거리(

)를 포함한다.safety distance (

) is the response time of AV expressed as (

) and braking distance (

).

는 AV의 응답 시간을 나타내고,

은 AV의 실제 속도이고,

는 AV의 최소 가속도이다.here,

represents the response time of AV,

is the actual speed of AV,

is the minimum acceleration of AV.

In the present invention, the RSS algorithm is applied to non-signalized intersections in order to maintain a safe distance from a driverless vehicle to a manually driven vehicle.

The AV should maintain a path different from the outside path if possible based on a mathematical expression. In other words, the RSS algorithm is to ensure that the automated vehicle responds appropriately if a dangerous situation may be caused by another vehicle.

First, the safety distance is calculated using a mathematical definition (Equation 12).

Second, the forward collision warning is determined from the relative distance and relative speed between the self-driving vehicle and the MIO (Most Important Object) track through the identified track.

Finally, the AV takes appropriate action to restore a safe state (e.g. speed control or clearance control in the ACC system).

에서 최대 가속도에 도달할 때까지 가속되었고 수동 구동 차량으로부터 안전한 거리를 유지하기 위해 응답 시간 후 최소 가속도에 의해 감속된다.According to the RSS algorithm, AV is the response time

accelerated until reaching maximum acceleration at , and then decelerated by minimum acceleration after a response time to maintain a safe distance from manually driven vehicles.

Therefore, the autonomous driving decision function is:

인 경우, 두 차량 모두 정상 주행 및 설정 속도(ACC 시스템의 속도 제어)를 따를 수 있다.if,

If , both vehicles can follow the normal driving and set speed (speed control of the ACC system).

인 경우, 자율 주행 차량은 안전 거리가 복원될 때까지 최소 가속으로 감속한다(ACC 시스템의 공간 제어).if,

, the autonomous vehicle decelerates with minimum acceleration until the safe distance is restored (space control of the ACC system).

And the Markov Decision Process (MDP) is a powerful framework used to determine the appropriate action in a fully observable random environment. However, AVs maneuver into uncertain environments, taking into account imprecise intent and sensor noise.

To solve this problem, POMDP is proposed as a partially observable MDP.

Here, POMDP is used to determine the appropriate action for each possible belief state over a time step specified by the tuple (S, A, T, R, O, Z).

는 전이 확률을 나타내고,

는 선택된 작용에 대한 보상을 정의하고, O는 관측치를 정의하고, Z는 관측 함수이다.S and A are the state and behavior of the participant. each,

represents the transition probability,

defines the reward for the selected action, O defines the observation, and Z is the observation function.

In a POMDP system, we maintain a belief state (e.g. including position, velocity, yaw and yaw rate due to imperfect system state).

When an autonomous vehicle takes action (such as accelerating, decelerating and maintaining a desired speed) and receives observations via radar and vision sensors, a new belief state is obtained based on Bayes' rules.

The POMDP framework aims to maximize the expected reward underlying the Monte Carlo method, defined as

As shown in FIG. 1B, the apparatus for improved passage of autonomous vehicles at non-signalized intersections according to the present invention includes a state initialization unit 10 for initializing the learning state of the POMDP algorithm, and a POMDP model for optimizing autonomous vehicle operation. The optimal action derivation unit 20 for deriving the optimal behavior by receiving the RSS algorithm, the action execution unit 30 for executing the optimal behavior derived from the optimal behavior derivation unit 20, and the vision of the autonomous vehicle in the simulation. The state observation unit 40, which observes the driving state by receiving data observed from sensors and radar sensors, and corrects autonomous vehicle behavior based on the RSS algorithm and adaptive MPC system according to safety, non-safety, failure, and target compensation recognition. If the compensation level is not the target value as a result of the determination of the compensation determination unit 50, the compensation level determination unit 60 for determining whether the compensation level of the compensation determination unit 50 is appropriate, and the compensation level determination unit 60 It includes a status update unit 70 that updates the status.

2A and 2B are configuration diagrams for explaining a Partial Observability Markov decision process (POMDP) process.

Partial Observability Markov decision process (POMDP) is a process that enables decision-making of learning subjects based on partial observations of the environment.

In the present invention, in order to simulate the actual self-driving environment in a simulation environment, the basis for learning and behavioral decisions is based on data obtained through autonomous vehicle sensors (partial observation) rather than all environments (entire observation) of the simulation. It is used with the goal of determining and maximizing the reward of reinforcement learning for an action.

POMDP determines an action based on data (partial observation) obtained through an autonomous vehicle sensor and compensates for the action by reinforcement learning. route) and autonomous vehicle behavior (acceleration, maintenance, deceleration), state observation (simulation time, vehicle location, vehicle speed, data obtained from vehicle sensors), and negative compensation when safety distance is not secured for reinforcement learning compensation, It carries out the process of general compensation when securing a safe distance, target compensation based on whether the vehicle has reached the target, and failure compensation when an accident occurs.

The RSS method using the POMDP framework is described in detail as follows.

A major problem in the autonomous decision-making process is how to understand uncertainty and determine an appropriate driving strategy for an autonomous vehicle (self-driving vehicle). The present invention focuses on online optimization in a closed loop setting.

The model according to the present invention is a fusion of the POMDP algorithm and the RSS method based on the adaptive MPC system, which can find true safety guarantees for autonomous vehicles in uncertain environments (e.g., unpredictable human drivers).

3 is an operational flowchart illustrating a method for improved passage of an autonomous vehicle at an unsignalized intersection according to the present invention.

The method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention includes a state initialization step of initializing the learning state of the POMDP algorithm (S301), and RSS algorithm is provided to the POMDP model for optimizing autonomous vehicle operation The optimal action derivation step (S302) of deriving the optimal action, the action execution step (S303) of executing the optimal action derived in the optimal action derivation step, and the data observed by the vision sensor and radar sensor of the autonomous vehicle in the simulation A state observation step (S304) of receiving and observing the driving state) and a compensation determination step (S305) of modifying autonomous vehicle behavior based on the RSS algorithm and the adaptive MPC system according to recognition of safety, non-safety, failure, and target compensation (S305) ), a compensation level determination step (S306) for determining whether the reward level in the compensation determination step is appropriate, and a state update step (S307) for updating the status when the reward level is not the target value as a result of the determination in the compensation level determination step (S307). include

The belief state of the POMDP algorithm is a continuous state space denoted by

는 자율 주행 차량의 믿음 상태(belief state)를 의미한다.here,

Means a belief state of the autonomous vehicle.

는 인간 운전 차량의 상태이며,

는 이 모델에서 인간이 운전하는 차량의 수를 나타낸다.

is the state of the human-driven vehicle,

represents the number of human-driven vehicles in this model.

) 및 yaw rate(

)로 구성된다.The state is the location (x,y), speed (v), yaw (

) and yaw rate (

) is composed of

The action space is as follows.

), 감속(

) 및 원하는 속도 유지(

)가 포함되었으며, ACC 모델의 간격과 속도 제어에 의해 제어된다.The ego action according to the present invention includes acceleration (

), deceleration (

) and maintaining the desired speed (

), which is controlled by the spacing and speed control of the ACC model.

로 정의되는 반면 수동 구동 차량의 작용은 MPC 알고리즘에 의한 시간 단계로 연속 시간 동안 추정한다.ego action is

While defined as , the action of a manually driven vehicle is estimated over continuous time in time steps by the MPC algorithm.

An explanation of the observation space is as follows.

) 및 운전자 없는 차량과 선행 차량의 상대 거리(

)와 같은 요인으로 구성된다. Observations are position, velocity, yaw, yaw rate, relative velocity (

) and the relative distance between the driverless vehicle and the preceding vehicle (

) is composed of factors such as

The lead vehicle is sensed by the MIO track, which is the closest track in front of the ego vehicle. Vision and radar sensors can also measure the position and speed of manually driven vehicles relative to autonomous vehicles.

The observation function is defined as

The reward function and optimal behavior are explained as follows.

Automated vehicles must follow simple rules, such as maintaining a safe following distance.

Thus, compensation is considered for four aspects (i.e. safe, non-safe, failure and targeted compensation). Compensation is judged on autonomous observable parameters through sensor fusion (using vision and radar sensors) and passive vehicles.

The operation of the AV based on the RSS algorithm and the adaptive MPC system to execute an appropriate response can be optimized for the next time step based on the compensation value.

Therefore, the reward function and the optimal action are updated according to the following rules.

은 안전한 보상을 의미한다. 따라서 AV는 설정된 속도(예: ACC 시스템의 속도 제어)까지 속도를 유지하거나 가속할 수 있다.(One)

means a safe reward. Thus, the AV can maintain speed or accelerate up to a set speed (e.g. speed control in an ACC system).

은 안전하지 않은 보상을 의미하므로 AV는 안전 거리(예: ACC 시스템의 차량 내 간격 제어)가 복원될 때까지 최소 가속으로 감속한다.(2)

means unsafe compensation, so the AV decelerates to minimum acceleration until a safe distance (e.g. in-vehicle clearance control of the ACC system) is restored.

은 고장 보상을 나타내는 것으로, 폐쇄 루프 설정이 중지된다.(3)

indicates fault compensation, the closed loop setup is stopped.

(4) Closed loop setup stops as one of the vehicles reaching the target position safely marks the target reward.

The entire RSS algorithm through the POMDP framework is presented in detail with pseudocode as the following algorithm.

4 is a pseudo code format configuration diagram showing RSS algorithm optimization through the POMDP framework.

A step-by-step description of the RSS algorithm optimization process through the POMDP framework of FIG. 4 is as follows.

1: Collection of operating conditions S0, S1, S2, S3… …

2 : Application of RSS algorithm based on POMDP framework on simulation

3: Enter simulation parameters

4: Repeat every time series (response every 0.1s, 0.1s)

5: Initial state setting (setting the initial state of vehicles in simulation)

6: S0 = (x0 y0 v0 θ0 ω0) T

7: Sk = (xk yk vk θk ωk) T

8: Optimal driving strategy calculation

9: π(b):= argmax aQ(b,a)

9: Action Execution, Action Mode = [Acceleration, deceleration, speed maintenance]

10: Data collection from vision and radar sensors of autonomous vehicles in simulation

11: Observed data collection = = (x, y, v, q, w, rel_v, rel_d)T

12: Reinforcement learning reward function execution

13: Compensation R = [safety compensation, non-safety compensation, failure compensation, target compensation level]

14 : Update new state b = t(b, a, O)

15: Simulation repeat decision: R = [failure reward, good reward] until the target reward is reached.

5a and 5b are simulation experiment configuration diagrams for performance evaluation of the RSS-based POMDP model.

The performance of the RSS-based POMDP model was evaluated by simulating an increasing number of human-driven vehicles. That is, the effect of platoon vehicles on the proposed RSS-based POMDP model under the adaptive MPC system was considered.

The RSS-based POMDP model is implemented in the adaptive MPC system so that the classical adaptive MPC model is used for comparison with the proposed model.

Two case experiments and specific settings were created to evaluate the performance of the RSS algorithm for enhancing traffic safety assurance, improving smooth driving, and reducing latency.

The proposed model and the classical adaptive MPC model have the same safety distance and initial speed, and the autonomous vehicle used here uses sensor fusion and identified tracks to track the MIO (e.g., the closest human in front of the autonomous vehicle). driving vehicle) predicts the relative distance and relative speed. Autonomous vehicles must understand the behavior of the lead vehicle and decide whether to maintain a safe distance. The two experiments are as follows.

5A shows a first experimental environment, in which an autonomous vehicle attempts to make a left turn, and considers two vehicles with human drivers approaching from both directions. Human-driven vehicles are approaching in a straight line from both directions.

5B shows a second experimental environment, in which an autonomous vehicle attempts a left turn, and a situation in which multiple vehicles enter from both directions is simulated, and the interval between vehicles entering is set to 40 m.

As shown in Table 1, as setting values for simulation at unsignalized intersections to verify the model according to the present invention, simulation time unit, autonomous vehicle reaction time, autonomous vehicle initial speed, and human driver initial speed , the minimum acceleration of the vehicle, the maximum acceleration of the vehicle, the number of lanes on roads entering non-signalized intersections, the number of roads entering non-signalized intersections, and the width of lanes. These items are parameters for autonomous driving during actual operation. can be used as

The figure of merit considering the delay time is described as follows.

))부터 가속 시작 시간(

)까지 제동 시간(

)을 계산하여 지연 시간을 고려한 성능을 분석하였다.Deceleration start time based on the relative distance between the autonomous vehicle and the lead vehicle via vision and radar sensors in the simulation scenario (

)) from the acceleration start time (

) until braking time (

) was calculated to analyze the performance considering delay time.

The braking time is defined as:

)는 다음과 같이 정리된다.Therefore, the time figure of merit (

) is arranged as follows.

는 고전적인 적응형 MPC 모델의 제동시간,

은 고전적인 적응형 MPC 모델의 가속 시작 시간이고,

은 고전적인 적응형 MPC 모델의 감속 시작 시간이다.here,

is the braking time of the classic adaptive MPC model,

is the acceleration start time of the classical adaptive MPC model,

is the deceleration start time of the classical adaptive MPC model.

는 본 발명에 따른 모델의 제동 시간,

는 본 발명에 따른 모델의 가속 시작 시간이고,

는 본 발명에 따른 모델의 감속 시작 시간이다.

is the braking time of the model according to the present invention,

is the acceleration start time of the model according to the present invention,

is the deceleration start time of the model according to the present invention.

The figure of merit considering smooth operation will be described as follows.

)는 부드러운 주행 고려의 성능을 분석하기 위해 최소 속도(

)와 설정 속도(

)를 고려한다.smooth driving level score (

) is the minimum speed (

) and set speed (

) is taken into account.

는 적응형 MPC 차량 기준 속도, here,

is the adaptive MPC vehicle reference speed,

는 본 발명에 따른 모델의 차량 기준속도이다.

Is the vehicle reference speed of the model according to the present invention.

는 적응형 모델 차량 최소 속도,

is the adaptive model vehicle minimum speed,

는 본 발명에 따른 모델의 차량 최소 속도이다.

is the minimum vehicle speed of the model according to the present invention.

6A and 6B are simulation result graphs using the output profile of the first experiment, and FIGS. 7A and 7B are simulation result graphs using the output profile of the second experiment.

8 is a performance comparison graph between the model according to the present invention and the previous adaptive MPC model.

Smooth running tended to improve gradually from the first to the second scenario, where there was clearly an increase in collisions.

Table 2 and FIG. 8 show that no collisions between autonomous vehicles and human-driven vehicles were detected in all experiments.

For example, the collision state value of FIGS. 6A and 6B and 7A and 7B showing that there is no collision between an autonomous vehicle and a human-driven vehicle is 0.

Therefore, the automated vehicle moved safely. Also, in the second experiment, the highest improvement effect of the smooth driving level score (53.26%) was observed.

The highest time figure of merit (31.60%) occurred in the first experiment.

Comparing the adaptive MPC model according to the present invention and the classical adaptive MPC model at unsignalized intersections, the improvement in smooth driving showed an upward trend, and the reduction in delay time decreased when the number of manual vehicles gradually increased. .

Smooth driving performance according to the present invention considering group vehicles is as follows.

9 is a simulation result graph according to the autonomous vehicle speed in the model according to the present invention, and FIG. 10 is a simulation result graph through acceleration of the autonomous vehicle in the model according to the present invention.

Tables 3 and 4 show the corresponding minimum, maximum and standard deviations of the two experiments.

Using RSS algorithms, the AV automatically tracked its relative distance and relative speed to guided vehicles (human-driven vehicles) while maintaining a safe distance.

As shown in FIG. 9 , the AV speed may always follow the speed set at the start of the simulation. When the relative distance between the two vehicles was less than the braking distance, the AV decelerated with minimum acceleration (-5.0 m/s2) until the safe distance was restored.

The standard deviation of the velocity distribution and acceleration distribution of the first experiment was smaller than that of the second experiment. Therefore, the fluctuation range of the first experiment was smaller than that of the second experiment in terms of velocity and acceleration distribution.

The first experiment was smoother than the second in terms of speed and acceleration. In other words, the improvement in smooth running was less than when the number of manual vehicles increased in the model according to the present invention.

The device and method for improved passage of autonomous vehicles at unsignalized intersections according to the present invention described above enable efficient passage at unsignalized intersections by utilizing the safety theory of responsibility sensitivity and the partially observed Markov decision procedure.

The present invention is a method of learning through information within the range that an autonomous vehicle can observe, such as in a real situation, and maximizes the reward of reinforcement learning for actions by using a Markov decision-making model (Partial Observability MDP, POMDP), which is reinforcement learning. that made it possible

As described above, it will be understood that the present invention is implemented in a modified form without departing from the essential characteristics of the present invention.

Therefore, the specified embodiments should be considered from an explanatory point of view rather than a limiting point of view, and the scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range are considered to be included in the present invention. will have to be interpreted

10. State Initialization Unit 20. Optimal Action Derivation Unit
30. Action Execution Unit 40. State Observation Unit
50. Compensation determination unit 60. Compensation level determination unit
70. Status update unit

Claims (19)

a state initialization unit that initializes a learning state of a Partial Observability Markov decision process (POMDP) algorithm;
An optimal action derivation unit that derives an optimal action by applying a Responsibility-Sensitive Safety (RSS) to a POMDP model for optimizing autonomous vehicle (AV) operation;
an action execution unit that executes the optimal behavior derived from the optimal behavior derivation unit;
a state observation unit that receives data observed from a vision sensor and a radar sensor of an autonomous vehicle (AV) and observes a driving state;
Compensation decision unit that modifies autonomous vehicle behavior based on the RSS algorithm for autonomous vehicles and the Adaptive Model Predictive Control System for human-driven vehicles according to safety, non-safety, failure, and target reward recognition. including;
By applying the Adaptive Model Predictive Control System to the human-driving vehicle, the relative distance and relative speed to the nearest human driver in front obtained from the sensor of the autonomous vehicle are identified, and the autonomous driving is controlled in the control variables. A device for improved passage of autonomous vehicles at non-signalized intersections, characterized in that the vehicle can operate in response to the behavior of a human driver in a way that autonomously maintains a certain distance from a human-driven vehicle in front. The method according to claim 1, further comprising: a compensation level determination unit determining whether the compensation level of the compensation determination unit is appropriate;
An apparatus for improved passage of autonomous vehicles at non-signalized intersections, further comprising a state update unit for updating a state when the compensation level is not the target value as a result of the determination of the compensation level determination unit. The method of claim 1, wherein the POMDP model determines an action based on data obtained through an autonomous vehicle sensor and compensates for the action by reinforcement learning,
Autonomous vehicle and human driver status, including time, vehicle position, speed, and determined route items;
Self-driving car behavior including acceleration, maintenance, and deceleration,
condition observation, including simulation time, vehicle position, vehicle speed, and data obtained from vehicle sensors;
For reinforcement learning compensation, non-safety compensation when the safety distance is not secured, safety compensation when the safety distance is secured, target compensation based on whether the vehicle has reached the target, and failure compensation in the event of an accident are performed. A device for improved passage of autonomous vehicles at signalized intersections. 4. The method of claim 3, the belief state of the POMDP model is

is a continuous state space denoted by
here,

is the belief state of the autonomous vehicle,
cast

is the state of the human-driven vehicle,

Apparatus for improved passage of autonomous vehicles at unsignalized intersections, characterized in that is the number of human-driven vehicles in this model. The method of claim 4, wherein the state is the location (x, y), speed (v), yaw (

) and yaw rate (

) Device for improved passage of autonomous vehicles at non-signalized intersections, characterized in that consisting of. The method of claim 3, wherein the ego action in the action space includes acceleration (

), deceleration (

) and maintaining the desired speed (

) is included and controlled by the spacing and speed control of the ACC model,
Observation in the observation space is position, velocity, yaw, yaw rate, relative velocity (

) and the relative distance between the driverless vehicle and the preceding vehicle (

) Device for improved passage of autonomous vehicles at unsignalized intersections, characterized in that consisting of factors. The method of claim 3, wherein the reward function and the optimal action in the reward decision unit are:
(One)

means safe compensation, and AV maintains speed up to the set speed or accelerates;
(2)

means unsafe compensation, AV decelerates to minimum acceleration until safe distance is restored,
(3)

indicates fault compensation, closed loop setting stops,
(4) A device for improved passage of autonomous vehicles at unsignalized intersections, characterized in that when one of the vehicles reaching the target position safely marks the target reward, it is updated according to the closed loop setting stop rule. The method of claim 1, wherein a safe distance is maintained to avoid dangerous situations by applying an RSS algorithm to mixed traffic to optimize autonomous vehicle operation,
safety distance (

)Is,

is defined as,
here,

is the reaction time of AV,

is the braking distance,

is the response time of AV,

is the actual speed of AV,

Apparatus for improved passage of autonomous vehicles at non-signalized intersections, characterized in that is the minimum acceleration of AV. 9. The method of claim 8, wherein AV is the response time.

accelerating until reaching a maximum acceleration at , and decelerating by a minimum acceleration after a response time to maintain a safe distance from a manually driven vehicle;
The autonomous driving decision function is

If , both vehicles follow normal driving and set speed, if

When , the autonomous vehicle decelerates with minimum acceleration until the safe distance is restored. delete The method of claim 1, as a set value for simulation at an unsignalized intersection,
Unit of simulation time, response time of autonomous vehicle, initial speed of autonomous vehicle, initial speed of human driver, minimum acceleration of vehicle, maximum acceleration of vehicle, number of lanes of roads entering non-signalized intersections, number of roads entering non-signalized intersections , A device for improved passage of autonomous vehicles at an unsignalized intersection, comprising a lane width item. The method of claim 11, for performance analysis considering delay time,
Deceleration start time based on the relative distance between the autonomous vehicle and the lead vehicle via vision and radar sensors in the simulation scenario (

)) from the acceleration start time (

) until braking time (

Apparatus for improved passage of autonomous vehicles at non-signalized intersections, characterized in that the performance is analyzed considering the delay time by calculating ). 13. The method of claim 12, wherein the braking time is

is defined as,
time performance index (

)Is,

,

,

is defined as,
here,

is the braking time of the classic adaptive MPC model,

is the acceleration start time of the classical adaptive MPC model,

is the deceleration start time of the classical adaptive MPC model,

is the braking time of the proposed model,

is the acceleration start time of the proposed model,

Apparatus for improved passage of autonomous vehicles at non-signalized intersections, characterized in that is the deceleration start time of the proposed model. 13. The method of claim 12, wherein the smooth driving level score (

) is the minimum speed (

) and set speed (

),

is defined as,

,

,
here,

is the adaptive MPC vehicle reference speed,

is the vehicle reference speed of the proposed model,

is the adaptive model vehicle minimum speed,

Apparatus for improved passage of autonomous vehicles at non-signalized intersections, characterized in that is the minimum vehicle speed of the proposed model. A state initialization step of initializing a learning state of a Partial Observability Markov decision process (POMDP) algorithm;
An optimal action derivation step of deriving an optimal action by applying a Responsibility-Sensitive Safety (RSS) to a POMDP model for optimizing autonomous vehicle (AV) operation;
an action execution step of executing the optimal action derived in the optimal action derivation step;
A state observation step of observing a driving state by receiving data observed from a vision sensor and a radar sensor of an autonomous vehicle (AV);
A compensation determination step of modifying autonomous vehicle behavior based on the RSS algorithm of autonomous vehicles and the Adaptive Model Predictive Control System of human-driven vehicles according to safety, non-safety, failure, and target reward recognition; include,
According to the application of the Adaptive Model Predictive Control System for human-driven vehicles, the relative distance and relative speed with the nearest human driver obtained from the sensor of the self-driving vehicle are identified, and the self-driving vehicle in the control variables A method for improved passage of an autonomous vehicle at an unsignalized intersection, characterized in that it can operate in response to the behavior of a human driver in a way that autonomously maintains a certain distance from the vehicle in front. [Claim 16] The method of claim 15, further comprising: a compensation level determination step of determining whether the compensation level of the compensation determination step is appropriate;
A method for improved passage of autonomous vehicles at non-signalized intersections, further comprising a state update step of updating a state when the compensation level is not the target value as a result of the determination in the compensation level determination step. The method of claim 15, wherein the POMDP model determines an action based on data obtained through an autonomous vehicle sensor and compensates for the action by reinforcement learning,
Autonomous vehicle and human driver status, including time, vehicle position, speed, and determined route items;
Self-driving car behavior including acceleration, maintenance, and deceleration,
condition observation, including simulation time, vehicle position, vehicle speed, and data obtained from vehicle sensors;
For reinforcement learning compensation, non-safety compensation when safety distance is not secured, safety compensation when safety distance is secured, target compensation based on whether the vehicle has reached the target, and failure compensation when an accident occurs. Methods for improved passage of autonomous vehicles at signalized intersections. 16. The method of claim 15, wherein in the reward determination step, the reward function and the optimal action are:
(One)

means safe compensation, and AV maintains speed up to the set speed or accelerates;
(2)

means unsafe compensation, AV decelerates to minimum acceleration until safe distance is restored,
(3)

indicates fault compensation, closed loop setting stops,
(4) A method for improved passage of autonomous vehicles at unsignaled intersections, characterized in that when one of the vehicles reaching the target position safely marks the target reward, it is updated according to the rule of stopping the closed loop setting. delete

Images