KR20220102395A - System and Method for Improving of Advanced Deep Reinforcement Learning Based Traffic in Non signalalized Intersections for the Multiple Self driving Vehicles - Google Patents

Abstract

The present invention relates to a device and a method for reinforcement learning-based traffic improvement at a non-traffic signal intersection for group driving of an autonomous vehicle, which can improve traffic at a non-traffic signal intersection and secure safety through autonomous vehicle group-driving learning in a mixed traffic situation in which a group of autonomous vehicles and human driver-based vehicles are mixed. The device comprises: a simulation-of-urban-mobility (SUMO) simulation execution unit which builds a simulation environment by utilizing SUMO and transfers data obtained from a speed, a position, and a sensor of an autonomous vehicle to a FLOW application unit; the FLOW application unit which builds a simulation environment in a reinforcement learning platform FLOW environment that can be linked with SUMO, derives driving behaviors without application of reinforcement learning, controls vehicles, updates a simulation state, and transfers state and reward information to a reinforcement learning library environment building unit; and the reinforcement learning library environment building unit which optimizes traffic control with multi-agent deep reinforcement learning by using the reinforcement learning platform FLOW that can be linked with SUMO. The SUMO simulation execution unit presents results for AV penetration rates from 1-100 % in a unit of 10 %, in a situation where a group of vehicles approach a non-traffic signal intersection and drive straight along four different directions, and at the non-traffic signal intersection, lane changes and turning left with respect to all vehicles are disregarded.

Description

BACKGROUND ART System and Method for Improving of Advanced Deep Reinforcement Learning Based Traffic in Non signalalized Intersections for the Multiple Self driving Vehicles

The present invention relates to the operation control of a plurality of autonomous vehicles, and specifically, in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed, autonomous vehicle platooning learning improves non-signal intersection traffic and secures safety It relates to an apparatus and method for reinforcement learning-based traffic improvement at non-signal intersections for autonomous vehicle cluster operation.

An autonomous vehicle is a vehicle equipped with a technology that recognizes a lane and performs automatic steering using a camera or a front object detection sensor. For autonomous vehicles, the lane width, the lateral position of the vehicle on the lane, the distance to both lanes and the shape of the lane, and the radius of curvature of the road are measured based on the camera's image processing or sensing of the front object. The vehicle's driving trajectory is estimated using the location and road information, and the lane is changed according to the estimated driving trajectory.

An autonomous vehicle automatically controls the vehicle's throttle valve, brake, and transmission through the location and distance of the preceding vehicle detected by a camera mounted on the front of the vehicle or a front object detection sensor to perform appropriate acceleration/deceleration. You can also make sure you keep an appropriate distance from the vehicle.

However, when such an autonomous vehicle passes through an intersection, it detects the movement of the preceding vehicle and then departs after stopping according to the traffic signal of a traffic light. can

In particular, when the driving environment is grasped using information input from a sensor, such as an autonomous vehicle, driving at a non-signaled intersection becomes a much more difficult task than driving on a general road.

On the other hand, IoT-related research is being actively conducted due to the development of wireless communication technology, and the Internet of Vehicles (IoV) is attracting attention as such. Vehicular Ad-hoc Network (VANET), a wireless network in which each vehicle acts as a node for communication between vehicles, is a form of Mobile Ad-hoc Network (MANET).

Simulation of Urban Mobility (SUMO) is an open source designed to simulate traffic networks on roads.

Using SUMO, traffic flow can be predicted by understanding the movement between vehicles on the road.

Although studies are being conducted for efficient driving at non-signal intersections by understanding the driving environment of autonomous vehicles through these technologies, There are still many challenges to be solved at non-signal intersections.

Therefore, there is a demand for the development of new technologies for improving the traffic at non-signal intersections and securing safety according to the platooning of autonomous vehicles.

Republic of Korea Patent Publication No. 10-2020-0071406 Republic of Korea Patent Publication No. 10-2020-0058613 Republic of Korea Patent Publication No. 10-2018-0065196

The present invention is to solve the problems of the autonomous vehicle operation control technology of the prior art, and improves the non-signaled intersection through self-driving vehicle platooning learning in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed. The purpose of the present invention is to provide an apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous driving vehicles to ensure safety and safety.

The present invention is a method of learning through information within the range that an autonomous vehicle can observe as in a real situation, and rewards reinforcement learning for behavior by applying reinforcement learning and using a Markov decision-making model (Partial Observability MDP, POMDP). An object of the present invention is to provide an apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for maximizing autonomous vehicle swarm operation.

The present invention is a method of learning the driving behavior of an autonomous driving vehicle in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed. An object of the present invention is to provide an apparatus and method for improving traffic based on reinforcement learning at a non-signaling intersection for the operation of a group of autonomous vehicles.

The present invention builds an experimental environment using SUMO (Simulation of Urban Mobility), defines a human driver with an ACC (Adaptive Cruise Control) system, and utilizes multi-agent deep reinforcement learning using FLOW, a reinforcement learning platform that can be linked with SUMO. (Multi agent Deep Reinforcement Learning) The purpose of this is to provide a device and method for improving traffic based on reinforcement learning at non-signaled intersections for autonomous vehicle swarm operation that enables traffic control to be optimized.

The present invention is for the operation of a group of autonomous vehicles that can improve the average travel speed in a fully autonomous vehicle environment compared to a fully human driver environment at non-signal intersections by adjusting reinforcement learning parameters and optimizing and verifying operation by autonomous vehicle share. An object of the present invention is to provide an apparatus and method for improving traffic based on reinforcement learning at non-signaling intersections.

The present invention determines the behavior of an autonomous vehicle in a simulation environment according to the partial observation Markov decision-making process (POMDP), learns the average speed as a reward, and applies a PPO (Proximal Policy Optimization) algorithm to perform multi-agent deep reinforcement learning. The purpose of this study is to provide a device and method for improving traffic based on reinforcement learning at non-signaled intersections for autonomous vehicle swarm operation that can optimize behavioral decisions.

In order to simulate the real autonomous driving environment in the simulation environment, the present invention determines the behavior based on the data (partial observation) obtained through the autonomous driving vehicle sensor, not the entire environment of the simulation, as the basis for learning and behavior decision. The purpose of this study is to provide an apparatus and method for improving the reinforcement learning-based traffic at non-signaled intersections for autonomous vehicle swarm operation so that the reward of reinforcement learning can be maximized.

Other objects of the present invention are not limited to the above-mentioned objects, and other objects not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the above object, the apparatus for improving traffic based on reinforcement learning at non-signal intersections for autonomous vehicle cluster operation according to the present invention for achieving the above object is to build a simulation environment using SUMO (Simulation of Urban Mobility), and SUMO simulation execution unit that transmits the data obtained from the speed, position, and sensor of the driving vehicle to the flow application unit; a reinforcement learning platform that can be linked with SUMO builds a simulation environment in the FLOW environment and derives driving behavior without reinforcement learning; FLOW application unit that updates vehicle control and simulation status and delivers status and reward information to reinforcement learning library environment building unit; Multi agent deep reinforcement learning using FLOW, a reinforcement learning platform that can be linked with SUMO and a reinforcement learning library environment construction unit for optimizing passage control.

The method for improvement of traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention for achieving another object is to build a simulation environment using SUMO (Simulation of Urban Mobility), and autonomous vehicle SUMO simulation execution step in which the data obtained from the speed, position, and sensor of and FLOW application step to update the simulation state and deliver the state and reward information to the reinforcement learning library environment building unit; use FLOW, a reinforcement learning platform that can be linked with SUMO, to pass through Multi-agent Deep Reinforcement Learning It characterized in that it comprises; the step of constructing a reinforcement learning library environment for optimizing the control.

As described above, the apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention have the following effects.

First, in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed, self-driving vehicle platooning learning improves non-signal crossing traffic and ensures safety.

Second, as in a real situation, it is a method of learning through information within the range that an autonomous vehicle can observe, and by applying reinforcement learning and using a Markov decision-making model (Partial Observability MDP, POMDP), the reward of reinforcement learning for behavior is maximized. make it possible

Third, it is possible to optimize traffic control by applying PPO, an algorithm for optimizing learning in artificial neural networks, as a method of learning the driving behavior of autonomous vehicles in a mixed traffic flow situation where clustered autonomous vehicles and human driver vehicles are mixed.

Fourth, build an experimental environment using SUMO (Simulation of Urban Mobility), define a human driver with the ACC (Adaptive Cruise Control) system, and utilize multi-agent deep reinforcement learning (FLOW), a reinforcement learning platform that can be linked with SUMO. Multi agent Deep Reinforcement Learning) to optimize traffic control.

Fifth, it is possible to improve the average travel speed in a fully autonomous vehicle environment compared to a fully human driver environment at non-signal intersections by adjusting reinforcement learning parameters and optimizing and verifying operation by autonomous vehicle share.

Sixth, according to the partial observation Markov decision-making process (POMDP), the behavior of the autonomous vehicle in the simulation environment is determined, the average speed is learned as a reward, and the PPO (Proximal Policy Optimization) algorithm is applied to perform multi-agent deep reinforcement learning. to optimize action decisions.

Seventh, in order to simulate the real autonomous driving environment in the simulation environment, the basis for learning and behavior decision is determined based on the data (partial observation) obtained through the autonomous vehicle sensor rather than the entire environment of the simulation, and the behavior is determined. Maximize the rewards of reinforcement learning.

1 is a block diagram showing a deep reinforcement learning architecture at a non-signal intersection according to the present invention
2 is a PPO algorithm using an adaptive KL penalty algorithm.
3 is a configuration diagram of a device for improving traffic based on reinforcement learning at a non-signal intersection for group operation of autonomous driving vehicles according to the present invention
4 is an operation flowchart showing a method for improving traffic based on reinforcement learning at a non-signal intersection for group operation of autonomous vehicles according to the present invention;
5 is a configuration diagram of a general SUMO simulator at a non-signal intersection.
6 is a configuration diagram showing an example of a general observation space (Observation Space);
7 is a configuration diagram showing experimental characteristics of a leading autonomous driving vehicle at a non-signal intersection.
8 is an experimental comparison diagram at a non-signaled intersection.
9 is a graph of the average compensation curve over 200 times based on AV occupancy.
10 is a graph of spatiotemporal dynamics through AV occupancy at non-signaled intersections.
11 is a graph showing the average speed, average delay time, average fuel consumption, and average exhaust gas value derivation characteristic graph in a SUMO simulation environment.

Hereinafter, a preferred embodiment of an apparatus and method for improving traffic based on reinforcement learning at a non-signal intersection for group operation of autonomous vehicles according to the present invention will be described in detail as follows.

Features and advantages of the apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention will become apparent through detailed description of each embodiment below.

1 is a block diagram showing a deep reinforcement learning architecture at a non-signal intersection according to the present invention, and FIG. 2 is a PPO algorithm using an adaptive KL penalty algorithm.

The apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for autonomous vehicle swarm operation according to the present invention is autonomous vehicle platooning learning in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed. This is to improve traffic at non-signaled intersections and ensure safety.

To this end, the present invention provides reinforcement learning for behavior by applying reinforcement learning and using Markov decision-making models (Partial Observability MDP, POMDP) as a method of learning through information within the range that an autonomous vehicle can observe, such as in a real situation. It may include a configuration to maximize the compensation of.

The present invention is a method of learning the driving behavior of an autonomous driving vehicle in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed. It may include a configuration that

The present invention builds an experimental environment using SUMO (Simulation of Urban Mobility), defines a human driver with an ACC (Adaptive Cruise Control) system, and utilizes multi-agent deep reinforcement learning using FLOW, a reinforcement learning platform that can be linked with SUMO. (Multi agent Deep Reinforcement Learning) may include a configuration to optimize traffic control.

The present invention is a configuration that determines the behavior of an autonomous driving vehicle in a simulation environment according to the reinforcement learning parameter adjustment, operation optimization and verification configuration by autonomous driving vehicle share, and partial observation Markov decision-making process (POMDP), and learns the average speed as a reward, In order to do multi-agent deep reinforcement learning, it may include a configuration for optimizing behavioral decisions by applying a PPO (Proximal Policy Optimization) algorithm.

Reinforcement learning (RL) is a subdomain of machine learning, in which agents learn actions to interact with their environment and maximize their cumulative reward.

A typical form of the RL algorithm is the Markov Decision Process (MDP), a powerful framework used to determine the appropriate behavior given the entire set of observations.

,

, T)이며, 여기서 S와 A는 각각 참가자의 상태와 행동이다.

는 전이 확률을 정의하며,

은 선택된 작용에 따라 보상을 정의하며,

은 초기 상태 분포를 정의하며,

는 0에서 1까지의 할인 계수(discount factor)를 정의하며, T는 시간 범위를 나타낸다. MDP is a tuple (S, A, P, R,

,

, T), where S and A are the participant's state and behavior, respectively.

defines the transition probability,

defines the reward according to the selected action,

defines the initial state distribution,

defines a discount factor from 0 to 1, and T represents the time range.

However, automated vehicles maneuver in uncertain environments that include inaccuracies, intent, and sensor noise. To solve this problem, partially observable MDP (POMDP) has been proposed, which uses two more elements: O, which defines a set of observations, and Z, which is an observation function.

를 최적화하여 몇 가지 타임 스텝에 걸쳐 예상 누적 할인 보상을 극대화한다.RL's objective learning agent is a policy

to maximize the expected cumulative discount reward over several time steps.

Deep neural networks (DNNs) have the ability to automatically perform shape extraction due to multiple hidden representation layers. For continuous controllers, artificial neural networks (ANNs) are a commonly used method that uses multiple hidden layers to represent complex functions.

In this task, MLP is applied to generate an output set (policy) from a set of inputs (states and observations). In addition, we apply PPO based on gradient descent optimization method to improve the performance of DNN.

The proposed deep RL framework, which fuses MLP and RL, is designed to consider the effects of AV at non-signaled junctions.

First, the SUMO simulator runs one simulation step.

Second, the Flow framework sends information about the state of the SUMO simulator to the RL library. Then, the RL library (RLlib) calculates appropriate actions according to the state of the SUMO simulator through MLP. The MLP policy is applied to maximize the cumulative compensation for the RL algorithm based on the traffic data.

Finally, the simulation resets and repeats the RL process.

1 shows a deep reinforcement learning architecture at a non-signaling intersection.

Importantly, 'policy' refers to the blueprint for communication between perceptions and actions in the environment. In other words, the policy is analogous to the controller of the traffic simulation.

In this task, the controller is an MLP policy with several hidden layers.

The parameters of the controller are iteratively updated using MLP policies to maximize cumulative compensation based on traffic data sampled from the SUMO simulator.

The main goal of the agent is to learn how to optimize a probabilistic policy as follows:

는 할인 계수(

)와 보상(

)에 의해 계산되는 예상 누적 할인 보상이다.here,

is the discount coefficient (

) and reward (

) is the expected cumulative discount reward calculated by

Longitudinal Dynamic Models are described as follows.

Basic vehicle dynamics can be defined by a vehicle-following model that describes the longitudinal dynamics of a manually operated vehicle based on observations of the vehicle itself and the vehicle in front.

Standard vehicle-following models are as follows:

는 차량 i의 가속도이고,

는 비선형 함수이며,

,

및

는 각각 차량 i의 속도, 상대 속도 및 방향이다.here,

is the acceleration of vehicle i,

is a nonlinear function,

,

and

are the speed, relative speed and direction of vehicle i, respectively.

In the present invention, IDM, a type of ACC system, is applied for longitudinal control of a human-driven vehicle due to its ability to depict driver behavior.

IDM is a commonly used car-following model.

In the acceleration command of the IDM, the vehicle speed in the non-signaled intersection environment, the identification (ID) of the leading vehicle, and the headway of the leading vehicle may be set to be obtained by a “get” method.

The vehicle's acceleration is calculated as follows.

은 차량의 가속이고,

는 원하는 속도이며,

는 가속도지수,

는 차량의 앞길(앞차와의 거리)이며,

는 원하는 방향을 나타내며, 다음과 같이 표현된다.here,

is the acceleration of the vehicle,

is the desired speed,

is the acceleration index,

is the front road of the vehicle (distance from the vehicle in front),

represents the desired direction, and is expressed as follows.

는 최소 갭을, T는 a시간 갭을,

는 선두 차량과 비교한 속도 차이(현재 속도 - 선두 속도), a는 가속 구간, b는 편안한 감속을 나타낸다.here,

is the minimum gap, T is the time gap,

is the speed difference compared with the leading vehicle (current speed - leading speed), a is the acceleration section, and b is the comfortable deceleration.

Representative parameters of the IDM controller for urban traffic are shown in Table 1.

Policy Optimization will be described as follows.

Policy gradient methods attempt to compute an estimator of a parameterized policy function using a gradient descent algorithm rather than an action-value or state-value function.

Thus, it avoids the convergence problems that arise in the estimation function due to nonlinear approximation and partial observation.

)의 확률에 대한 기대치와 시간 스텝 t

에서의 어드밴티지 함수(advantage function)의 추정치에 기초하는 정책 경사법은 다음과 같이 표현된다.The present invention applies the MLP policy to directly optimize the control policy in the simulation of the non-signaled intersection. policy action (

) and the time step t

A policy gradient method based on an estimate of the advantage function in

는 유한한 표본 배치에 대한 기대 연산자이며,

는 확률적 정책을 나타내며,

는 디스카운트된 보상 합계와 기준 추정치로 정의되며,

와

는 시간 스텝 t의 행동과 상태를 각각 나타낸다.here,

is the expected operator for a finite sample batch,

represents a probabilistic policy,

is defined as the sum of the discounted rewards and the baseline estimate,

Wow

denotes the behavior and state of time step t, respectively.

The PPO proposed by Schulman et al. is a simple TRPO provided by the RLlib library.

In other words, the goal of the PPO is the same as the TRPO, and the TRPO uses a trusted region constraint to force the policy to be updated so that the new policy is not too far from the old one.

There are two types of PPOs: adaptive Kullback-Leibler (KL) penalties and clipped objectives.

The PPO adopts a surrogate loss function to generate policy updates. This process prevents performance degradation during the training process.

)는 다음과 같이 설명된다.surrogate object (

) is described as

는 업데이트 전 정책 매개 변수,

는 업데이트 후 정책 매개 변수,

는 확률비를 나타낸다.here,

is the pre-update policy parameter,

is the policy parameter after update,

represents the probability ratio.

For continuous actions, the policy output of the PPO is the parameter of the Gaussian distribution for each action.

The policy then generates continuous outputs based on these distributions.

In the present invention, the PPO with adaptive KL penalty is used to optimize the KL penalty target using minibatch stochastic gradient descent (SGD) as follows.

는 매 정책 업데이트 후 업데이트되는 가중 조절 계수(weight control coefficient)이다.here,

is a weight control coefficient updated after every policy update.

It increases when the current KL difference is greater than the target KL deviation, and decreases when the current KL divergence is less than the target KL divergence.

In the PPO algorithm, the current policy first interacts with the environment to create a sequence of episodes. Next, an advantage function is estimated using the reference estimate for the state value.

Finally, we collect all the experiences and run the gradient descent algorithm through the policy network. The overall PPO of the adaptive KL penalty algorithm is represented by the pseudo code of Algorithm 1 in FIG.

3 is a block diagram of a device for improving traffic based on reinforcement learning at a non-signal intersection for group operation of autonomous vehicles according to the present invention.

The device for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention builds a simulation environment using SUMO (Simulation of Urban Mobility), and the speed, position, and sensor of the autonomous vehicle The SUMO simulation execution unit 100 that delivers the data obtained from the FLOW to the FLOW application unit 200, and the reinforcement learning platform FLOW environment that can be linked with SUMO builds a simulation environment, and derives driving behavior that does not apply reinforcement learning; Multi-agent deep reinforcement using the FLOW application unit 200 that updates the vehicle control and simulation status and delivers status and reward information to the reinforcement learning library environment construction unit 300, and the reinforcement learning platform FLOW that can be linked with SUMO and a reinforcement learning library environment construction unit 300 that optimizes traffic control by learning (Multi agent Deep Reinforcement Learning).

Here, the SUMO simulation execution unit 100 builds a simulation environment using SUMO (Simulation of Urban Mobility) and performs the SUMO simulation, and the SUMO simulation unit 10 generates exhaust gas, velocity, and position value files and autonomously It includes a result file generating unit 11 that transmits the data obtained from the speed, position, and sensor of the driving vehicle to the FLOW application unit 200 .

And the FLOW application unit 200 is a simulation initialization unit 20 that sets the simulation environment, such as human driver definition, deep reinforcement learning input value setting and vehicle speed, acceleration, starting point, etc. The flow environment construction unit 21 that delivers to the reinforcement learning library, the driving behavior derivation unit 22 that derives driving behavior without reinforcement learning applied, and the vehicle control and control information are transferred to the SUMO simulation execution unit 100 The vehicle control module 23 that delivers, and the update unit 24 that receives the simulation state from the SUMO simulation execution unit 100, updates the simulation state, and transmits the state and reward information to the reinforcement learning library environment construction unit 300. include

And the reinforcement learning library environment construction unit 300 includes a reinforcement learning library 31 receiving a state from the FLOW application unit 200, a data sampling unit 32 sampling data to be learned, and a driving behavior ( Policy) The policy training unit 33 for training, the training result evaluation unit 34 that evaluates the training results and delivers the learned behavior (driving method) to the FLOW application unit 200, and the autonomous vehicle can observe A policy optimization unit 35 that maximizes the reward of reinforcement learning for behavior by applying reinforcement learning and the use of a Markov decision-making model (Partial Observability MDP, POMDP) as a method of learning through information within a given range; and FLOW It includes a policy update storage unit 36 that receives the state of the autonomous vehicle from the application unit 200 and updates and stores the policy, and a learning loop condition determination unit 37 that determines whether the updated policy satisfies the learning loop condition. do.

4 is an operation flowchart illustrating a method for improving traffic based on reinforcement learning at a non-signal intersection for group operation of autonomous vehicles according to the present invention.

The method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention builds a simulation environment using SUMO (Simulation of Urban Mobility), and the speed, position, and sensor of the autonomous vehicle The SUMO simulation execution step of delivering the data obtained from the FLOW to the FLOW application unit 200, the reinforcement learning platform that can be linked with SUMO, builds a simulation environment in the FLOW environment, derives driving behavior without reinforcement learning, vehicle control and Multi-agent deep reinforcement learning (Multi-agent Deep Reinforcement) using the FLOW application step that updates the simulation state and delivers the state and reward information to the reinforcement learning library environment construction unit 300, and the reinforcement learning platform FLOW that can be linked with SUMO Learning), including the step of building a reinforcement learning library environment that optimizes traffic control.

Here, the SUMO simulation execution step includes the SUMO simulation step (S409) in which the SUMO simulation is performed by building a simulation environment using SUMO (Simulation of Urban Mobility), and the exhaust gas, speed, and position value files are generated to control the autonomous driving vehicle. It includes a result file creation step (S410) of transferring the data obtained from the speed, position, and sensor to the FLOW application unit 200 .

And the FLOW application step includes a simulation initialization step (S401) that sets the simulation environment such as human driver definition, deep reinforcement learning input value setting and vehicle speed, acceleration, and starting point, and a FLOW environment construction to set the state to the reinforcement learning library. A vehicle that controls the vehicle and transmits the control information to the SUMO simulation execution unit 100 with a flow environment construction step (S402) that delivers to It includes a control step (S408) and an update step (S411) of receiving the simulation state from the SUMO simulation execution unit 100, updating the simulation state, and transferring the state and reward information to the reinforcement learning library environment construction unit 300.

And the step of constructing the reinforcement learning library environment is a step (S404) of the reinforcement learning library 31 receiving a state from the FLOW application unit 200, a data sampling step of sampling the data to be learned (S405), and driving A policy training step (S406) of behavior (policy) training, a training result evaluation step (S407) of evaluating the training result and delivering the learned behavior (driving method) to the FLOW application unit 200, and the autonomous vehicle Policy optimization step to maximize the reward of reinforcement learning for behavior using reinforcement learning and Markov decision-making model (Partial Observability MDP, POMDP) as a method of learning through information within the observable range (S412) And, a policy update storage step (S413) of receiving the state of the autonomous vehicle from the FLOW application unit 200 and updating and storing the policy, and a learning loop condition determination step (S414) of determining whether the updated policy satisfies the learning loop condition ) is included.

5 is a schematic diagram of a general SUMO simulator at a non-signal intersection.

Developed by the Transport Systems Research Center at the German Aerospace Center, SUMO is an open-source microscopic transport simulator. SUMO can simulate city-scale transport networks with traffic lights, vehicles, pedestrians and public transport. TraCI also makes it possible to connect SUMO to Python to apply deep RL to the SUMO simulator.

A general SUMO simulator at a non-signaled intersection is shown in FIG. 5 .

Flow, developed at UC Berkeley, provides an interface between deep RL algorithms and custom road networks. Flow can also analyze and validate training policies.

Flow's advantages include the ability to easily implement various road networks to improve the control of autonomous vehicles with deep RL. In Flow, custom environments can be used to create key subset classes, including initialized simulations, observation spaces, state spaces, workspaces, controllers, and compensation functions for various scenarios.

The initialized simulation represents the initial setup of the simulation environment for the starting episode.

In the present invention, position, velocity, acceleration, starting point, trajectory and number of vehicles are set as well as parameters of IDM rules and deep RL framework.

In particular, the trajectory of every vehicle is established in the initial simulation process by the SUMO simulator, including specific nodes (point locations in the network), specific edges (connecting nodes together) and specific paths (sequences through which edge vehicles pass).

Next, the acceleration of the human-driven vehicle is controlled by the SUMO simulator and the acceleration of the AV is controlled by the Rllib library.

6 is a configuration diagram illustrating an example of a general observation space.

The observation space represents the number and type of observable features such as AV velocity (own vehicle velocity), AV position (own vehicle position) and its preceding and AV velocity and bumper-to-bumper headway.

An observable output is fed into the state space to predict the appropriate policy.

And the state space represents the vector of autonomous agents and surrounding vehicles based on the observation space, including the before and after AV as well as the position and velocity of the AV.

Functions within the environment are extracted using the get_state method and fed into the policy.

First, we get the IDs of all vehicles at the unsigned intersection. It then determines the positions and velocities of all vehicles to create a state space.

The important thing is that your current location is based on a predefined starting point.

The state space is defined as

,

및

는 각각 AV, 이전 AV 및 다음 AV의 해당 속도이고, where S is the state of a specific vehicle, x0 is the corresponding coordinate of AV,

,

and

are the corresponding speeds of AV, previous AV, and next AV, respectively,

와

는 각각 이전 AV와 다음 AV의 범퍼-대-범퍼 헤드웨이이다.

Wow

are the bumper-to-bumper headways of the previous AV and the next AV, respectively.

The action space represents the behavior of the autonomous agent in the traffic environment provided by the OpenAI gym.

The standard behavior of an automated vehicle is acceleration, and the range of behavior in the action space is from maximum deceleration to maximum acceleration.

The apply_RL_ action function is applied to convert a specific command into an actual action in the SUMO simulator.

First, identify all AVs at unsigned intersections. The action command is then converted to acceleration using the default environment method.

Controllers control the behavior of actors, including human-driven vehicles and AVs. Shared controls allow a single controller to apply to multiple actors. In the present invention, a human-driven vehicle is controlled by the flow framework, and an automated vehicle is controlled by the RLlib library.

The reward function will be described as follows.

To reduce traffic congestion, the average speed of the network should be optimized by reducing latency and queue length. Therefore, average speed becomes a promising metric for training deep RL policies in reality.

The reward function defines how the autonomous agent optimizes the policy.

The goal of the RL agent in the present invention is to achieve high average speed while suppressing vehicle-to-vehicle collisions at unsigned intersections.

In the present invention, the L2 norm is used to estimate a distance of a given amount for a given vehicle speed at a non-signaling intersection based on the target speed (the desired speed of all vehicles at the non-signaling intersection).

In particular, the Get-speed method is applied, which obtains the current speed of all vehicles at a non-signaled intersection and returns the average speed as a compensation.

The compensation function is expressed as in Equation 11.

는 임의의 원하는 속도를 나타내고

는 비신호화된 교차로에서 모든 차량의 속도를 나타낸다.here,

represents any desired speed

represents the speed of all vehicles at unsigned intersections.

A simulation environment setting and results using the apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention will be described as follows.

7 is a block diagram showing the experimental characteristics of a leading autonomous driving vehicle at a non-signaled intersection, and FIG. 8 is a schematic diagram showing an experimental comparison at a non-signaled intersection.

Table 2 shows an example of non-signal intersection simulation settings.

The simulation scenario is as follows.

In the present invention, the vehicle crossing the non-signaled intersection followed the right-of-way rule provided by the SUMO simulator. The purpose of the railroad rules is to enforce traffic rules and prevent traffic collisions.

It also observed the positions of all vehicles and switched the environment from POMDP to MDP. Importantly, autonomous agents learn how to optimize specific rewards for rollouts using the RLlib library. Simulations use RL agents to represent the overall driving flow in human-driven driving and mixed autonomous driving.

The RL agent receives the updated state and fetches the new state in a time step of 0.1 seconds, and in the case of a human driven vehicle, the acceleration behavior is controlled by the IDM model. In addition, continuous routing is applied to keep vehicles within the network.

0.1 second time step, 3.2 m lane width, 2 lanes in each direction, 420 m lane length, maximum acceleration 3 m/s ² , minimum acceleration -3 m/s ² , maximum speed 12 m/s, field of view 600, training process A simulation experiment was performed with 200 iterations.

1000 vehicles per hour in each direction were set, and the range of the non-signaled intersection was between 200m and 220m.

Various scenarios need to be simulated in the field, and our focus is limited to the effect of leading autonomous vehicles at unsigned intersections.

A platoon vehicle approaches a non-signal intersection and drives straight in four different directions. In addition, the results for AV penetration rates of 1% to 100% are presented in units of 10%, and lane changes and left turns are ignored for all vehicles at non-signaled intersections.

Fig. 7 (a) is an experimental environment for a leading autonomous driving vehicle at a non-signaled intersection in a mixed traffic situation with autonomous driving (AV) occupancy in the range of 10% to 90%, and (b) is 100% autonomous driving (AV) ) is the experimental environment of the share.

To demonstrate the superiority of leading autonomous vehicle trials, a leading autonomous vehicle trial was compared with other trials, including the leading human-led vehicle trial and all human-led vehicle trials. 8 shows a comparison of experiments at unsigned intersections.

9 is a graph of an average compensation curve of 200 or more times based on AV occupancy.

Training Policy's Performance is as follows.

RL training outcomes through AV occupancy were used to evaluate learning outcomes. 9 shows an average compensation curve of 200 or more times based on AV occupancy.

The flattening of the curve in all circumstances indicates that education policy is almost converged. Also, average compensation increased as AV occupancy of non-signaling intersections increased except for 50% AV occupancy. Fully autonomous driving outperformed other AV shares, resulting in the highest average compensation and significant curve flattening. In particular, the overall autonomous driving improved 6.8 times compared to the 10% AV share.

Thus, overall autonomous driving outperformed other AV occupancy in all situations, and the effect of leading autonomous vehicle trials at unsigned intersections became more evident as AV occupancy increased.

10 is a graph of spatiotemporal dynamics through AV occupancy at a non-signaled intersection.

The impact of leading autonomous vehicles on smooth driving speeds is as follows:

In FIG. 10, a point is color-coded according to speed, and a point close to the top indicates smooth traffic. In contrast, points close to the floor represent congested traffic.

In the case of low AV occupancy, disturbances caused by static and moving waves of human-driven vehicle motion resulted in reduced speed in unsigned intersection areas (range 200m to 220m). As shown in Fig. 10, almost all points are close to the floor at the non-signaled intersection with low AV occupancy.

This is because human-driven vehicles approach unsigned intersection areas at the same time and slow down according to the railroad rules. At high AV occupancy, the point is closer to the top, and the AV slows down in a shorter time, so there are fewer and fewer stationary and moving waves at unsigned intersections.

Total autonomous driving achieved the highest smooth driving speed among all AV shares. Therefore, the traffic jam was partially resolved, and the traffic flow was smoothed as the AV occupancy increased.

11 is a graph showing characteristics of deriving average speed, average delay time, average fuel consumption, and average exhaust gas value in a SUMO simulation environment.

11 shows the MOE evaluation in terms of emissions according to the average speed, delay time, fuel consumption, and AV occupancy, and the MOE evaluation results indicate that the simulation was more effective as the AV occupancy increased.

Regarding mobility, the average speed gradually increased as AV occupancy increased, and the latency gradually decreased.

As shown in (a) (b) of FIG. 11 , the average speed and delay time were improved by 1.19 times and 1.76 times in the fully autonomous driving compared to the 10% AV occupancy. Energy efficiency, fuel consumption and emissions decreased slightly as AV penetration increased.

As shown in (c)(d) of FIG. 11 , the fully autonomous driving improved fuel consumption by 1.05 times and exhaust gas emission by 1.22 times compared to the 10% AV occupancy.

Therefore, it can be confirmed that leading autonomous vehicles are more effective in terms of mobility and energy efficiency when AV occupancy increases.

The apparatus and method for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles according to the present invention described above is an autonomous vehicle in a mixed traffic flow situation in which clustered autonomous vehicles and human driver vehicles are mixed. It is designed to improve traffic at non-signal intersections and ensure safety through platoon driving learning.

The present invention is a method of learning through information within the range that an autonomous vehicle can observe, such as in a real situation, by applying reinforcement learning and using a Markov decision-making model (Partial Observability MDP, POMDP) to reward reinforcement learning for behavior. so that it can be maximized.

It will be understood that the present invention is implemented in a modified form without departing from the essential characteristics of the present invention as described above.

Therefore, the specified embodiments are to be considered in an illustrative rather than a restrictive view, the scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto are included in the present invention. will have to be interpreted.

100. SUMO Simulation Execution Unit
200. FLOW application part
300. Reinforcement Learning Library Environment Construction Department

Claims (15)

a SUMO simulation execution unit that builds a simulation environment using SUMO (Simulation of Urban Mobility) and delivers the data obtained from the speed, position, and sensor of the autonomous vehicle to the FLOW application unit;
FLOW that builds a simulation environment in the FLOW environment, a reinforcement learning platform that can be linked with SUMO, derives driving behavior without reinforcement learning, and updates vehicle control and simulation status to deliver status and reward information to the reinforcement learning library environment building unit application part;
A reinforcement learning library environment construction unit that optimizes traffic control with multi-agent deep reinforcement learning using FLOW, a reinforcement learning platform that can be linked with SUMO; A device for improvement of traffic based on reinforcement learning at non-signaled intersections. The method of claim 1, wherein the SUMO simulation execution unit,
A SUMO simulation unit that builds a simulation environment using SUMO (Simulation of Urban Mobility) and performs SUMO simulation;
At a non-signal intersection for group operation of autonomous vehicles, characterized in that it includes a result file generator that generates exhaust gas, speed and location value files and transmits the data obtained from the speed, location, and sensors of the autonomous vehicle to the FLOW application unit A device for improving traffic based on reinforcement learning. The method according to claim 1, wherein the SUMO simulation execution unit is configured to:
Presenting the results for AV penetration rate of 1% to 100% in units of 10%, and ignoring lane changes and left turns for all vehicles at non-signaling intersections. A device for improving traffic based on reinforcement learning. According to claim 1, FLOW application unit,
A simulation initialization unit for defining a human driver, setting input values for deep reinforcement learning, and setting a simulation environment including vehicle speed, acceleration, and starting point;
A FLOW environment construction unit that builds a FLOW environment and delivers the state to the reinforcement learning library,
A driving behavior derivation unit that derives driving behavior without applying reinforcement learning;
A vehicle control module that controls the vehicle and transmits the control information to the SUMO simulation execution unit;
Reinforcement learning base at non-signal intersection for autonomous driving vehicle cluster operation, characterized in that it includes an update unit that receives the simulation state from the SUMO simulation execution unit, updates the simulation state, and transmits the state and reward information to the reinforcement learning library environment construction unit Device for improving traffic. The method of claim 1, wherein the reinforcement learning library environment construction unit,
A reinforcement learning library that receives a state from the flow application unit,
a data sampling unit for sampling the data to be learned;
a policy training department that conducts driving behavior (policy) training;
A training result evaluation unit that evaluates the training result and delivers the learned behavior to the FLOW application unit;
Policy optimization to maximize the reward of reinforcement learning for behavior by applying reinforcement learning and the use of Markov decision-making models (Partial Observability MDP, POMDP) as a method of learning through information within the range that autonomous vehicles can observe Apparatus for reinforcement learning-based traffic improvement at non-signal intersections for autonomous vehicle swarm operation, characterized in that it comprises a part. The method of claim 5, wherein the reinforcement learning library environment construction unit,
A policy update storage unit that receives the status of the autonomous vehicle from the FLOW application unit and updates and stores the policy;
An apparatus for improving traffic based on reinforcement learning at non-signal intersections for autonomous vehicle swarm operation, characterized in that it further comprises a learning loop condition determining unit that determines whether the updated policy satisfies the learning loop condition. The method of claim 5, wherein the reinforcement learning library environment construction unit,
By applying the policy gradient methods to compute the estimator of the parameterized policy function using a gradient descent algorithm rather than an action value or state value function for policy optimization,
A device for reinforcement learning-based traffic improvement at non-signal intersections for autonomous vehicle swarm operation, characterized in that it avoids convergence problems that occur in the estimation function due to nonlinear approximation and partial observation. The method of claim 7, wherein the reinforcement learning library environment construction unit,
Apply the multilayer perceptron (MLP) policy to directly optimize the control policy in the simulation of non-signaled intersections,
policy action (

) and the time step t

A policy gradient method based on an estimate of the advantage function in

defined as,
here,

is the expected operator for a finite sample batch,

is a probabilistic policy,

is defined as the sum of the discounted rewards and the baseline estimate,

Wow

A device for reinforcement learning-based travel improvement at non-signaled intersections for autonomous vehicle swarm operation, characterized in that is the behavior and state of time step t. The method of claim 5, wherein the reinforcement learning library environment construction unit,
Apply PPO (Proximal policy optimization) to generate policy updates by adopting a surrogate loss function to prevent performance degradation during the training process,
surrogate object (

)Is

is defined as

is the pre-update policy parameter,

is the policy parameter after update,

A device for reinforcement learning-based traffic improvement at non-signal intersections for group operation of autonomous vehicles, characterized in that is a probability ratio. 10. The method of claim 9, wherein in the case of continuous actions, the policy output of the PPO is a parameter of the Gaussian distribution for each action,
PPO with adaptive KL penalty is used to optimize the KL penalty target using minibatch stochastic gradient descent (SGD),

,

here,

is a weight control coefficient updated after every policy update. An apparatus for reinforcement learning-based traffic improvement at non-signal intersections for group operation of autonomous vehicles. 11. The method of claim 10, wherein it is increased when the current KL difference is greater than the target KL deviation, and decreased when the current KL divergence is less than the target KL divergence;
In the PPO algorithm, first the current policy interacts with the environment to generate the sequence of episodes, and the advantage function is estimated using a baseline estimate of the state value to collect all experiences and run the gradient descent algorithm through the policy network. A device for improving traffic based on reinforcement learning at non-signal intersections for group operation of autonomous vehicles, characterized in that SUMO simulation execution step of constructing a simulation environment using SUMO (Simulation of Urban Mobility) and delivering data obtained from the speed, position, and sensor of the autonomous vehicle to the FLOW application unit;
FLOW that builds a simulation environment in the FLOW environment, a reinforcement learning platform that can be linked with SUMO, derives driving behavior without reinforcement learning, and updates vehicle control and simulation status to deliver status and reward information to the reinforcement learning library environment building unit application step;
Building a reinforcement learning library environment that optimizes traffic control with multi-agent deep reinforcement learning using the reinforcement learning platform FLOW that can be linked with SUMO A method for improving traffic based on reinforcement learning at non-signaling intersections for The method of claim 12, wherein the SUMO simulation execution step comprises:
The SUMO simulation step of constructing a simulation environment using SUMO (Simulation of Urban Mobility) and performing SUMO simulation;
Non-signal intersection for group operation of autonomous vehicles, characterized in that it includes a result file generation step of generating exhaust gas, speed, and position value files and transmitting the data obtained from the speed, position, and sensor of the autonomous vehicle to the FLOW application unit A method for reinforcement learning-based transit improvement in The method of claim 12, wherein the applying FLOW comprises:
A simulation initialization step to define a human driver, set input values for deep reinforcement learning, and set the simulation environment such as vehicle speed, acceleration, and starting point;
A FLOW environment construction step of constructing a FLOW environment and delivering the state to the reinforcement learning library,
A driving behavior derivation step of deriving driving behavior without applying reinforcement learning;
A vehicle control step of controlling the vehicle and transmitting the control information to the SUMO simulation execution unit;
Reinforcement learning at a non-signal intersection for swarm operation of autonomous vehicles, characterized in that it includes an update step of receiving the simulation state from the SUMO simulation execution unit, updating the simulation state, and transmitting the state and reward information to the reinforcement learning library environment construction unit Methods for improving infrastructure traffic. 13. The method of claim 12, wherein the step of constructing a reinforcement learning library environment comprises:
A step in which the reinforcement learning library receives a state from the FLOW application unit,
A data sampling step of sampling the data to be learned;
a policy training stage of driving behavior (policy) training;
A training result evaluation step of evaluating the training result and delivering the learned behavior to the FLOW application unit;
Policy optimization to maximize the reward of reinforcement learning for behavior by applying reinforcement learning and the use of Markov decision-making models (Partial Observability MDP, POMDP) as a method of learning through information within the range that autonomous vehicles can observe step and
A policy update storage step of receiving the status of the autonomous vehicle from the FLOW application unit and updating and storing the policy;
A method for improving traffic based on reinforcement learning at non-signal intersections for autonomous vehicle swarm operation, comprising the step of determining whether the updated policy satisfies the learning loop condition.

Images