KR102461362B1 - Control server that generates route guidance data through traffic prediction based on reinforcement learning - Google Patents

Abstract

According to an embodiment of the present invention, a control server receives, in real-time, vehicle location data and travel data from a vehicle module and traffic data for each road section from a traffic sensor. The control server capable of providing a reliable starting point-arrival point movement route by considering a travel time related to uncertain traffic conditions comprises: a database which stores the vehicle location data and travel data, and stores the traffic data as real-time traffic data and past traffic data; a traffic prediction module which generates a first weight through deep learning on the past traffic data and predicts a future speed for each road section by applying the first weight to the real-time traffic data; and a traffic prediction routing module which generates a second weight by applying reinforcement learning to the past traffic data and generates route guidance data by applying the second weight to the vehicle location data, the travel data, and the future speed for each road section.

Description

{CONTROL SERVER THAT GENERATES ROUTE GUIDANCE DATA THROUGH TRAFFIC PREDICTION BASED ON REINFORCEMENT LEARNING}

The present invention relates to a vehicle navigation system, and more particularly, to a control server that generates and provides route guidance data to a vehicle through traffic prediction based on reinforcement learning.

A vehicle navigation system is a route guidance system designed to provide a global route from an origin to a destination. The moving route of the origin-to-destination (hereinafter, OD) is determined by a route planning algorithm built into the vehicle navigation system. In order to provide an optimal OD route that minimizes the travel time from the origin to the destination, many efforts have been made over the past few decades to develop various new routing algorithms. However, some problems still exist regarding uncertainty related to future traffic conditions in non-recurring traffic congestion caused by unforeseen events.

For example, since conventional techniques for a route schedule including predicted traffic information assume that a prediction error of a prediction model is low, a moving route may be greatly affected by prediction accuracy. In addition, although there are many traffic prediction models developed using deep learning technology, it is still difficult to determine the optimal route due to the poor prediction function due to the influence of unexpected events, especially in non-repeating traffic jams.

(1) Korean Patent Publication No. 10-1843683 (Mar. 23, 2018) (2) Korean Patent Publication No. 10-2022-0019204 (2022.02.16)

SUMMARY OF THE INVENTION It is an object of the present invention to provide a vehicle navigation system that provides a reliable source-destination travel route in consideration of travel time related to an uncertain traffic situation.

A control server that receives vehicle location data and travel data from a vehicle module and traffic data for each road section from a traffic sensor in real time from a vehicle module according to an embodiment of the present invention stores the vehicle location data and travel data, and A database that stores data as real-time traffic data and historical traffic data, generates a first weight through deep learning learning for the past traffic data, and applies the first weight to the real-time traffic data to apply the first weight to the future speed for each road section A traffic prediction module to predict Includes a traffic prediction routing module that generates route guidance data.

In this embodiment, the traffic detector corresponds to a hybrid traffic detector comprising a C-ITS detector and an ITS detector.

In this embodiment, the traffic prediction module includes a first batch process of generating the first weight by learning a deep learning model after constructing a data set by preprocessing the past traffic data, and predicting based on the first weight A first real-time process for predicting the future speed for each road section with respect to a prediction horizon, wherein the first real-time process uses a Graph WaveNet as a prediction model.

In this embodiment, the traffic prediction routing module comprises a second batch process for generating the second weight by learning a reinforcement learning model after configuring the past traffic data as a simulation data set; and a second real-time process of generating the route guidance data by applying the future speed for each road section, the vehicle location data, and the travel data to a traffic prediction vehicle routing algorithm based on reinforcement learning.

In this embodiment, the traffic prediction vehicle routing algorithm for generating the route guidance data includes an agent, an action, a state, and a reward function, and the reward function is expected Includes predictive compensation to determine if there is an acceptable gap between travel time and actual travel time.

According to the above-described embodiment of the present invention, it is possible to implement a vehicle navigation system that provides a reliable departure-destination movement route capable of minimizing travel time even in an uncertain traffic situation.

1 is a diagram schematically illustrating a vehicle navigation system according to an embodiment of the present invention.
FIG. 2 is a block diagram schematically illustrating a main configuration and movement of data of the vehicle navigation system of FIG. 1 .
3 is a flowchart schematically illustrating a method of generating path prediction data of the control server shown in FIG. 1 .
4 is a block diagram showing the configuration and operation of a traffic prediction module and a traffic prediction routing module for generating path prediction data in the control server of the present invention.
5 shows the concept of traffic prediction routing in a reinforcement learning-based traffic prediction vehicle routing algorithm (RL-TPVR).
6 is a diagram showing a road network and traffic demand pattern used in simulation for evaluating the performance of the traffic prediction routing algorithm of the present invention.
7 shows examples of repetitive and non-repetitive congestion situations of traffic jams in the road network and traffic demand pattern of FIG. 6 for each scenario.
8 is a table showing details of hyperparameter values used in the traffic prediction model of RL-TPVR according to the present invention.
9 is a table showing hyperparameter values used in the RL-TPVR traffic prediction routing model of the present invention.
10 is a graph showing average compensation measured using a moving average over five consecutive times when training a traffic prediction routing model of RL-TPVR.
11 is a graph showing OD movement time comparison results between RL-TPVR algorithms with and without predictive compensation in various scenario cases.
12 is a graph showing a performance gap between RL-TPVRs having different prediction errors for each scenario.
13 is a diagram showing examples of route guidance of various algorithms for each scenario.
14 is a table showing the mean and standard deviation of OD transit times for each algorithm in various scenarios.
15 is a table showing the p-value for the one-sided Wilcoxon signed rank test for the OD transit time of each scenario.

Hereinafter, some embodiments of the present invention will be described in detail with reference to exemplary drawings. In adding reference numerals to components of each drawing, the same components may have the same reference numerals as much as possible even though they are indicated in different drawings. In addition, in describing the present invention, when it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description may be omitted.

1 is a diagram schematically illustrating a vehicle navigation system according to an embodiment of the present invention. Referring to FIG. 1 , a vehicle navigation system 1000 includes a vehicle module 1100 , a traffic detector 1200 , a roadside base station 1300 , a cloud server 1400 , and a control server 1500 .

The vehicle module 1100 may include a location sensor and a navigation device. The location sensor of the vehicle module 1100 is mounted on a vehicle or an autonomous vehicle to transmit vehicle location information to the roadside base station 1300 or the cloud server 1400 . Here, the vehicle location information represents accurate location information of a service user. The location sensor may be, for example, a Global Positioning System (GPS) sensor. Navigation devices are used to input travel information such as origin, destination, and departure time for autonomous vehicles. Travel information describes the origin, destination, and departure time. After the travel information is processed by the control server 1500, it is generated as route guidance data provided as an embodiment of the present invention. The route guidance data generated by the control server 1500 is provided to the navigation device of the vehicle module 1100 through the cloud server 1400 . The navigation device may use the received route guidance data to provide a departure-destination (OD) movement route of the autonomous vehicle based on route planning.

The traffic detector 1200 detects the traffic condition of the road and transmits it to the roadside base station 1300 . The traffic detector 1200 may include a camera, radar, and lidar sensor installed on a roadside, and sensors for monitoring an abnormal situation such as an accident or illegal parking on the road. In particular, the traffic detector 1200 is, for example, a hybrid intelligent transportation system that combines an existing intelligent transport system (hereinafter referred to as ITS) and a next-generation intelligent transportation system (Cooperative-ITS: hereinafter, C-ITS). can Because traffic data obtained from conventional inductive loop detectors is derived from point measurements such as speed, flow and occupancy, ITS detectors often provide underestimated or overestimated traffic data. Conversely, C-ITS detectors can provide more accurate traffic data using traffic monitoring cameras with extensive detection capabilities based on several advanced computing technologies such as deep learning-based object detection, tracking, and edge computing. However, the next-generation intelligent transportation system (C-ITS) has not yet been deployed in the entire road network. Therefore, the traffic detector 1200 of the present invention can use more stable traffic data by operating based on a hybrid intelligent traffic system combining ITS and C-ITS. In the present invention, more accurate traffic information for each road section can be obtained from section traffic information by using the C-ITS/ITS detector as the traffic detector 1200 . The traffic data is transmitted to the roadside base station 1300 and used in the control server 1500 .

The roadside base station 1300 (Roadside Equipment) is installed on a road on which an autonomous vehicle operates, and collects traffic data transmitted from the vehicle module 1100 and the traffic sensor 1200 . The roadside base station 1300 transmits the collected traffic data to the control server 1500 . The traffic data may include section traffic data indicating a traffic condition for each road section and location information provided by the vehicle module 1100 of the autonomous vehicle. Section traffic data is mainly collected by integrating section traffic information for individual links, and the roadside base station 1300 transmits the collected section traffic information to the control server 1500 . The roadside base station 1300 also transmits vehicle location information to the control server 1500 in the same manner as section traffic information.

The cloud server 1400 receives location information or travel information transmitted from the moving vehicle module 1100 and transmits it to the control server 1500 , or transmits route guidance data transmitted from the control server 1500 to the vehicle module 1100 . provided to Here, the cloud server 1400 may be considered to support a mobile communication service based on Long-Term Evolution (LTE) or a 5G communication system. However, the communication standards supported by the cloud server 1400 are V2X, Vehicle to Everything, 3GPP (3rd Generation Partnership Project), WIMAX (World Interoperability for Microwave Access), Wi-Fi, 2G, 3G, 4G, 5G, 6G, and the like, and the communication standard of the present invention is not limited thereto.

The control server 1500 collects the location information and travel information transmitted from the vehicle module 1100 through the cloud server 1400 or the roadside base station 1300 . In addition, the control server 1500 collects traffic data transmitted in real time from the traffic detector 1200 through the roadside base station 1300 . The control server 1500 generates route guidance data based on the collected travel information, location information, and traffic data of the autonomous vehicle. In addition, the control server 1500 may provide the generated route guidance data to the vehicle module 1100 of the autonomous vehicle via the cloud server 1400 .

The control server 1500 according to an embodiment of the present invention performs three main functions. First, the control server 1500 collects traffic data and vehicle data in real time. Real-time traffic data is not only stored in the historical traffic information database along with the corresponding road section information, but is also used to predict future traffic conditions for each road section. Vehicle data is also stored in the historical traffic database. At the same time, vehicle data is utilized to generate OD travel routes. A second function of the control server 1500 generates route guidance data based on the traffic prediction module and the traffic prediction routing module. The traffic prediction module includes a deep learning model that can be used to predict the future speed of each road segment. Based on historical traffic data, the traffic prediction module trains and validates the deep learning model used to update the reinforcement model associated with the traffic prediction routing module. The third function of the control server 1500 generates route guidance information through a batch process including real-time traffic data and deep learning models, and a traffic prediction module and a traffic prediction routing module. The function of the above control server 1500 will be described in more detail with reference to the drawings to be described later.

The vehicle navigation system 1000 according to the embodiment of the present invention described above uses a traffic prediction module and a traffic prediction routing module based on a deep-learning model to generate route guidance data based on future traffic conditions. can do. Therefore, it is possible to provide a highly reliable route guidance data service by reflecting the future traffic conditions and solving the problem of deterioration of the route prediction function in non-repeating traffic jams or unexpected events resulting from route calculation based on the current traffic conditions.

The road 100 to which the vehicle navigation system 1000 of the present invention is applied may be applied to various roads such as highways, arterial roads, general roads, industrial roads, and ports. The road 100 to which the vehicle navigation system 1000 of the present invention is applied is not limited to the above-described road types and may be applied to various roads in which C-ITS/ITS sensors and roadside base stations are installed.

FIG. 2 is a block diagram schematically illustrating a main configuration and movement of data of the vehicle navigation system of FIG. 1 . Referring to FIG. 2 , the vehicle navigation system 1000 for generating route guidance data by predicting future traffic conditions includes a vehicle module 1100 , a traffic detector 1200 , a roadside base station 1300 , a cloud server 1400 , and It includes a control server 1500 .

The vehicle module 1100 may include a location sensor 1120 and a navigation device 1140 . The position sensor 1120 detects the current position of the autonomous vehicle and generates vehicle position data according to the detection result. The vehicle location data generated by the location sensor 1120 is transmitted to the control server 1500 through the roadside base station 1300 . The location sensor 1120 may be a GPS sensor, but may also be a sensor that receives location information provided by a GLONASS or Galileo system. The navigation device 1140 receives travel information such as a departure point, a destination, and a departure time of the autonomous vehicle and provides it to the control server 1500 as travel data. The travel data will be used in the departure-destination (OD) setting and time setting for generating the route guidance data in the control server 1500 . The navigation device 1140 may set the route of the autonomous vehicle using route guidance data transmitted from the control server 1500 .

The traffic detector 1200 detects the traffic state of the road and transmits it to the roadside base station 1300 as traffic information that can be classified for each road section. The traffic detector 1200 may be implemented as a hybrid intelligent transportation system combining an intelligent transportation system (ITS) and a next-generation intelligent transportation system (C-ITS). Accordingly, the traffic detector 1200 of the present invention can obtain more accurate traffic information for each road section from the section traffic information collected from the sensors. The traffic information is transmitted to the control server 1500 via the roadside base station 1300 .

The roadside base station 1300 transmits vehicle location data provided from the vehicle module 1100 and traffic information provided from the traffic detector 1200 to the control server 1500 .

The control server 1500 may include a traffic prediction module 1520 and a traffic prediction routing module 1540 . The traffic prediction module 1520 and the traffic prediction routing module 1540 generate route guidance data reflecting future traffic conditions by using the travel data and traffic information. The traffic prediction module 1520 configures a data set including average speed data for each road section based on past traffic data. And based on the data set, a weight set is created through learning through a deep-learning model. The traffic prediction routing module 1540 calculates a future speed for each road section based on a weight set generated from past traffic data and real-time traffic data. The traffic prediction routing module 1540 may perform training of the reinforcement learning model based on the simulation data, and generate route guidance data, which is optimal routing information, by taking the trained weight set into consideration of vehicle location data and travel data.

FIG. 3 is a flowchart schematically illustrating a method of generating route guidance data of the control server shown in FIG. 1 . Referring to FIG. 3 , the traffic prediction module 1520 and the traffic prediction routing module 1540 of the control server 1500 include vehicle location data and travel data provided from the vehicle module 1100 , and traffic from the traffic detector 1200 . It predicts future traffic conditions based on the data and generates route guidance data.

In step S110 , the control server 1500 receives and collects real-time traffic data 1010 , vehicle location data 1030 , and travel data 1040 , Trip data.

In step S120, the real-time traffic data 1010 is stored in the database together with each road section information. The real-time traffic data 1010 will be used to predict future traffic conditions for each road section later.

In step S130 , the traffic prediction module 1520 learns and verifies a deep learning model for predicting a future speed for each road section based on past traffic data.

In step S140 , the traffic prediction module 1520 predicts a future speed for each road section or a future traffic state for each road section using the learned and verified weight values of the deep learning model.

In step S150, the traffic prediction routing module 1540 generates simulation data by using the learned and verified weight values of the deep learning model. Based on the simulation data, the traffic prediction routing module 1540 trains the reinforcement learning model, and as a result is updated with the learned weight values.

In step S160 , the traffic prediction routing module 1540 receives and stores vehicle location data and travel data provided in real time.

In step S170, the traffic prediction routing module 1540 generates route prediction data by applying real-time vehicle location data, travel data, and trained weights of the reinforcement learning model.

In step S180 , the traffic prediction routing module 1540 transmits the generated route prediction data to the autonomous vehicle when a navigation service use request occurs.

4 is a block diagram showing the configuration and operation of a traffic prediction module and a traffic prediction routing module for generating path prediction data in the control server of the present invention. Referring to FIG. 4 , the traffic prediction routing module 1540 may predict the future speed for each road section with reference to the future traffic state predicted by the traffic prediction module 1520 .

The traffic prediction module 1520 may learn a deep learning model by using the real-time traffic data 1010 and the past traffic data 1020 , and predict future traffic conditions for each road section from the result. To this end, the traffic prediction module 1520 predicts the future speed of each road section based on a batch process that generates a weight value learned using a deep learning model, and the output of the batch process and real-time traffic data. It includes a real-time process of prediction.

The batch process of the traffic prediction module 1520 generates a set of trained weight values of the deep learning model based on a data set including average speed data of each road section. The batch process of the traffic prediction module 1520 pre-processes the past traffic data 1020 (S210) to configure the data set (S220), learns a deep learning model using the data set, and validates the weights (S230). It can be represented as a step of generating the weighted weight S240. Here, the data set for training the deep learning model is derived from raw historical traffic data aggregated over a unit time interval. The unit time interval may be determined according to the characteristics of a given road network. For example, it may be appropriate to use a shorter time interval for an arterial road and a larger time interval for a highway.

The real-time process of the traffic prediction module 1520 predicts the future speed of each road section with respect to the prediction horizon based on the real-time traffic data 1010 and the pre-learned weight (S250) that is the output of the batch process (S260) ) procedures are included. The predicted horizon may be determined according to the characteristics of each trip. The forecast horizon should be much larger than the estimated travel time of each trip.

Similar to the traffic prediction model 1520 , the traffic prediction routing module 1540 is also configured as a batch process and a real-time process. Simulation data for reinforcement learning is generated using the past traffic data 1020 in the batch process of the traffic prediction routing module 1540 (S310). And it includes the steps of validating the training and weight of the reinforcement learning model using the generated simulation data (S320) (S330). The learned weight ( S340 ) is transmitted to the real-time process ( S350 ), and is combined with the future speed for each road section generated by the traffic prediction module 1520 , vehicle location data 1030 , and travel data 1040 to provide route guidance data is created with Based on the simulation data generation ( S310 ), a set of trained weight values of the reinforcement learning model is generated, which is used as an optimal solution for maximizing the expected cumulative reward of the reinforcement learning model. Through the pre-learned weight (S350) value obtained from the batch process, the real-time process of the traffic prediction routing module 1540 can provide optimal route information for the departure-destination (OD) movement path when there is a request to use the navigation service. have.

Here, the Reinforcement-Learning-Based Traffic-Predictive Vehicle Routing Algorithm (hereinafter, RL-TPVR) performs two functions of traffic prediction and vehicle route generation. These functions of the traffic prediction vehicle routing algorithm (RL-TPVR) are directly related to the traffic prediction module 1520 and the traffic prediction routing module 1540 respectively. The traffic prediction module 1520 requires a prediction model suitable for actual application. Although various deep learning-based prediction models have considered the spatial and temporal relationships of traffic to achieve more appropriate feature extraction, in the present invention, RL-TPVR is one of the most effective methods for predicting the future speed of each road section. , Wu et al., 2019).

One of the most striking features of Graph WaveNet is that inference time is greatly reduced. These features have been demonstrated in other predictive models such as Diffusion Convolutional Recurrent Neural Networks (DCRNNs) (Li et al., 2017) and spatiotemporal graph convolutional networks (STGCNs) (Yu et al., 2017). It can provide multiple predictions in a single run with much shorter compute time compared to Therefore, long-term prediction is required for the entire road network, but this method can provide the predicted speed value for each road section within a short time. Therefore, the Graph WaveNet learning method is used for traffic prediction of the reinforcement learning-based traffic prediction vehicle routing algorithm (RL-TPVR) of the present invention, and can be described as described below.

^N×F'로 나타낼 수 있다. 시간 단계 t에서 교통 패턴을 나타내는 동적 특성 행렬은 'X ^t∈

^N×F'로 표현될 수 있다. 여기서, 'N'은 도로망의 C-ITS/ITS 감지기 수이고, 'F'는 각 검출기의 특징의 수이다. RL-TPVR의 트래픽 예측 모델은 과거 그래프 신호 'H'를 기반으로 예측 지평선 'T'에서 미래 그래프 신호를 예측하는 데 사용되는 매핑 함수 ζ(ㆍ)를 찾는 것을 목표로 한다. 이러한 관계는 아래 수학식 1과 같이 표현될 수 있다.The spatial distribution of C-ITS/ITS detectors in the road network is represented by the graph relationship 'G = (V, E, A )'. Here, 'V' is a C-ITS/ITS detector set, 'E' is an edge set between detectors, and matrix ' A ' is an adjacency matrix indicating proximity between detectors ' A ∈'

It can be expressed as ^N×F '. The dynamic characteristic matrix representing the traffic pattern at time step t is ' X ^t ∈

It can be expressed as ^N×F '. Here, 'N' is the number of C-ITS/ITS detectors in the road network, and 'F' is the number of features of each detector. The traffic prediction model of RL-TPVR aims to find the mapping function ζ(·) used to predict the future graph signal at the prediction horizon 'T' based on the past graph signal 'H'. This relationship can be expressed as Equation 1 below.

^N×F×H'이고, 'X ^(t+1):t+T∈

^N×F×H'이다. where ' X ^(tH):T ∈

^N×F×H ', and ' X ^(t+1):t+T ∈

It is ^N×F×H '.

In order to achieve more effective modeling considering spatiotemporal characteristics, the traffic prediction model of RL-TPVR consists of L spatiotemporal layers. Each layer consists of two types of building blocks: graph and temporal convolution layers. Graph convolutional layer uses DCCNN, while temporal convolutional layer adopts Extended Causal Convolutional Neural Network (DCCNN). R-TPVR also considers the self-adaptive adjacency matrix A _adt , which is one of the major contributions of Graph WaveNet, which is used to capture hidden spatial dependencies through parameters that can be learned without prior knowledge. The adaptive adjacency matrix A _adt is calculated as in Equation 2 below.

, E ₁과 E ₂는 학습 가능한 매개변수가 있는 임베딩 행렬을 나타내고, 'B'는 노드 임베딩의 기능 차원 수이다. 임베딩 방법을 사용하면 그래프 컨볼루션 레이어의 출력은

로 나타낼 수 있고, 아래 수학식 3으로 표현될 수 있다.where E ₁ , E ₂

, E ₁ and E ₂ represent embedding matrices with learnable parameters, and 'B' is the number of functional dimensions of node embeddings. With the embedding method, the output of the graph convolution layer is

It can be expressed as , and can be expressed by Equation 3 below.

)은 그래프 컨볼루션 레이어의 l 번째 출력에서 모델 매개변수를 나타낸다. 순방향 전이 행렬은 P ^l _forward = A/rowsum(A), 역방향 전이 행렬은 P ^l _backward = A ^T/rowsum(A ^T)로 정의된다. Here, ' P ^l _forward ' and ' P ^l _backward ' represent the forward and backward transition matrices of the diffusion process used for the l-th output of the graph convolution layer, and a series of W _l matrices ( W

) represents the model parameters in the l-th output of the graph convolution layer. The forward transition matrix is defined as P ^l _forward = A /rowsum( A ), and the backward transition matrix is defined as P ^l _backward = A ^T /rowsum( A ^T ).

및 필터 f_θ

의 인과관계 컨볼루션 연산(★)은 아래 수학식 4로 표현될 수 있다. In contrast, DCCNN of temporal convolutional layer plays an important role in temporal feature extraction. By extending the receptive field layer-by-layer through extended causal convolution, DCCNN can significantly reduce computation time and consider long-range sequence data non-recursively. input at time step t x

and filter f _θ

The causal convolution operation (★) of can be expressed by Equation 4 below.

Here, 'd' represents a dilation factor required to skip a specific number of input values that increase according to the layer depth.

The traffic prediction model architecture of RL-TPVR, which consists of a graph convolution layer on each building block and two gating mechanism-based temporal convolution layers, is the same as that of Graph WaveNet except for the loss function. In this study, unlike the existing model, RMSE (Root Mean Square Error) is used as a loss function for learning the predictive model of RL-TPVR, and is defined as Equation 5 below.

Here, 'Θ _P ' corresponds to the parameter set of the RL-TPVR traffic prediction model used to represent the mapping function ζ(·). Because RL-TPVR performs link-based traffic prediction, F is equal to '1' because links are grouped into lanes or multi-lanes in road sections between intersections.

5 shows the concept of traffic prediction routing in a reinforcement learning-based traffic prediction vehicle routing algorithm (RL-TPVR). It includes four important elements: Agent, Action, State, and Reward function. The agent refers to an ego vehicle in which a navigation service created from an optimal policy is provided. The self-vehicle receives the route guidance and takes an action to reach the destination in the shortest travel time.

Action represents the path selection corresponding to the routing decision made by the Agent in all links from the source to the destination. An action for a link at time step t may be expressed as a _t ∈A, where 'A' represents an action space. The working space depends on the number of links connected to the current link where the ego vehicle is located. For example, as shown, the working space A={turn right, go straight, turn left} has three links to subsequent options except for the U-turn. RL-TPVR requires the ego to choose a path within a decision area. Establishing a decision zone can provide sufficient time for the self-vehicle to change lanes and follow the route of movement provided by the navigation system. However, the purpose of introducing the decision domain in RL-TPVR is not only to consider the routing problem as a discrete-time stochastic control process, but also to update the agent with the latest traffic information in a timely manner. The determination region d _t at the time step t is expressed by Equation 6 below.

Here, 'L _t ' represents the length of a link where the agent is located at time t, and 'm _t ' represents the minimum distance required to stop safely. The minimum stopping distance 'm _t ' may be calculated by Equation 7 below.

는 시간 단계 t에서 에이전트가 위치한 링크의 자유 흐름 속도 및 예상 평균 속도를 나타내고, 'a_dec'는 자아 차량(Ego vehicle)의 최대 감속 속도를 나타내며, 'τ'는 지각 반응 시간을 나타낸다. where 'V _t,max ' and

denotes the free-flow velocity and the expected average velocity of the link where the agent is located at time step t, 'a _dec ' denotes the maximum deceleration velocity of the ego vehicle, and 'τ' denotes the perceptual reaction time.

State describes the space-time traffic environment based on observation and prediction of the control server 1500 or the C-ITS/ITS center module. The more traffic variables involved in a state, the more accurately it can represent a given traffic situation. However, the state space grows exponentially with the number of traffic variables considered in the state, often resulting in excessive computation time and low convergence. Therefore, in order to deal with the dimensionality problem and the complexity of the dynamic traffic situation, the state of the link at time step t is expressed as Equation (8).

의 예측에 사용된다. 동시에 RL-TPVR은 트래픽 예측 모듈에서 's_t+1'에 사용된 예측 평균 속도 세트를 재귀적으로 로드한다. 상태 정의에 관련된 모든 변수는 RL-TPVR의 트래픽 예측 기능을 사용하여 결정할 수 있다. 이는 MDP 공식의 상태 정의가 RL-TPVR이 로컬 경로 계획뿐만 아니라 글로벌 경로 계획을 통해 생성된 이동 경로를 제공할 수 있도록 함을 시사한다.where 'l _t ' is the estimated position of the agent at time step t and 'P _t ' describes the set of predicted average velocities for subsequent links connected to the link where the agent is located at time step t. For example, in FIG. 5 , since there are three links connected to the current link, P _t =[p _t→R , p _t→S , p _t→L ]. If the agent takes the 'straight' action, 'p _t→S ' is the value of 's _t+1 '.

used in the prediction of At the same time, RL-TPVR recursively loads the predicted average speed set used for 's _t+1 ' in the traffic prediction module. All variables related to state definition can be determined using the traffic prediction function of RL-TPVR. This suggests that the state definition in the MDP formula allows RL-TPVR to provide not only local route planning but also travel routes generated through global route planning.

In RL-TPVR, typical vehicle positions were considered as two-dimensional vectors such as longitude and latitude. In order to reduce the dimension of the state space, the RL-TPVR of the present invention uses the Euclidean distance (ED) between the estimated vehicle position and the destination 'l _s ' as shown in Equation 9 below, and the estimated vehicle position 'l _t ' in the one-dimensional space to specify

Here, 's ₁ ' and 's ₂ ' represent the longitude and latitude of the agent's destination, respectively, and 'l _t,e1 ' and 'l _t,e2 ' are the longitude and latitude of the vehicle location estimated at time step t, respectively. indicates The vehicle position can be accurately estimated using linear integration based on the geometric information of the road, such as the start and end points of a road segment, but can also be calculated using triangular similarity. For example, as shown in FIG. 5 , 'l _t,e1 ' is considered to be d _t away from the center longitude of the road, whereas 'l _t,e2 ' corresponds to the latitude of the center.

The most important element in the traffic prediction routing of RL-TPVR is the reward function. It is directly linked to the goal of the MDP optimization process to determine the optimal policy to maximize the expected cumulative reward. In order to provide a departure-to-destination (OD) route in order to minimize the travel time associated with an uncertain traffic situation, a compensation function 'r _t ' is formulated as Equation (10).

Here, each of the rewards r _t,distance , r _t,time , r _t,prediction , r _t,terminal may be expressed by Equation 11 below.

where the rewards r _t,distance , r _t,time , r _t,prediction , r _t,terminal respectively represent distance, time, prediction and final compensation at time step t; 'clip(·, min, max)' denotes a clipping function adjusted to set limits; 'ED _j→ld ' represents the Euclidean distance (ED) between the destination and the vehicle location determined by Action j; 'TT _t ' is the travel time from the origin to the point s _t ; 'κ' represents the final reward value.

From the point of view of the Markov decision process (MDP), RL-TPVR cannot effectively learn a decision policy without an intrinsic reward because there is a sparse reward in the routing problem to provide the shortest travel time path. Therefore, RL-TPVR includes both an intrinsic reward and an extrinsic reward, where the intrinsic reward is a distance reward (r _t,distance ), a time reward (r _t,time ), and a predictive reward ( r _t,prediction ), and the extrinsic reward points to the final reward (r _t,terminal ).

As expressed in Equation 11, in order to consider the scalability of the distance compensation, the RL-TPVR expresses the distance compensation (r _t,distance ) as a ratio of a value between '0' and '1'. This allows the agent to reach its destination. Similarly, time compensation (r _t,time ) and prediction compensation (r _t,prediction ) are expressed as ratios, but range from '-1' to '1'. Time compensation (r _t,time ) is designed to minimize origin-destination (OD) travel time, whereas prediction compensation (r _t,prediction ) is intended to account for travel time variability.

The biggest feature of RL-TPVR is that it considers the prediction compensation (r _t,prediction ) in the compensation function. This can be an important criterion for determining whether there is an acceptable gap between the expected travel time and the actual travel time, which are affected by the service reliability of navigation from the viewpoint of the mobility service. Therefore, it is expected that RL-TPVR will help the proposed system to provide a robust navigation service by reducing the variability of the OD travel time based on the reward function. Finally, the reward function of RL-TPVR includes an extrinsic reward using the final reward (r _t,terminal ). A large positive reward value is given when the terminal state is on the destination link, while a large negative reward value is given when the terminal state is on another boundary link in the road network.

The traffic prediction routing module of RL-TPVR formalizes the routing problem as MDP, but still requires a reinforcement learning model to obtain an optimal policy. The traffic prediction routing function is implemented as a batch process. However, there is a trade-off between training time and accuracy due to exploration and exploitation problems. Furthermore, the observation space is likely to be much larger than would normally be expected because the present invention addresses the routing problem into the realm of sequential decision-making actions in uncertain traffic conditions. Therefore, it is more appropriate to use the 'Off-policy' reinforcement learning model rather than the 'On-policy' reinforcement learning model. In particular, it is advantageous to consider prioritized experiential play (PER) when a reinforcement learning model uses a play buffer to remove correlations between successive samples. Sample important transitions (s _i , a _i , r _i , s _i+1 ) in the playback buffer. Therefore, in the present invention, the PER algorithm for the reinforcement learning model is adopted in the traffic prediction routing function of RL-TPVR, which is called PDDQN (PER-based Double-deep Q-network). Specific details of the learning method used for RL-TPVR traffic prediction routing are as follows.

PDDQN is an extended version of DDQN (Van Hasselt et al., 2016) to deal with the overestimation problem associated with deep-Q-network (DQN). DDQN decomposes the maximum task in the original DQN's target into task selection and evaluation. The DDQN is updated based on the time difference (TD) error 'δ _i ' as shown in Equation 12 below.

Here, 'γ' represents a discount factor, and Q _θ ^- (s, a) is an action-value that evaluates the quality of an action when a state with a set of weights related to the target neural network 'θ ^- ' is given. ) represents a function. Q _θ (s _i , a _i ) denotes a set of weighted action-value functions of the online neural network θ for pairs of 's _i ' and 'a _i '. Similar to the DQN learning method, the parameters of the target neural network 'θ ^- ' are periodically updated with a copy of the weight parameters included in the online neural network θ. Non-uniform sampling using importance sampling technique to prioritize transitions in the playback buffer based on TD error is further considered in PDDQN as shown in equations 13 and 14.

where U(i) denotes the probability of sampling transition i, 'u _i ^α ' denotes the priority of sampling transition i, 'α' is the priority index, and 'b' is the mini-batch indicates the size of

Here, 'w _i ' denotes an importance-sampling weight, 'B' denotes a buffer size, and 'β' denotes a priority importance-sampling exponent. In the use of Equations 12 to 14, the weight parameter related to the online neural network θ is updated as shown in Equation 15 below.

Here, 'η' represents the step size.

6 is a diagram showing a road network and traffic demand pattern used in simulation for evaluating the performance of the traffic prediction routing algorithm of the present invention. Referring to FIG. 6 , the city network in the form of a 3×3 grid has four lanes on each link, and all are considered to be bidirectional sites. The free flow velocity is 50 km/h on all roads and the link lengths L ₁ and L ₂ are 200 m and 300 m respectively.

Since there is often asymmetric traffic demand on the urban road network during rush hours, the site considers two major and two minor demand streams. The two minor demand flows occur west and south, while the two major demand flows occur east and north, with eastbound traffic being slightly larger than northbound traffic. At individual intersections, the green signal time in each direction with a four-phase signaling scheme is set to 30 seconds, except for the east direction, which is set to 40 seconds. In addition, the simulation experiments allow for simple coordination between multiple signals for east-facing traffic flows by adjusting the signal offsets to avoid queue spillback due to large-scale east-facing traffic. It also generates daily traffic demand in each direction using a set of random variables that follow a normal distribution with different mean and standard deviation values to account for daily changes in traffic demand. Based on the trip generation and distribution, this study uses SUMO's DUArouter tool for dynamic traffic allocation, but there are other options available for choosing a traffic allocation tool from SUMO. Traffic allocation tools can control how routes are selected and routing algorithms, so it may be more appropriate to use heterogeneously loaded traffic from the network to represent commuting traffic demand patterns.

7 shows examples of repetitive and non-repetitive congestion situations of traffic jams in the road network and traffic demand pattern of FIG. 6 for each scenario. Referring to FIG. 7 , since the performance of the RL-TPVR is analyzed in an uncertain traffic situation, several traffic scenarios including repetitive traffic congestion cases and non-recurrent traffic congestion cases can be considered in the simulation experiment.

Scenario 1 shown in (a) describes the normal case of repeated traffic congestion due to large-scale traffic demand, and the other scenarios (b) to (e) are abnormal cases of abnormal traffic congestion caused by vehicles stopped at other designated links. indicates the case. In an abnormal case, a stopped vehicle affects the discharge flow, resulting in a decrease in capacity. As a result, unexpected delays occur when traversing a given link. The stopped vehicle will be at a predetermined link just before the agent vehicle departs from the origin. Agent vehicles are often expected to experience delays when a link is included in the initial global route provided by the navigation system, requiring route decisions that may take detours or additional travel time to traverse congested roads. Otherwise, a reasonable travel time is required on the designated route if the initial route does not include congested roads. Therefore, these traffic scenarios can be used to analyze the performance of the proposed algorithm considering possible changes in traffic conditions in the near future.

8 is a table showing details of hyperparameter values used in the traffic prediction model of RL-TPVR according to the present invention. For training and testing of RL-TPVR using the parameters in Fig. 8, the simulation generated traffic data for 20 days.

For the training set, the data set of the first 16 days was used, and the data set of the second day and the data set of the remaining two days were used as the validation set and the test set, respectively. The simulation runtime was 240 minutes per day, and the time range between 60 and 180 minutes was considered the maximum commuting time. Traffic data for average link speed was collected every 5 minutes, assuming that C-ITS/ITS detectors were extensively deployed in the test. This means that the unit time interval of the traffic prediction module in the proposed system is set to 5 minutes.

Using hyperparameter settings, the deep learning model in the traffic prediction module 1520 of RL-TPVR is trained in a batch process. Thereafter, when there is a request to use the proposed navigation, the traffic prediction module 1520 generates a traffic prediction value using real-time traffic data based on a real-time process. The traffic prediction value is shared with the historical traffic database and the traffic prediction routing module 1540 for state information used for reinforcement learning.

9 is a table showing hyperparameter values used in the RL-TPVR traffic prediction routing model of the present invention. Referring to FIG. 9 , similarly to the deep learning model used in the traffic prediction module 1520 , the traffic prediction routing module 1540 must specify hyperparameter settings for training the reinforcement learning model.

The traffic prediction routing model 1540 is trained with specified hyperparameter settings using traffic information obtained from a historical traffic database. After training the reinforcement learning model of the traffic prediction routing module 1540 as a batch process, a navigation service may be provided as a real-time process.

10 is a graph showing average compensation measured using a moving average over five consecutive times when training a traffic prediction routing model of RL-TPVR. Referring to FIG. 10 , the overall average compensation approaches an optimal value as the exploration rate gradually decreases. Due to the high initial search rate, the average reward fluctuates greatly, reaching 600 times. In addition, it can be observed that the average reward converges when the number of episodes exceeds 600.

The proposed system does not include non-repetitive traffic congestion cases when training the neural network associated with the traffic prediction module 1520 and the traffic prediction routing module 1540 . Non-repetitive traffic congestion due to anomalies is observed only in the test data set. Therefore, the proposed system should be evaluated using previously unseen traffic cases to prove the validity of RL-TPVR.

11 is a graph showing OD movement time comparison results between RL-TPVR algorithms with and without predictive compensation in various scenario cases. Referring to Fig. 11, the OD transit times obtained using RL-TPVR with and without predictive compensation for 100 independent cases of each scenario are shown using blue and orange boxplots, respectively.

In Scenario 1, a recurrent traffic congestion situation, the RL-TPVR with predictive reward exhibited lower median OD travel times with much smaller fluctuations than the RL-TPVR without predictive reward. Other scenarios (Scenario 2 ~ Scenario 5) are similar to Scenario 1, and RL-TPVR with predictive compensation shows better routing performance than RL-TPVR without predictive compensation.

In addition, in Scenario 1, it can be observed that RL-TPVR without predictive compensation shows a large deviation in OD movement time. This trend is also observed in other scenarios. It can be seen that the dispersion of OD travel time slightly increases in some non-repetitive traffic congestion cases such as Scenario 2, Scenario 3, and Scenario 5. Conversely, it can be easily observed that RL-TPVR with predictive compensation shows: Even in non-repetitive congestion situations, the variance is lower than otherwise. This indicates that RL-TPVR with predictive compensation shows stable routing performance regardless of congestion type. These results suggest that the predictive reward of RL-TPVR contributes to robust route guidance by reducing the variability in OD travel time. Therefore, these results can be understood as providing empirical evidence to establish the effect of predictive reward related to RL-TPVR on route guidance.

12 is a graph showing a performance gap between RL-TPVRs having different prediction errors for each scenario. Referring to Figure 12, to further explore the effect of the prediction function on the performance of RL-TPVR, the OD transit time between RL-TPVR with perfect prediction and different prediction errors for 100 independent cases in each scenario. The performance gap is shown.

In the graph shown, the average value and standard deviation value of the performance difference between the RL-TPVR with and without traffic prediction error are indicated by color bars and black error bar graphs, respectively. Traffic prediction errors ranging from 5% to 25% are considered. For example, if the actual average speed of a road section during a specific period is 30 km/h, 28.5 km/h or 31.5 km/h is considered to be the predicted average speed when the prediction error is 5%.

As in Scenario 1, as the traffic prediction error increases, the average value and standard deviation of the performance gap increase. Similarly, in other scenarios, the performance gap decreases as the prediction error decreases. RL-TPVR shows slightly improved routing performance as traffic prediction accuracy increases. Conversely, in the case of non-recurrent traffic congestion, the average value and standard deviation of the performance difference decreased sharply compared to the case of repeated congestion. For example, it can be observed that this performance gap is reduced by more than half in Scenario 4 compared to Scenario 1. This trend shows that the proposed routing algorithm has poor predictive ability.

13 is a diagram showing examples of route guidance of various algorithms for each scenario. Referring to FIG. 13 , several examples of each routing algorithm solution for various scenarios involving the same traffic demand pattern are shown. Here, the source and destination are set to the lower left and upper right links of the network, respectively.

In Scenario 1, which is a case of repeated traffic congestion, it can be observed that the Dijkstra algorithm shows a much longer OD travel time than other algorithms. Compared to the proposed algorithm, the Dijkstra algorithm suffers from a time delay of 30% or more. It can be observed that the agent vehicle spends more time reaching its destination even if it follows one of the shortest travel paths provided by the Dijkstra algorithm, because there is a lot of traffic on the travel path corresponding to the upper left. edge of the network. Conversely, different routing algorithms provide different travel paths. They have a common detour path, such as avoiding the upper left corner, thus significantly reducing the OD travel time compared to the Dijkstra algorithm.

In Scenario 2, which is a non-recurrent traffic congestion case, it can be seen that the OD travel time of the Dijkstra algorithm increases by about 12% compared to the RL-TPVR due to the abnormal traffic conditions of the moving route. Similarly, scenario 3 shows the worst-case performance of the A* algorithm, and the OD travel time of this algorithm increases by about 44% compared to the general case.

In addition, it can be observed that the existing algorithm has poor adaptability considering the impact of non-repetitive traffic congestion by adhering to routing decisions without considering possible changes in future traffic. Scenario 4 shows that when agent vehicles use RL-VR in their navigation system, more than 28% additional travel time is required to get to their destination. Although RL-VR is designed to provide the shortest travel time path through a compensation modeling system, this algorithm suffers from significant time delays due to non-repeating traffic congestion.

The best performance can be obtained from RL-TPVR. Unlike previously developed algorithms, the proposed routing algorithm provides a dynamic movement path in abnormal traffic conditions. As shown in Scenario 4, RL-TPVR can change travel routes to avoid possible traffic congestion and reduce OD travel time by 22.5% compared to RL-VR. This phenomenon can be observed because the predictive function of RL-TPVR can exclude travel routes related to abnormal traffic conditions. This trend is also observed in Scenario 5, where the OD travel time of RL-VR increased by almost 24.81% compared to Scenario 1, while the OD travel time of the proposed algorithm is almost the same as in the case of repeated traffic congestion. In addition, while the previously developed routing algorithm is invariant to traffic congestion caused by abnormal traffic conditions, RL-TPVR develops a flexible and dynamic movement route to alleviate unexpected delays even in non-repetitive traffic congestion situations. to provide. These results suggest that RL-TPVR is the most effective routing algorithm used in navigation systems to mitigate unexpected delays caused by non-repetitive traffic congestion.

14 is a table showing the mean and standard deviation of OD transit times for each algorithm in various scenarios. Referring to FIG. 14 , the routing performance of RL-TPVR and a previously developed routing algorithm are compared for various scenarios related to specific traffic demand patterns generated using random samples of size s=100. In each case, a more comprehensive analysis of the routing algorithm for departure time, origin, destination, and traffic demand, various traffic conditions can be performed.

In Scenario 1, the A ^* algorithm showed the worst performance requiring much more travel time than other routing algorithms. Also, since the standard deviation obtained by the A ^* algorithm is much larger than that of other algorithms, the arrival time may be estimated inaccurately.

Conversely, RL-VR outperforms Dijkstra and A ^* algorithms much better. In addition, the proposed algorithm outperforms the existing routing algorithms. Compared with the Dijkstra, A ^* and RL-VR algorithms, RL-TPVR reduces the OD transit time by approximately 27%, 39% and 9%, respectively. This trend was also observed in non-recurrent traffic congestion cases. For example, RL-TPVR can reduce the average travel time of Scenario 4 by 24.33%, 35.43% and 27.12%, respectively, compared to the Dijkstra, A ^* and RL-VR algorithms.

Importantly, the proposed algorithm was shown to significantly reduce the OD travel time with small deviations in all traffic conditions, indicating that the routing performance can be stabilized even when non-repetitive congestion occurs. This means that RL-TPVR reduces the variability in OD travel time, which is consistent with previous results of performance gap analysis, so that the navigation system can provide reliable route guidance.

According to the statistical results, the best performance can be obtained from RL-TPVR. Nevertheless, it is reasonable to suspect that some extreme cases will invalidate the overall performance of this system and that the performance of the proposed algorithm may be overestimated compared to other existing algorithms. To confirm that RL-TPVR has better routing performance than the existing algorithm under all the same traffic conditions, a one-tailed Wilcoxon signed rank test with significance level of 0.05 is performed. The difference in OD travel time between the previously developed algorithm and RL-TPVR in the same traffic situation i is denoted by δ _i . The median of δ _i for n samples can be expressed as m _δn , where n is equal to 100.

15 is a table showing the p-value for the one-sided Wilcoxon signed rank test for the OD transit time of each scenario. Referring to FIG. 15 , the p-value is well below the significance level of 0.05 in all scenarios. This suggests that there is sufficient evidence to support the alternative hypothesis at a given level of significance. These results suggest that the shortest travel time path can be obtained using RL-TPVR in all scenarios. In other words, these results of this study indicate that the navigation system can obtain greater benefits by utilizing RL-TPVR in both repetitive and non-repetitive traffic congestion situations. Therefore, it can be concluded that the most effective way to reduce the OD travel time associated with uncertain traffic conditions is to use RL-TPVR-equipped navigation.

The contents described above are specific embodiments for carrying out the present invention. The present invention will include not only the above-described embodiments, but also simple design changes or easily changeable embodiments. In addition, the present invention will include techniques that can be easily modified and implemented using the embodiments. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be defined by the claims described below as well as the claims and equivalents of the present invention.

Claims (5)

A control server that receives vehicle location data and travel data from a vehicle module and traffic data for each road section from a traffic sensor in real time, comprising:
a database storing the vehicle location data and the travel data, and storing the traffic data as real-time traffic data and historical traffic data;
a traffic prediction module that generates a first weight through deep learning on the past traffic data and predicts a future speed for each road section by applying the first weight to the real-time traffic data; and
Traffic prediction that generates a second weight by applying reinforcement learning to the past traffic data, and generates route guidance data by applying the second weight to the vehicle location data, the travel data, and the future speed for each road section a routing module;
The traffic prediction routing module performs the reinforcement learning by applying a compensation function including a distance compensation, a time compensation, a prediction compensation, and a final compensation,
The distance compensation is a compensation value of '0' or more and '1' or less according to the Euclidean distance between the destination and the vehicle location, and the time compensation is '-1' or more and '1' depending on the travel time from the origin to the destination. a compensation value below, and the prediction compensation is a compensation value of '-1' or more and '1' or less depending on whether the gap between the expected travel time and the actual travel time is acceptable, and the final reward is that the vehicle location is A control server that is provided with a positive or negative compensation value depending on whether it is located on the destination link of The method of claim 1,
The traffic detector is a control server corresponding to a hybrid traffic detector including a C-ITS detector and an ITS detector. The method of claim 1,
The traffic prediction module includes:
A first batch process for generating the first weight by learning a deep learning model after constructing a data set by pre-processing the past traffic data, and for each road section for a prediction horizon based on the first weight a first real-time process of predicting a future rate;
The first real-time process is a control server using Graph WaveNet as a prediction model. 4. The method of claim 3,
The traffic prediction routing module includes:
a second batch process of generating the second weight by learning a reinforcement learning model after composing the past traffic data into a simulation data set; the future speed for each road section and the vehicle location data based on the second weight; A control server including a second real-time process for generating the route guidance data by applying travel data to a reinforcement learning-based traffic prediction vehicle routing algorithm. 5. The method of claim 4,
The traffic prediction vehicle routing algorithm for generating the route guidance data is a control server that executes the reinforcement learning by applying an agent, an action, a state, and the reward function.

Images