KR102549744B1 - Method for controliing traffic flow using deep reinforcement learning based autonomous vehicles in road information system, recording medium and device for performing the method - Google Patents

Abstract

A road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle is a road information system using a selected deep reinforcement learning algorithm, and the location and speed of self-driving vehicles and non-autonomous vehicles are converted to status information at predetermined times. receiving the transmission; Selecting an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on the state information; deriving a compensation value for an action of an autonomous vehicle based on a target speed to be reached and the speeds of autonomous vehicles and non-autonomous vehicles; Updating a policy based on the compensation value; and executing the learned self-driving car together with the non-self-driving cars in the road information system based on the learning result and the location and speed status information of the real-time self-driving car and non-self-driving cars. Accordingly, on a road where traffic congestion frequently occurs, road flow can be smoothed based on speed control of an autonomous vehicle.

Description

Road flow control method in road information system using deep reinforcement learning-based autonomous vehicle, recording medium and device for performing this PERFORMING THE METHOD}

The present invention relates to a method for controlling road flow in a road information system using a deep reinforcement learning-based autonomous vehicle, and a recording medium and apparatus for performing the same, and more particularly, to a method for controlling autonomous vehicles at intersections where traffic congestion frequently occurs. It relates to a technology that smoothly controls road flow based on speed control.

Along with the leap in artificial intelligence technology, the field of self-driving technology is attracting attention as one of the interesting topics for both the public and researchers. The Society of Automotive Engineers provides a guide that classifies autonomous driving technology into levels 0-5. It can be divided into a total of six levels, from complete non-autonomous driving (level 0) to driver assistance, partial autonomous driving, conditional autonomous driving, advanced autonomous driving, and complete autonomous driving technology (level 5).

Currently, domestic and foreign companies have successfully developed level 3 self-driving vehicles, and Waymo, GM, and Hyundai Kia are in the process of developing level 4 autonomous vehicles. In addition, Korea announced the world's first level 3 autonomous vehicle safety standards, and from July 2020, the release and sale of level 3 vehicles became possible. According to a report by the Korea Institute of Science and Technology Evaluation and Planning in 2019, from 2020 to 2035, the level 4 self-driving vehicle market is expected to grow at an average annual rate of 84.2%, and in the case of level 3, an average annual growth rate of 33.6%.

Components of autonomous driving technology include cognitive technology such as environment and location recognition, decision technology, and control technology, as well as an interface that provides information to passengers. Fully self-driving vehicles can be commercialized when the three elements are perfectly learned, and these studies are being intensively studied through deep learning, which has played a key role in the development of artificial intelligence.

On the other hand, traffic congestion in downtown areas has become a daily routine as traffic demand and the number of cars continue to increase. According to the Korea Transport Institute, the cost of domestic traffic congestion in 2017 is about 59.6 trillion won, which is a serious environmental and economic problem caused by traffic congestion.

In this situation, the possibility of commercialization of autonomous vehicles is increasing, and autonomous vehicles are expected as a new alternative to solve road traffic problems.

KR 10-2018-0086602 A CN 107886750 A US 9,792,575 B2

Accordingly, the technical problem of the present invention is conceived in this respect, and an object of the present invention is to provide a road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle.

Another object of the present invention is to provide a recording medium on which a computer program for performing the method for controlling road flow in a road information system using the deep reinforcement learning-based autonomous vehicle is recorded.

Another object of the present invention is to provide an apparatus for performing the road flow control method in a road information system using the deep reinforcement learning-based autonomous vehicle.

A road flow control method in a road information system using a deep reinforcement learning-based self-driving vehicle according to an embodiment for realizing the object of the present invention described above is an autonomous vehicle at a predetermined time from the road information system using a selected deep reinforcement learning algorithm. Receiving the location and speed of driving vehicles and non-autonomous vehicles as state information; Selecting an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on the state information; deriving a compensation value for an action of an autonomous vehicle based on a target speed to be reached and the speeds of autonomous vehicles and non-autonomous vehicles; Updating a policy based on the compensation value; and executing the learned self-driving car together with the non-self-driving cars in the road information system based on the learning result and the location and speed status information of the real-time self-driving car and non-self-driving cars.

In an embodiment of the present invention, the step of deriving a compensation value for the action of the autonomous vehicle may derive a compensation value based on the total number of vehicles in the road information system and a target speed to be reached.

In an embodiment of the present invention, the road environment of the road information system may be a road structure including an intersection without traffic lights.

In an embodiment of the present invention, the action of an autonomous vehicle is acceleration, which is a positive continuous value indicating the strength of stepping on the accelerator while driving, and deceleration, which is a negative continuous value indicating the strength of stepping on the brake while driving. ) can be configured.

In an embodiment of the present invention, the selected deep reinforcement learning algorithm may be a Proximal Policy Optimization (PPO) algorithm.

In an embodiment of the present invention, the step of learning the self-driving vehicle may further include limiting a policy update rate to clipping when iteratively learning to optimize the PPO algorithm.

A computer readable storage medium according to an embodiment for realizing another object of the present invention described above includes a computer program for performing a method for controlling road flow in a road information system using the deep reinforcement learning-based autonomous vehicle. It is recorded.

An apparatus for controlling road flow in a road information system using a deep reinforcement learning-based self-driving vehicle according to an embodiment for realizing another object of the present invention described above is provided at a predetermined time from the road information system using a selected deep reinforcement learning algorithm. a road monitoring unit that receives the location and speed of autonomous vehicles and non-autonomous vehicles as state information; an action selection unit that selects an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on the state information; an action compensator for deriving a compensation value for an action of an autonomous vehicle based on a target speed to be reached and the speeds of autonomous vehicles and non-autonomous vehicles; a policy update unit for updating a policy based on a compensation value; And based on the learning result of the action selection unit and the location and speed status information of the real-time self-driving car and non-self-driving cars delivered from the road monitoring unit, the self-driving car learned along with the non-self-driving cars in the road information system It includes; an autonomous driving execution unit that executes.

In an embodiment of the present invention, the action compensation unit may derive a compensation value based on the total number of vehicles in the road information system and a target speed to be reached.

In an embodiment of the present invention, the road environment of the road information system may be a road structure including an intersection without traffic lights.

In an embodiment of the present invention, the action of an autonomous vehicle is acceleration, which is a positive continuous value indicating the strength of stepping on the accelerator while driving, and deceleration, which is a negative continuous value indicating the strength of stepping on the brake while driving. ) can be configured.

In an embodiment of the present invention, when proximal policy optimization (PPO) is selected as a deep reinforcement learning algorithm, a policy update rate may be limited to clipping during iterative learning to be optimized by the PPO algorithm.

According to the road flow control method in the road information system using such a deep reinforcement learning-based autonomous vehicle, all vehicles on the road are induced to drive close to the target speed by using the autonomous vehicle based on the deep reinforcement learning algorithm. By doing so, the road flow can be smoothly induced.

1 is a block diagram of a road flow control device in a road information system using a deep reinforcement learning-based autonomous vehicle according to an embodiment of the present invention.
2 is a diagram showing a road information system having a road structure in which an intersection without a traffic light and a circular road are combined to be learned in the present invention.
3 is a diagram showing a road situation simulation in the case where there is no autonomous vehicle.
4 is a diagram showing a simulation of a road situation when one autonomous vehicle learned according to the present invention is included.
5 is a reward function graph of an autonomous vehicle learning process according to the present invention.
6 is a loss function graph of an autonomous vehicle learning process according to the present invention.
7 is a flowchart of an autonomous driving learning step in a road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle according to an embodiment of the present invention.
8 is a flowchart of an autonomous driving execution step in a road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a road flow control device in a road information system using a deep reinforcement learning-based autonomous vehicle according to an embodiment of the present invention.

The road flow control device (10, hereinafter) in the road information system using a deep reinforcement learning-based self-driving vehicle according to the present invention learns the self-driving vehicle to control the speed according to the road environment based on deep reinforcement learning. After that, it is operated together with non-autonomous vehicles to induce road flow to be improved based on the speed control of autonomous vehicles.

Referring to FIG. 1 , the device 10 according to the present invention includes a road monitoring unit 110, an autonomous driving learning unit 130, and an autonomous driving executing unit 150, and the autonomous driving learning unit 130 comprises It includes an action selection unit 131, an action compensation unit 133 and a policy update unit 135.

The device 10 may be included in or constitute part of a control module of an autonomous vehicle.

In the device 10 of the present invention, software (application) for performing road flow control in a road information system using a deep reinforcement learning-based autonomous vehicle may be installed and executed, and the road monitoring unit 110, the The configuration of the action selection unit 131, the action compensation unit 133, the policy update unit 135, and the autonomous driving execution unit 150 is the deep reinforcement learning-based autonomous vehicle executed in the device 10. It can be controlled by software for performing road flow control in the road information system using

The device 10 may be a separate terminal or a part of a module of the terminal. In addition, the configuration of the road monitoring unit 110, the action selection unit 131, the action compensation unit 133, the policy update unit 135, and the autonomous driving execution unit 150 are formed as an integrated module or , may consist of one or more modules. However, on the contrary, each component may be composed of a separate module.

The device 10 may be mobile or stationary. The apparatus 10 may be in the form of a server or engine, and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS), or a wireless device. It can be called by other terms such as wireless device, handheld device, etc.

The device 10 may execute or manufacture various software based on an operating system (OS), that is, a system. The operating system is a system program for enabling software to use the hardware of the device, and is a mobile computer operating system such as Android OS, iOS, Windows mobile OS, Bada OS, Symbian OS, Blackberry OS, and Windows-based, Linux-based, Unix-based, It can include all computer operating systems such as MAC, AIX, and HP-UX.

The road monitoring unit 110 receives the location and speed of self-driving vehicles and non-autonomous vehicles as status information at predetermined time intervals from the road information system (1, see FIG. 2) using the selected deep reinforcement learning algorithm.

As shown in FIG. 2, the road information system 1 includes a real-time road monitoring module 20, through which road condition information can be collected and transmitted to the road monitoring unit 110 periodically or when necessary. there is.

The self-driving learning unit 110 learns the self-driving vehicle to control speed according to the road environment through the selected deep reinforcement learning algorithm.

Referring back to FIG. 1, the self-driving learning unit 110 includes an action selection unit 131 that selects an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on state information, a target speed to be reached, and autonomy. An action compensation unit 133 that derives a compensation value for an action of an autonomous vehicle based on the speed of a driving vehicle and non-autonomous vehicles and a policy update unit 135 that updates a policy based on the compensation value include

For example, among deep reinforcement learning algorithms, an autonomous vehicle may be trained by using Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed DDPG (TD3). However, the above deep reinforcement learning algorithms are only examples, and other algorithms can be additionally utilized.

Reinforcement learning is a method in which an agent learns by interacting with the environment. When an agent takes an action in a given state, the state changes, and the environment delivers the changed state and reward to the agent. At this time, the agent learns in the direction of maximizing the cumulative value of rewards.

The action selection unit 131 selects an action of the autonomous vehicle, which is the strength of stepping on the accelerator or brake, by using the positions and speeds of autonomous vehicles and non-autonomous vehicles as state information at predetermined time intervals.

In one embodiment of the present invention, the environment may be a road structure including an intersection without traffic lights. 2 is an example of a road structure, which is an 8-shaped structure in which a circular road is combined.

In the present invention, the agent is an autonomous vehicle, receives a plurality of non-autonomous vehicles (eg, 13 vehicles) and its own location and speed every time t seconds, and selects an action.

Actions consist of continuous values, acceleration and deceleration. Acceleration refers to the intensity of pressing the accelerator while driving, and deceleration refers to the intensity of braking while driving.

The action compensation unit 133 derives a compensation value for the action of the autonomous vehicle based on the target speed to be reached and the speed of the autonomous vehicle and non-autonomous vehicles, and the policy update unit 135 calculates the compensation value Update the policy based on

In one embodiment, a reward function for deriving a reward value used in learning is as shown in Equation 1 below.

[Equation 1]

Here, v _d is the target speed to be reached, and can be set to 10 m/s, for example. i is the vehicle ID, and N is the total number of vehicles. The compensation function induces all vehicles on the road to drive close to the target speed.

Hereinafter, a case in which PPO is used as a deep reinforcement learning algorithm to control an action of an autonomous vehicle will be described as an example. The PPO algorithm is an iterative learning method to optimize a surrogate objective function using data sampling and stochastic gradient ascent.

At this time, by limiting the policy update rate to creeping, it is possible to control the policy not to change rapidly. The total loss function of PPO is shown in Equation 2 below.

[Equation 2]

는 정책 손실 함수(policy loss function),

는 가치 손실 함수(Value loss function), 그리고 S는 엔트로피 보너스(entropy bonus)이다. C₁과 C₂는 계수(coefficient)이며, 예를 들어, C₁은 1.0, C₂는 0.0으로 설정할 수 있다.here,

is the policy loss function,

is the value loss function, and S is the entropy bonus. C ₁ and C ₂ are coefficients, and for example, C ₁ can be set to 1.0 and C ₂ can be set to 0.0.

The self-driving execution unit 150 determines the location and speed of real-time self-driving vehicles and non-autonomous vehicles transmitted from the road monitoring unit 110 and the learning result of the action selection unit 131 based on the state information. In the information system 1, the flow of the entire road is controlled by executing the learned self-driving car together with non-self-driving cars.

In one embodiment, FLOW may be used as a simulator for implementing an environment in which an autonomous vehicle drives. FLOW provides an experimental environment that connects the traffic simulator SUMO and the deep reinforcement learning open source library RLlib.

In one embodiment of the present invention, a simulation time step may be set to 0.1 second to implement a simulation environment similar to a real environment. When an episode starts, all vehicles drive while maintaining a constant distance from the front and rear vehicles as shown in FIG. 2 . After that, the action is controlled by the PPO algorithm, and if a collision occurs while driving, the episode ends and the previous episode starts again.

Referring to FIG. 3 , traffic congestion occurred at an intersection when 14 non-autonomous vehicles (white) were driving without an autonomous vehicle. This causes an abrupt stop and reduces the vehicle's average speed.

On the other hand, when one autonomous vehicle and 13 non-autonomous vehicles learned according to the present invention drive, the self-driving vehicle leads the non-autonomous vehicles so that all vehicles on the road can pass through the intersection efficiently. This can be confirmed in the simulation image of FIG. 4, and the autonomous vehicle is black and the non-autonomous vehicle is gray.

An example of a reward graph at this time is shown in FIG. 5 and an example of a total loss graph is shown in FIG. 6 . Referring to FIGS. 5 and 6 , it can be seen that the reward increases and the loss converges as the learning progresses.

The degree of road flow improvement by autonomous vehicles is shown in Table 1 below.

[Table 1]

Referring to Table 1, it can be seen that while the self-driving vehicle is driving, the average speed increases by about 20.7% compared to the case without the self-driving vehicle. It can be seen that the number of sudden stops also significantly decreased from about 50 times to 2 times.

The present invention proposes a road flow improvement method in a road structure in which an intersection without a traffic light and a circular road are combined based on an autonomous vehicle. Among deep reinforcement learning, the self-driving vehicle learns the judgment technology among the technologies necessary to implement the autonomous driving system by using the PPO (Proximal Policy Optimization) algorithm.

As learning progresses, the self-driving vehicle accelerates and decelerates so that all vehicles on the road can pass through the intersection smoothly. Through this, it was confirmed through simulation that the overall traffic flow was improved compared to the case without an autonomous vehicle by appropriately controlling the speed of the autonomous vehicle.

7 is a flowchart of a road flow control method in a road information system using a deep reinforcement learning-based self-driving vehicle according to an embodiment of the present invention.

The method for controlling road flow in a road information system using a deep reinforcement learning-based self-driving vehicle according to the present embodiment may be performed in substantially the same configuration as the device 10 of FIG. 1 and the road information system 1 of FIG. there is. Accordingly, components identical to those of the device 10 of FIG. 1 and the road information system 1 of FIG. 2 are given the same reference numerals, and repeated descriptions are omitted.

In addition, the method for controlling road flow in a road information system using a deep reinforcement learning-based autonomous vehicle according to the present embodiment is a software (application) for performing road flow control in a road information system using a deep reinforcement learning-based autonomous vehicle. ) can be executed.

The present invention trains an autonomous vehicle to control its speed according to the road environment based on deep reinforcement learning, and then operates it together with non-autonomous vehicles to improve the road flow based on the speed control of the autonomous vehicle. .

Referring to FIG. 7 , a method for controlling road flow in a road information system using a deep reinforcement learning-based self-driving car according to the present embodiment is configured to control the speed of the self-driving car according to the road environment through a selected deep reinforcement learning algorithm. learn

In one embodiment, the road environment may have a structure including an intersection without traffic lights, and may be, for example, a figure-eight road structure in which a circular road is combined.

Autonomy, which is the intensity of stepping on the accelerator or brake based on the status information, by receiving the location and speed of self-driving cars and non-autonomous cars as status information at regular intervals from the road information system using the selected deep reinforcement learning algorithm (step S10). An action of the driving vehicle is selected (step S20).

Actions that can be selected by the self-driving vehicle may include acceleration, which is a continuous value indicating the strength of stepping on the accelerator while driving, and deceleration, which is a continuous value indicating the strength of stepping on the brake while driving.

Based on the target speed to be reached and the speed of the autonomous vehicle and non-autonomous vehicles, a compensation value for the action of the autonomous vehicle is derived (step S30), and a policy is updated based on the compensation value (step S30). S40). When the predetermined time elapses (step S50), learning may be terminated.

For example, among deep reinforcement learning algorithms, an autonomous vehicle may be trained by using Proximal Policy Optimization (PPO), Deep Deterministic Policy Gradient (DDPG), and Twin Delayed DDPG (TD3). However, the above deep reinforcement learning algorithms are only examples, and other algorithms can be additionally utilized.

Reinforcement learning is a method in which an agent learns by interacting with the environment. When an agent takes an action in a given state, the state changes, and the environment delivers the changed state and reward to the agent. At this time, the agent learns in the direction of maximizing the cumulative value of rewards.

In the present invention, the agent is an autonomous vehicle, receives a plurality of non-autonomous vehicles (eg, 13 vehicles) and its own location and speed as states at a predetermined time and selects an action.

In one embodiment, the selected deep reinforcement learning algorithm may be a Proximal Policy Optimization (PPO) algorithm. When iteratively learning to be optimized by the PPO algorithm, the policy update rate can be limited to clipping so that the policy does not change rapidly.

Referring to FIG. 8 , a learning pattern is executed by driving autonomous vehicles and non-autonomous vehicles that have completed learning together.

Position and speed state information of autonomous vehicles and non-autonomous vehicles may be received at predetermined times while the vehicle is in operation (step S60). For example, a plurality of non-autonomous vehicles (eg, 13 vehicles) and their own location and speed may be received as states and an action may be selected every time t seconds.

Based on the learning result and the location and speed of real-time self-driving cars and non-autonomous cars, the learned self-driving car is executed along with non-autonomous cars in the road information system, and the intensity of stepping on the accelerator or brake The road flow may be controlled by selecting an action of the autonomous vehicle (step S70).

In one embodiment, the compensation function is as shown in Equation 1, and the compensation function induces all vehicles on the road to drive close to the target speed. As learning proceeds, the reward increases and the loss converges

The present invention uses deep reinforcement learning to improve road flow at intersections where traffic congestion frequently occurs. To this end, after learning to control the speed according to the road environment based on deep reinforcement learning, the self-driving vehicle is operated at an intersection along with other vehicles.

As a result, it was confirmed that the road flow was improved, such that the number of sudden stops of all vehicles on the road was significantly reduced compared to the case without autonomous vehicles, and the average speed of all vehicles increased by about 20%.

Such a method for controlling road flow in a road information system using a deep reinforcement learning-based autonomous vehicle is implemented as an application or implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. It can be. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes such as those produced by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the above has been described with reference to embodiments, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will understand.

The present invention improves the overall traffic flow through appropriate speed control of the learned self-driving vehicle. Therefore, even if fully autonomous vehicles are commercialized with the development of autonomous driving technology in the future, even if a small number of autonomous vehicles are introduced on the road, it is expected that the road flow improvement and economic and environmental effects that can be obtained through this will be very large.

10: Road flow control device in road information system using deep reinforcement learning-based autonomous vehicle
110: road monitoring unit
130: autonomous driving learning unit
131: action selection unit
133: action compensation unit
135: policy update unit
150: autonomous driving execution unit

Claims (12)

Receiving location and speed of self-driving vehicles and non-autonomous vehicles as status information at predetermined time intervals from a road information system using a selected deep reinforcement learning algorithm;
Selecting an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on the state information;
deriving a compensation value for an action of an autonomous vehicle based on a target speed to be reached and the speeds of autonomous vehicles and non-autonomous vehicles;
Updating a policy based on the reward value; and
Executing the learned self-driving car together with non-self-driving cars in the road information system based on the learning result and the location and speed status information of the real-time self-driving car and non-self-driving car; including,
The step of deriving a compensation value for the action of the autonomous vehicle,
Derive a compensation value based on the total number of vehicles in the road information system and the target speed to be reached,
The compensation value is derived through the compensation function of Equation 1 below, a method for controlling road flow in a road information system using a deep reinforcement learning-based autonomous vehicle.
[Equation 1]

Here, v _d is the target speed to be reached, i is the vehicle ID, and N is the total number of vehicles.
delete According to claim 1,
A road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle, in which the road environment of the road information system is a road structure including an intersection without traffic lights.
According to claim 1,
The action of an autonomous vehicle is deep reinforcement, consisting of acceleration, which is a continuous positive value indicating the strength of stepping on the accelerator while driving, and deceleration, a continuous negative value indicating the strength of stepping on the brake while driving. Road flow control method in road information system using learning-based autonomous vehicle.
According to claim 1,
The selected deep reinforcement learning algorithm is a PPO (Proximal Policy Optimization) algorithm, a road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle.
The method of claim 5, wherein the step of learning the self-driving vehicle,
A road flow control method in a road information system using a deep reinforcement learning-based autonomous vehicle, further comprising: limiting a policy update rate to clipping when iteratively learning to be optimized with the PPO algorithm.
A computer-readable storage medium in which a computer program for performing the method of controlling road flow in a road information system using the deep reinforcement learning-based autonomous vehicle according to claim 1 is recorded.
a road monitoring unit that receives location and speed of self-driving cars and non-autonomous cars as status information from a road information system using a selected deep reinforcement learning algorithm;
an action selection unit that selects an action of the self-driving vehicle, which is the strength of stepping on the accelerator or brake, based on the state information;
an action compensator for deriving a compensation value for an action of an autonomous vehicle based on a target speed to be reached and the speeds of autonomous vehicles and non-autonomous vehicles;
a policy update unit for updating a policy based on a compensation value; and
Based on the learning result of the action selection unit and the location and speed status information of the real-time self-driving and non-self-driving cars delivered from the road monitoring unit, the learned self-driving car is executed along with the non-self-driving cars in the road information system. Including; an autonomous driving execution unit that
The action compensation unit,
Derive a compensation value based on the total number of vehicles in the road information system and the target speed to be reached,
The compensation value is derived through the compensation function of Equation 1 below, a road flow control device in a road information system using a deep reinforcement learning-based autonomous vehicle.
[Equation 1]

Here, v _d is the target speed to be reached, i is the vehicle ID, and N is the total number of vehicles.
delete According to claim 8,
A road flow control device in a road information system using a deep reinforcement learning-based autonomous vehicle, in which the road environment of the road information system is a road structure including an intersection without traffic lights.
According to claim 8,
The action of an autonomous vehicle is deep reinforcement, consisting of acceleration, which is a continuous positive value indicating the strength of stepping on the accelerator while driving, and deceleration, a continuous negative value indicating the strength of stepping on the brake while driving. A road flow control device in a road information system using a learning-based autonomous vehicle.
According to claim 8,
When PPO (Proximal Policy Optimization) is selected as the deep reinforcement learning algorithm, roads using deep reinforcement learning-based self-driving cars that limit the policy update rate to clipping when iteratively learning to be optimized by the PPO algorithm Road flow control device in information system.

Images