KR20220167730A - Method of lane change for autonomous vehicles based deep reinforcement learning, recording medium and device for performing the method - Google Patents

Abstract

A deep reinforcement learning-based lane change method for an autonomous vehicle comprises the steps of: collecting data by partially observing the state of a nearby road by an autonomous vehicle in a multi-lane road environment; performing at least one action of lane change and acceleration control based on the collected data; reflecting the performed action of the autonomous vehicle and deriving a compensation value for the action of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a rear vehicle; learning an action policy for at least one of lane change and acceleration control using a deep reinforcement learning algorithm based on at least one of the partially observed data, the performed action information, and the derived compensation value; and performing optimal driving of the autonomous vehicle for a current road state based on the learned action policy. Accordingly, a safe and efficient lane change of an autonomous vehicle is possible through a deep reinforcement learning algorithm.

Description

Lane change method based on deep reinforcement learning for autonomous vehicles, recording medium and device for performing the same

The present invention relates to a deep reinforcement learning-based lane changing method for an autonomous vehicle, a recording medium and an apparatus for performing the same, and more particularly, an autonomous vehicle is efficient and efficient by using a deep reinforcement learning algorithm. It's about the technology that learns to change lanes safely.

Recently, research related to autonomous vehicles is accelerating, and the possibility of commercialization is being reviewed and institutional development is being made. In order for self-driving vehicles to be commercialized, technology for vehicle stability and efficient driving is required for complex road conditions. In particular, a strategy for frequently changing lanes to reach the driver's goal is an essential technology for improving driving safety and efficiency.

For stable lane change of the existing registered technology, it is divided into stages such as perception of the surrounding environment, safety judgment, and function control. Then, for each step, it is learned through the use of classical control theory or deep learning.

Patent Document 1 of the prior art literature presents a lane change control device and method for an autonomous vehicle, but various situational information to be considered for safety when changing lanes due to an invention related to a lane change control device and method for an autonomous vehicle. is subdivided into groups to perform deep learning.

Patent Document 2 of the prior art literature presents a lane change method, device, and storage medium of an unmanned vehicle, but through selection of candidate lanes and selective performance of candidate lanes, selection of target lanes, safety review, and control of lane change execution. theory, etc.

Accordingly, there is a need for a safe and efficient lane change technology suitable for various situations and behaviors by performing end-to-end learning without segmentation by situation or dynamic behavior.

KR 10-2021-0044960 A KR 10-2020-0116409 A

Erdmann, "SUMO's Lane-Changing Model," Springer, 2015.

Therefore, the technical problem of the present invention is conceived in this respect, and an object of the present invention is to provide a deep reinforcement learning-based lane changing method for an autonomous vehicle.

Another object of the present invention is to provide a recording medium on which a computer program for performing the deep reinforcement learning-based lane changing method for an autonomous vehicle is recorded.

Another object of the present invention is to provide an apparatus for performing the deep reinforcement learning-based lane changing method for the self-driving vehicle.

In the deep reinforcement learning-based lane changing method for an autonomous vehicle according to an embodiment for realizing the object of the present invention, the autonomous vehicle partially observes the state of a nearby road in a multi-lane road environment to obtain data. collecting; performing at least one of lane change and acceleration control based on the collected data; Deriving a compensation value for the behavior of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a vehicle behind the self-driving vehicle by reflecting the performed behavior of the autonomous vehicle; Based on at least one of partially observed data, performed action information, and the derived compensation value, a policy for at least one of lane change and acceleration control is determined using a deep reinforcement learning algorithm. learning step; and performing optimal driving of the self-driving vehicle for current road conditions based on the learned action policy.

In an embodiment of the present invention, the step of partially observing the state of the nearby road and collecting data includes the speed of the leading vehicle in each lane, the speed of the rear vehicle in each lane, the speed of the autonomous vehicle, and the leading vehicle in each lane. It is possible to collect the relative distance between the vehicle and the autonomous vehicle, the relative distance between the vehicle behind each lane and the autonomous vehicle, and the number of the lane where each vehicle is located.

In an embodiment of the present invention, the step of performing at least one of the lane change and acceleration control may include: the acceleration of the self-driving vehicle has a continuous range between -1 and 1; It may have values indicating a lane change to the right lane and a lane change to the left lane.

In an embodiment of the present invention, the compensation function includes a compensation term for allowing the autonomous vehicle to drive close to a target speed and a penalty term for invading a safe distance from a rear vehicle when the autonomous vehicle changes lanes. can include

In an embodiment of the present invention, the penalty clause may be generated based on a minimum allowable distance between vehicles and a minimum allowable time required for the lead vehicle and the rear vehicle to reach the same position.

In an embodiment of the present invention, in the step of learning using the deep reinforcement learning algorithm, a proximal policy optimization (PPO) algorithm may be used during deep reinforcement learning.

In an embodiment of the present invention, the deep reinforcement learning-based lane changing method for an autonomous vehicle may further include updating an action policy based on the derived compensation value for the behavior of the autonomous vehicle.

In an embodiment of the present invention, the multi-lane road environment may be a two-lane circular road.

A computer program for performing a deep reinforcement learning-based lane changing method for the self-driving vehicle is recorded in a computer-readable storage medium according to an embodiment for realizing another object of the present invention.

A deep reinforcement learning-based lane changing device for an autonomous vehicle according to an embodiment for realizing another object of the present invention described above is a multi-lane road environment in which an autonomous vehicle partially observes the state of a nearby road Observation unit for collecting data; Based on the collected data, at least one of the lane change and acceleration control is performed, and a reward value for the performed autonomous vehicle action is derived, and at least one of the partially observed data, the performed action information, and the derived reward value is obtained. An integrated learning unit that learns an action policy for at least one of lane change and acceleration control based on one piece of information using a deep reinforcement learning algorithm; and a policy utilization unit that performs optimal driving of the self-driving vehicle for current road conditions based on the learned behavioral policy.

In an embodiment of the present invention, the observer may include the speed of the leading vehicle in each lane, the speed of the rear vehicle in each lane, the speed of the autonomous vehicle, the relative distance between the leading vehicle in each lane and the autonomous vehicle, and the rear of each lane. The relative distance between the vehicle and the autonomous vehicle and the number of the lane each vehicle is located on can be collected.

In an embodiment of the present invention, an action unit performing at least one of lane change and acceleration control based on the collected data; a compensation unit for deriving a compensation value for the behavior of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a rear vehicle by reflecting the performed behavior of the autonomous vehicle; and a policy learning unit that learns a behavioral policy for at least one of lane change and acceleration control using a deep reinforcement learning algorithm based on at least one of partially observed data, performed behavioral information, and the derived compensation value. can include

In an embodiment of the present invention, the action unit may determine that the acceleration of the self-driving vehicle has a continuous range of -1 to 1, and the lane-changing directions include lane maintenance, lane change to the right lane, and lane change to the left lane, respectively. can have a value that means

In an embodiment of the present invention, the compensation function includes a compensation term for allowing the autonomous vehicle to drive close to a target speed and a penalty term for invading a safe distance from a rear vehicle when the autonomous vehicle changes lanes. can include

In an embodiment of the present invention, the penalty clause may be generated based on a minimum allowable distance between vehicles and a minimum allowable time required for the lead vehicle and the rear vehicle to reach the same position.

In an embodiment of the present invention, the integrated learning unit may use a Proximal Policy Optimization (PPO) algorithm during deep reinforcement learning.

In an embodiment of the present invention, the integrated learning unit may update the action policy based on the derived reward value for the action of the autonomous vehicle.

In an embodiment of the present invention, the multi-lane road environment may be a two-lane circular road.

According to such a deep reinforcement learning-based lane changing method for self-driving vehicles, by using deep reinforcement learning, end-to-end learning is performed without subdividing situational information and behavior into groups, thereby providing safety and safety of autonomous vehicles. Enables efficient lane changes.

As a result of the experiment by applying the present invention, the variance for the target speed was greatly reduced and efficient driving performance was shown. In addition, when changing lanes, it showed the ability to change lanes while maintaining a safe distance from the vehicle behind it.

1 is a block diagram of a deep reinforcement learning-based lane changing device for an autonomous vehicle according to an embodiment of the present invention.
2 is an exemplary diagram of a multi-lane road environment applied to the present invention.
3 is a diagram for explaining the state of a nearby road partially observed by an autonomous vehicle of the present invention.
4 is a diagram for explaining the learning process of the integrated learning unit of the present invention.
5 is a diagram for explaining the safety distance of the punishment term among the reward functions in the present invention.
6 is a graph showing a result of comparing speed change of a lane-changing vehicle over time with the prior art in order to verify the performance of the present invention.
7 is a flowchart of a deep reinforcement learning-based lane changing method for an autonomous vehicle according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The detailed description of the present invention which follows refers to the accompanying drawings which illustrate, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable one skilled in the art to practice the present invention. It should be understood that the various embodiments of the present invention are different from each other but are not necessarily mutually exclusive. For example, specific shapes, structures, and characteristics described herein may be implemented in one embodiment in another embodiment without departing from the spirit and scope of the invention. Additionally, it should be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. Accordingly, the detailed description set forth below is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is limited only by the appended claims, along with all equivalents as claimed by those claims. Like reference numbers in the drawings indicate the same or similar function throughout the various aspects.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the drawings.

1 is a block diagram of a deep reinforcement learning-based lane changing device for an autonomous vehicle according to an embodiment of the present invention.

The deep reinforcement learning-based lane changing device 100 (hereinafter referred to as the device) for an autonomous vehicle according to the present invention proposes a lane changing learning method for an autonomous vehicle using a deep reinforcement learning algorithm.

In the present invention, an entity (autonomous vehicle) learns through interaction with the environment (road conditions). The entity observes road conditions and then performs actions based on the learned information. At this time, since absolute road information cannot be confirmed, partial and incomplete observed information is used.

As a result of performing actions (changing lanes and adjusting acceleration), a new state is obtained and a reward is obtained based on this. The object learns in a direction that maximizes the reward.

Referring to FIG. 1 , an apparatus 100 according to the present invention includes an observation unit 110 , an integrated learning unit 130 and a policy utilization unit 150 . The device 10 may be included in or constitute part of a control module of an autonomous vehicle.

In the apparatus 100 of the present invention, software (application) for performing lane change based on deep reinforcement learning for an autonomous vehicle may be installed and executed, and the observation unit 110, the integrated learning unit 130 and The configuration of the policy utilization unit 150 may be controlled by software for performing lane change based on deep reinforcement learning for the self-driving vehicle executed in the device 100 .

The device 100 may be a separate terminal or a part of a module of the terminal. In addition, the observation unit 110, the integrated learning unit 130, and the policy utilization unit 150 may be formed as an integrated module or may be composed of one or more modules. However, on the contrary, each component may be composed of a separate module.

The device 100 may be mobile or stationary. The apparatus 100 may be in the form of a server or an engine, and may be a device, an apparatus, a terminal, a user equipment (UE), a mobile station (MS), or a wireless device. It can be called by other terms such as wireless device, handheld device, etc.

The device 100 may execute or manufacture various software based on an operating system (OS), that is, a system. The operating system is a system program for enabling software to use the hardware of the device, and is a mobile computer operating system such as Android OS, iOS, Windows mobile OS, Bada OS, Symbian OS, Blackberry OS, and Windows-based, Linux-based, Unix-based, It can include all computer operating systems such as MAC, AIX, and HP-UX.

The observation unit 110 collects data by partially observing the state of a road adjacent to the self-driving vehicle in a multi-lane road environment. For example, the observation unit 110 may collect the speed and lane of the autonomous vehicle, the speed of the leading vehicle, the speed of the rear vehicle, the relative position of the leading vehicle, the relative position of the rear vehicle, and the lane number in which each vehicle is located. can

In one embodiment of the present invention, the multi-lane road environment may be a two-lane circular road (see FIG. 2). Non-autonomous vehicles on the road travel at a slow constant speed, creating an environment in which the autonomous vehicle 10 can reach the target speed only when changing lanes.

은 비 자율주행차량의 집합

와 자율주행차량의 집합

으로 구성한다. 도로에 배치된 전체 차량의 수

대이다. 예를 들어, 차선 번호 k는 가장 바깥쪽 차선이 0번 차선이며 안쪽으로 갈수록 차선의 번호는 증가할 수 있다.collection of vehicles on the road

is a set of non-autonomous vehicles

and a set of autonomous vehicles

composed of Total number of vehicles deployed on the road

it's a stand For example, in the lane number k, the outermost lane is lane 0, and the number of lanes may increase toward the innermost lane.

은 다음과 같은 14차원으로 정의할 수 있다.

Referring to FIG. 3 , in the present invention, the self-driving vehicle e _N 10 can only partially observe the state of a nearby road, not the state information s _t of the entire road. Observation information of autonomous vehicle e _N (10)

can be defined in the following 14 dimensions.

는 각각 0번 차선 선두차량의 속도, 1번 차선 선두차량의 속도, 0번 차선 후방차량 속도, 1번 차선 후방차량의 속도, 자율주행차량의 속도를 의미한다. here,

denotes the speed of the leading vehicle in lane 0, the speed of the leading vehicle in lane 1, the speed of the rear vehicle in lane 0, the speed of the rear vehicle in lane 1, and the speed of the autonomous vehicle, respectively.

는 각각 0번 차선 선두차량, 1번 차선 선두차량, 0번 차선 후방차량, 1번 차선 후방차량과 자율주행차량 사이의 상대 거리를 의미한다.

denotes the relative distance between the leading vehicle in lane 0, the leading vehicle in lane 1, the rear vehicle in lane 0, and the rear vehicle in lane 1 and the autonomous vehicle, respectively.

는 0번 차선 선두차량, 1번 차선 선두차량, 0번 차선 후방차량, 1번 차선 후방차량 그리고 자율주행차량의 시간 t에서의 차선을 의미한다.Finally,

denotes the lane at time t of the leading vehicle in lane 0, the leading vehicle in lane 1, the rear vehicle in lane 0, the rear vehicle in lane 1, and the autonomous vehicle.

As shown in FIG. 4, the integrated learning unit 130 includes a deep neural network 11, performs actions based on data (observation values) collected through sensors 13, etc., and performs actions of the autonomous vehicle. is reflected to derive the reward value for the behavior of the autonomous vehicle.

In addition, the integrated learning unit 130 learns a behavior policy for lane change and acceleration control of the self-driving vehicle using a deep reinforcement learning algorithm.

In one embodiment, a Proximal Policy Optimization (PPO) algorithm may be used among deep reinforcement learning algorithms for learning a behavioral policy.

The integrated learning unit 130 may include an action unit 131, a compensation unit 133, and a policy learning unit 135.

The action unit 131 may perform at least one of lane change and acceleration control based on the collected data.

The compensation unit 133 may derive a compensation value for the behavior of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a rear vehicle by reflecting the behavior of the autonomous vehicle. there is.

The policy learning unit 135 uses a deep reinforcement learning algorithm to determine an action policy for at least one of lane change and acceleration control based on at least one of partially observed data, performed action information, and a derived compensation value. can learn In addition, an action policy may be updated based on the derived compensation value for the action of the autonomous vehicle.

In the present invention, by modeling the Markov Decision Process (MDP) for rapid driving and stability, and learning the self-driving vehicle through the Proximal Policy Optimization (PPO) algorithm, one of the deep reinforcement learning algorithms, to confirm the effect do.

로 정의할 수 있다. Reinforcement learning is one of the machine learning methods in which an object (10, self-driving vehicle), which is the subject of learning, learns through interaction with the environment (state of road (1)). In an embodiment of the present invention, reinforcement learning follows MDP. MDP is a method of probabilistically modeling the decision-making process performed by an entity. It is a set of tuples.

can be defined as

는 개체가 상호작용하는 환경의 시간 t에서의 상태(state) s_t의 집합이다. 관측 공간(observation space)

는 개체가 환경을 관측 정보(observation) o_t의 집합이다. 이때 개체가 관측 가능한 상태 정보의 집합이 상태공간과 동일한 경우 완전 관측(full observation)이라고 하며, 일부로 한정되는 경우를 부분 관측(partial observation)이라고 한다. state space

is the set of states s _t at time t of the environment with which the entity interacts. observation space

is a set of observations o _t of the environment by the entity. At this time, if the set of observable state information of the object is the same as the state space, it is called full observation, and if it is limited to a part, it is called partial observation.

는 개체가 취할 수 있는 모든 행동(action) a_t의 집합이다. 보상함수

(s_t,a_t,s_t+1)(이하

_t로 표기)은 상태 s_t에서 행동 a_t를 취할 때 변한 상태 s_t+1에 대해 환경이 개체에게 주는 보상을 의미한다. 개체는 특정 상태 s_t에서 보상

_t가 최대가 되는 행동 a_t를 취하는 방향으로 학습한다. 마지막으로

는 시간에 따른 감가율(discount factor)을 의미한다.action space

is the set of all actions a _t that the object can take. reward function

(s _t ,a _t ,s _t+1 ) (below

_t ) means the reward given by the environment to the individual for the changed state s _{t +1} when the action a _t is taken in the state s t. An object compensates in a particular state s _t

It learns in the direction of taking the action a _t that maximizes _t . Finally

is the discount factor over time.

In the present invention, a compensation function for enabling an object to efficiently drive through a lane change and at the same time not interfering with the driving of surrounding vehicles is as shown in Equation 1 below.

[Equation 1]

는 보상항으로 자율주행차량이 목표 속도

에 가깝게 주행할 수 있도록 한다. 만약 이 목표 속도

와 동일하다면 최댓값인 1의 보상이 주어지며,

에서 증가하거나 감소하는 경우 그보다 낮은 보상이 주어진다. ??first,

is the compensation term, and the target speed of the self-driving vehicle is

to be able to drive close to If this target speed

If it is equal to , the maximum value of 1 is given,

If it increases or decreases in , a lower reward is given. ??

는 자율주행차량이 차선 변경 했을 때 후방차량의 안전 범위를 침범하는 것에 대한 처벌항이다.

is a penalty clause for invading the safety range of the vehicle behind when the autonomous vehicle changes lanes.

는 시간 t+1에서 후방차량과 자율주행차량 사이의 상대 거리를 의미한다.

는 안전 거리며 이는 환경 설정 및 사용자에 의해 조절될 수 있다. Referring to Figure 5,

Means the relative distance between the rear vehicle and the autonomous vehicle at time t+1.

is the safety distance, which can be adjusted by environment settings and by the user.

The safety distance may vary depending on the safety system used. For example, if it is built into an automatic stability assist system of an autonomous vehicle, the safety distance used in each system can be used.

는 아래의 수학식 2와 같이 설정할 수 있다.In addition, if the design of a non-autonomous vehicle is based on an IDM controller, the safety distance is controlled by the IDM controller.

Can be set as shown in Equation 2 below.

[Equation 2]

는 차량 간 최소 허용 거리이며,

은 time headway로 선두차량과 후방차량이 동일한 위치에 도달하는데 필요한 최소 허용 시간이다.here,

is the minimum allowable distance between vehicles,

is the time headway, which is the minimum permissible time required for the lead vehicle and the rear vehicle to reach the same position.

로 나타낼 수 있다. acc는 자율주행차량의 가속도를 의미하며,

의 연속적인 범위를 갖는다.

는 자율주행차량의 차선 변경 방향을 의미하며,

와 같은 이산적인 값을 갖는다. 예를 들어, 0은 차선을 유지하는 경우, -1은 우측 차선으로의 차선 변경, 1은 좌측 차선으로의 차선 변경을 의미할 수 있다.Actions an object can take

can be expressed as acc means the acceleration of the autonomous vehicle,

has a continuous range of

Means the lane change direction of the autonomous vehicle,

has a discrete value such as For example, 0 may mean maintaining a lane, -1 may mean a lane change to the right lane, and 1 may mean a lane change to the left lane.

The policy utilization unit 150 performs optimal driving of the self-driving vehicle for current road conditions based on the learned deep reinforcement learning.

Hereinafter, the results of evaluating the performance of the present invention using the deep reinforcement learning framework FLOW for road traffic simulators will be described.

대 이다. 여기서, 자율주행차량의 수

대 이며 비 자율주행차량의 수

대 이다. 비 자율주행차량은 모두 IDM 컨트롤러를 사용하며 주행 속도는 1m/s로 고정하였다.The composition of the road is a 260m two-lane circular road (Fig. 2) and the number of vehicles

is a stand Here, the number of autonomous vehicles

and the number of non-autonomous vehicles

is a stand All non-autonomous vehicles use the IDM controller, and the driving speed is fixed at 1 m/s.

, time headway

, 목표 속도

로 설정하였다. 본 시뮬레이션에서 수학식 1의

는 10, 1로 설정하였으며, 1 time step

로 정의하였다.minimum allowable distance

, time headway

, target speed

was set to In this simulation, Equation 1

is set to 10, 1, 1 time step

defined as

To evaluate the performance of the vehicle learned with the deep reinforcement learning algorithm PPO, performance comparison was performed with the case where the control theory-based LC2013 (non-patent document 1 of the prior art document) lane change model was applied.

6, it can be seen that both vehicles try to maintain the target speed of 3 m/s. Table 1 below shows the average speed and speed variance of a single vehicle performing a lane change in a single episode in detail.

[Table 1]

In the case of the LC2013 model, it can be seen that the lane change decision is not immediately made, and the driver delays time by driving behind the non-autonomous lead vehicle at the speed of the lead vehicle. This can be clearly confirmed through the state of maintaining the speed of non-autonomous vehicles at 1 m/s near time steps 1700 and 2500 in FIG. 6 .

On the other hand, in the case of using PPO, it was confirmed that the vehicle was driving while maintaining a constant speed because it changed lanes without waiting meaninglessly when the vehicle in front was blocking the road.

As a result, it was confirmed that more natural lane changes were performed when learning using PPO, a deep reinforcement learning-based model, than when using LC2013, a lane changing model based on control theory.

In conclusion, the self-driving vehicle learned through PPO, a deep reinforcement learning algorithm according to the present invention, showed higher performance compared to non-autonomous vehicles to which the traditional control theory-based lane change model was applied. The average speed of self-driving vehicles increased by about 10% for non-autonomous vehicles, and showed driving ability close to the target speed.

7 is a flowchart of a deep reinforcement learning-based lane changing method for an autonomous vehicle according to an embodiment of the present invention.

The deep reinforcement learning-based lane changing method for an autonomous vehicle according to the present embodiment may be performed in substantially the same configuration as the apparatus 100 of FIG. 1 . Accordingly, components identical to those of the device 100 of FIG. 1 are given the same reference numerals, and repeated descriptions are omitted.

In addition, the method for changing lanes based on deep reinforcement learning for autonomous vehicles according to the present embodiment may be executed by software (application) for performing lane changing based on deep reinforcement learning for autonomous vehicles.

The present invention proposes a PPO-based lane change learning method for an autonomous vehicle as an embodiment of a deep reinforcement learning algorithm. In the present invention, an entity (autonomous vehicle) learns through interaction with the environment (road conditions). The entity observes road conditions and then performs actions based on the learned information. At this time, since absolute road information cannot be confirmed, partial and incomplete observed information is used.

As a result of performing actions (changing lanes and adjusting acceleration), a new state is obtained and a reward is obtained based on this. The object learns in a direction that maximizes the reward.

Referring to FIG. 7 , in the deep reinforcement learning-based lane changing method for an autonomous vehicle according to the present embodiment, the autonomous vehicle collects data by partially observing the state of a nearby road in a multi-lane road environment (step S10).

For example, through partial observation, the speed of the leading vehicle in each lane, the speed of the rear vehicle in each lane, the speed of the autonomous vehicle, the relative distance between the leading vehicle in each lane and the autonomous vehicle, and the vehicle behind the autonomous vehicle in each lane It is possible to collect the relative distance between driving vehicles and the number of the lane where each vehicle is located.

For example, a multi-lane road environment may be a two-lane circular road.

Based on the collected data, at least one of lane change and acceleration control is performed (step S20). Here, the acceleration of the self-driving vehicle has a continuous range between -1 and 1, and the lane change directions may have values indicating lane maintenance, lane change to the right lane, and lane change to the left lane, respectively.

A compensation value for the behavior of the autonomous vehicle is derived from a compensation function based on the target speed of the autonomous vehicle and a safe distance from a vehicle behind by reflecting the behavior of the autonomous vehicle that has been performed (step S30).

The compensation function may include a compensation term for allowing the autonomous vehicle to drive close to a target speed and a penalty term for encroaching on a safe distance from a rear vehicle when the autonomous vehicle changes lanes. In particular, the penalty clause may be generated based on the minimum allowable distance between vehicles and the minimum allowable time required for the lead vehicle and the rear vehicle to reach the same position.

The safety distance may vary depending on the safety system used. For example, if it is built into an automatic stability assist system of an autonomous vehicle, the safety distance used in each system can be used. In addition, if the design of the non-autonomous vehicle is based on the IDM controller, it may be controlled by the IDM controller.

Based on at least one of partially observed data, performed action information, and the derived compensation value, a policy for at least one of lane change and acceleration control is determined using a deep reinforcement learning algorithm. Learn (step S40).

For example, during deep reinforcement learning, a behavioral policy can be learned using a Proximal Policy Optimization (PPO) algorithm, and the behavioral policy can be updated based on the derived reward value for the behavior of the autonomous vehicle. .

If the performance of the policy learned so far exceeds the preset reference value (step S50), the autonomous vehicle performs optimal driving for the current road condition based on the learned action policy (step S60).

On the other hand, if the performance of the policy learned so far does not reach the preset reference value (step S50), return to step S10 and learn again.

Such a deep reinforcement learning-based lane changing method for an autonomous vehicle may be implemented as an application or implemented in the form of program commands that can be executed through various computer components and recorded on a computer-readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination.

Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in the art of computer software.

Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like.

Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. The hardware device may be configured to act as one or more software modules to perform processing according to the present invention and vice versa.

Although the above has been described with reference to embodiments, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. You will understand.

According to the present invention, the safety and efficiency of driving can be improved with respect to lane changes that occur frequently to reach a driver's goal. Therefore, it can be usefully applied to autonomous driving technology that is developing along with the development of current artificial intelligence technology.

10: autonomous vehicle
100: Lane changing device based on deep reinforcement learning for autonomous vehicles
110: observation unit
130: integrated learning unit
131: action department
133: compensation unit
135: policy learning unit
150: policy utilization unit
1: Road
11: Deep Neural Networks
13: sensor

Claims (18)

Collecting data by partially observing the state of a nearby road by an autonomous vehicle in a multi-lane road environment;
performing at least one of lane change and acceleration control based on the collected data;
Deriving a compensation value for the behavior of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a vehicle behind the self-driving vehicle by reflecting the performed behavior of the autonomous vehicle;
Based on at least one of partially observed data, performed action information, and the derived compensation value, a policy for at least one of lane change and acceleration control is determined using a deep reinforcement learning algorithm. learning step; and
A method for changing a lane based on deep reinforcement learning for an autonomous vehicle, comprising: performing an optimal driving of the autonomous vehicle for current road conditions based on the learned action policy.
The method of claim 1, wherein the step of collecting data by partially observing the state of the proximity road,
The speed of the leading vehicle in each lane, the speed of the rear vehicle in each lane, the speed of the autonomous vehicle, the relative distance between the leading vehicle in each lane and the autonomous vehicle, the relative distance between the vehicle behind each lane and the autonomous vehicle, and the A deep reinforcement learning-based lane change method for autonomous vehicles that collects the lane number where the vehicle is located.
The method of claim 1, wherein performing at least one of the lane change and acceleration control comprises:
The acceleration of the self-driving vehicle has a continuous range between -1 and 1, and the lane-changing direction has a value indicating lane maintenance, lane change to the right lane, and lane change to the left lane, respectively. A deep reinforcement learning-based lane changing method for
The method of claim 1, wherein the compensation function,
Deep Reinforcement Learning for Autonomous Vehicles, Including a Compensation Term for Allowing the Autonomous Vehicle to Drive Close to the Target Speed and a Punishment Term for Invading a Safe Distance from a Rear Vehicle When the Autonomous Vehicle Changes Lane based lane change method.
The method of claim 4, wherein the penalty clause,
A lane change method based on deep reinforcement learning for autonomous vehicles, which is generated based on the minimum allowable distance between vehicles and the minimum allowable time required for a leading vehicle and a rear vehicle to reach the same location.
The method of claim 1, wherein the learning using the deep reinforcement learning algorithm comprises:
A lane change method based on deep reinforcement learning for self-driving vehicles using PPO (Proximal Policy Optimization) algorithm during deep reinforcement learning.
According to claim 1,
A method for changing a lane based on deep reinforcement learning for an autonomous vehicle, further comprising: updating a behavioral policy based on the derived compensation value for the behavior of the autonomous vehicle.
According to claim 1,
A multi-lane road environment is a two-lane circular road, a deep reinforcement learning-based lane change method for autonomous vehicles.
A computer-readable storage medium on which a computer program for performing the deep reinforcement learning-based lane changing method for an autonomous vehicle according to claim 1 is recorded.
An observation unit for collecting data by partially observing the state of an autonomous vehicle on a nearby road in a multi-lane road environment;
Based on the collected data, at least one of the lane change and acceleration control is performed, and a reward value for the performed autonomous vehicle action is derived, and at least one of the partially observed data, the performed action information, and the derived reward value is obtained. An integrated learning unit that learns an action policy for at least one of lane change and acceleration control based on one piece of information using a deep reinforcement learning algorithm; and
A deep reinforcement learning-based lane changing device for an autonomous vehicle, comprising: a policy utilization unit that performs optimal driving of the autonomous vehicle for current road conditions based on the learned behavioral policy.
The method of claim 11, wherein the observation unit,
The speed of the leading vehicle in each lane, the speed of the rear vehicle in each lane, the speed of the autonomous vehicle, the relative distance between the leading vehicle in each lane and the autonomous vehicle, the relative distance between the vehicle behind each lane and the autonomous vehicle, and the A deep reinforcement learning-based lane change device for autonomous vehicles that collects the lane number where the vehicle is located.
The method of claim 11, wherein the integrated learning unit,
an action unit that performs at least one of lane change and acceleration control based on the collected data;
a compensation unit for deriving a compensation value for the behavior of the autonomous vehicle from a compensation function based on a target speed of the autonomous vehicle and a safe distance from a rear vehicle by reflecting the performed behavior of the autonomous vehicle; and
and a policy learning unit that learns a behavioral policy for at least one of lane change and acceleration control using a deep reinforcement learning algorithm based on at least one of partially observed data, performed behavioral information, and the derived compensation value. A lane changing device based on deep reinforcement learning for autonomous vehicles.
The method of claim 12, wherein the action unit,
The acceleration of the self-driving vehicle has a continuous range between -1 and 1, and the lane-changing direction has a value indicating lane maintenance, lane change to the right lane, and lane change to the left lane, respectively. A lane-changing device based on deep reinforcement learning for
The method of claim 11, wherein the compensation function,
Deep Reinforcement Learning for Autonomous Vehicles, Including a Compensation Term for Allowing the Autonomous Vehicle to Drive Close to the Target Speed and a Punishment Term for Invading a Safe Distance from a Rear Vehicle When the Autonomous Vehicle Changes Lane based lane change device.
The method of claim 14, wherein the punishment clause,
A lane change device based on deep reinforcement learning for an autonomous vehicle, which is generated based on the minimum allowable distance between vehicles and the minimum allowable time required for a leading vehicle and a rear vehicle to reach the same location.
The method of claim 11, wherein the integrated learning unit,
A deep reinforcement learning-based lane changing device for autonomous vehicles using PPO (Proximal Policy Optimization) algorithm during deep reinforcement learning.
The method of claim 11, wherein the integrated learning unit,
A lane change device based on deep reinforcement learning for an autonomous vehicle that updates an action policy based on a reward value for an autonomous vehicle's behavior.
According to claim 11,
A multi-lane road environment is a two-lane circular road, a deep reinforcement learning-based lane change device for autonomous vehicles.

Images