KR102166811B1 - Method and Apparatus for Controlling of Autonomous Vehicle using Deep Reinforcement Learning and Driver Assistance System - Google Patents

Abstract

A method and apparatus for controlling autonomous vehicles using in-depth reinforcement learning and driver assistance systems are presented. A method of controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment includes the steps of receiving measured sensor data and photographed image data through a deep reinforcement learning algorithm; Determining an action for vehicle control using the sensor data and the image data received from the deep reinforcement learning algorithm; And controlling the vehicle by selecting a Driver Assistance Systems (DAS) according to the determined behavior.

Description

Method and Apparatus for Controlling of Autonomous Vehicle using Deep Reinforcement Learning and Driver Assistance System}

The following embodiments relate to a method and apparatus for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system.

Research on the safe driving of autonomous vehicles has developed a lot in recent years. One of the most important topics in safe autonomous driving is finding a driving policy that can provide safety and comfort to drivers. The difficulty with this problem is that the driving policy must be robust to a variety of road conditions and vehicle parameters that can easily change.

Existing autonomous vehicle driving proceeds through the stages of recognition, planning, and control. First, it recognizes information on surrounding vehicles, roads, and environments through sensors, and plans the route the vehicle will travel based on this. Finally, when the final route is determined, vehicle control is performed so that the corresponding route can be followed. At this time, in the case of autonomous vehicle control, the steering wheel, brake, and excel are directly controlled through techniques such as PID, sliding mode control, and model predictive control, which are generally used in control. .

There have also been attempts in the past to control autonomous vehicles through reinforcement learning. In most cases, it was a method of directly controlling the vehicle's steering wheel, brakes, and accelerators using reinforcement learning, and the learning was conducted by deriving the optimal control value according to the goal set by the driver.

However, in the case of the prior art, it is rule-based in which a person directly determines and sets everything in the part of recognition, judgment, and control. In this case, it is possible to respond well and control the situation considered by the designer, but it is difficult to cope with a special situation that is not considered. The driving of an autonomous vehicle is probabilistic and an environment in which various situations can occur. Accordingly, it is very difficult for a person to respond to all possible situations while driving.

Also, there have been attempts to control vehicles using reinforcement learning. These techniques attempted to directly control the vehicle's steering wheel, brakes and excel. In the case of reinforcement learning, there is a possibility of showing unstable movement even when learning is completed. Accordingly, direct control using reinforcement learning can lead to unstable control because a person feels it, and in some cases, dangerous situations can occur.

V. Mnih, K. Kavukcuoglu, D. Silver, AA Rusu, J. Veness, MG Bellemare, A. Graves, M. Riedmiller, AK Fidjeland, G. Ostrovski, et al., "Human-level control through deep reinforcement learning ," Nature, vol. 518, no. 7540, p. 529, 2015. H. Van Hasselt, A. Guez, and D. Silver, "Deep reinforcement learning with double q-learning." in AAAI, vol. 16, 2016, pp. 2094-2100. Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Freitas, "Dueling network architectures for deep reinforcement learning," arXiv preprint arXiv:1511.06581, 2015. D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint arXiv:1412.6980, 2014. M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, et al., "Tensorflow: A system for large-scale machine learning." in OSDI, vol. 16, 2016, pp. 265-283.

The embodiments describe a method and apparatus for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system, and more specifically, in various situations through an algorithm that determines the optimal behavior using deep reinforcement learning. It provides technology to control autonomous vehicles by determining an appropriate driver assistance system.

The embodiments use a system that determines which driver assistance system to select based on an in-depth reinforcement learning algorithm, selects the behavior of the vehicle according to each situation, and controls the vehicle accordingly, thereby planning and controlling the path of the control of the autonomous vehicle. The objective is to provide a method and apparatus for controlling an autonomous vehicle using in-depth reinforcement learning and driver assistance systems that can perform stably.

A method of controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment includes the steps of receiving measured sensor data and photographed image data through a deep reinforcement learning algorithm; Determining an action for vehicle control using the sensor data and the image data received from the deep reinforcement learning algorithm; And controlling the vehicle by selecting a Driver Assistance Systems (DAS) according to the determined behavior.

In the step of receiving the measured sensor data and the photographed image data as an in-depth reinforcement learning algorithm, the sensor data measured through the LIDAR sensor configured in the vehicle and the image data photographed through the camera may be input through the deep reinforcement learning algorithm. have.

The determining of an action for vehicle control may include: purifying the input sensor data and the image data, respectively; Forming connected data by connecting the refined sensor data and the image data; Inputting the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value; And determining an action according to the Q value.

The refining of the sensor data and the image data may include refining the sensor data using Long Short-Term Memory (LSTM); And purifying the image data using a convolutional neural network (CNN).

In the step of calculating the Q value by inputting the fully connected layer, the fully connected layer is divided into two parts of a state value and a behavioral advantage that evaluates an advantage for each behavior, and the state value and the behavioral advantage By integrating the Q value can be obtained.

The step of determining the behavior for vehicle control includes five actions of increasing the target speed, decreasing the target speed, changing the lane to the left, changing the lane to the right, and maintaining the current state so that the vehicle can be driven simultaneously in the longitudinal and transverse directions. It may include.

It may further include the step of learning the deep reinforcement learning algorithm with an optimal driving policy.

In the step of learning the in-depth reinforcement learning algorithm as an optimal driving policy, an optimal driving policy based on the high-speed driving of the vehicle, driving on a non-collision trajectory and unnecessary lane changes by designing a compensation according to the result of the action It is possible to find and learn the deep reinforcement learning algorithm with the driving policy.

An apparatus for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to another embodiment includes: an input unit receiving measured sensor data and photographed image data through a deep reinforcement learning algorithm; A deep reinforcement learning unit for determining an action for vehicle control by using the sensor data and the image data input from the deep reinforcement learning algorithm; And a vehicle controller configured to control a vehicle by selecting a driver assistance system (DAS) according to the determined action.

The input unit may receive the sensor data measured through the LIDAR sensor configured in the vehicle and the image data photographed through the camera as an in-depth reinforcement learning algorithm.

The in-depth reinforcement learning unit may include a sensor data purification unit that purifies the received sensor data; An image data refiner that refines the input image data; A connection data forming unit connecting the refined sensor data and the image data to form connected data; A fully connected layer unit that inputs the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value; And an action determination unit that determines an action according to the Q value.

The sensor data refiner may refine the sensor data using Long Short-Term Memory (LSTM), and the image data refiner may refine the image data using a Convolutional Neural Network (CNN).

The fully connected layer unit uses a network in which the fully connected layer is divided into two parts of a state value and an action advantage that evaluates an advantage for each action, and obtains the Q value by integrating the state value and the action advantage. I can.

The vehicle control unit may include five actions of increasing a target speed, decreasing a target speed, changing a lane to the left, changing a lane to the right, and maintaining a current state so that the vehicle can be driven simultaneously in a longitudinal direction and a transverse direction.

The in-depth reinforcement learning unit designs compensation according to the result of the action to find an optimal driving policy based on high-speed driving of the vehicle, driving on a non-collision trajectory, and excluding unnecessary lane changes, and the in-depth reinforcement learning with the driving policy. You can train the algorithm.

According to embodiments, in-depth reinforcement learning to control an autonomous vehicle by determining an appropriate driver assistance system for various situations through an algorithm that determines the optimal behavior using in-depth reinforcement learning, and a control method of an autonomous vehicle using a driver assistance system And an apparatus.

According to embodiments, by using a system that determines which driver assistance system to select based on an in-depth reinforcement learning algorithm, the vehicle behavior is selected according to each situation and the vehicle is controlled accordingly, thereby planning the path of the control of the autonomous vehicle. And it is possible to provide a control method and apparatus for an autonomous vehicle using in-depth reinforcement learning and a driver assistance system capable of stably performing control.

1 is a diagram illustrating a configuration of a sensor and a camera of a vehicle according to an exemplary embodiment.
2 is a diagram for describing an example of a LIDAR sensor for a nearby vehicle according to an exemplary embodiment.
3 is a diagram for explaining immediate compensation according to overtaking of a slow vehicle according to an exemplary embodiment.
4 is a diagram illustrating an example of a simulator according to an embodiment.
5 is a flowchart showing a method of controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment.
6 is a diagram schematically showing the structure of an autonomous vehicle control apparatus using in-depth reinforcement learning and a driver assistance system according to an embodiment.
7 is a diagram illustrating an example of an average speed through an algorithm for determining a driving policy according to an embodiment.
8 is a diagram illustrating an example of an average number of lane changes through an algorithm for determining a driving policy according to an embodiment.
9 is a diagram illustrating an example of an average number of passing through an algorithm for determining a driving policy according to an embodiment.

Hereinafter, embodiments will be described with reference to the accompanying drawings. However, the described embodiments may be modified in various forms, and the scope of the present invention is not limited by the embodiments described below. In addition, various embodiments are provided to more completely explain the present invention to those of ordinary skill in the art. In the drawings, the shapes and sizes of elements may be exaggerated for clearer explanation.

With the recent commercialization of various Driver Assistance Systems (DAS), most of the vehicles are partially self-driving vehicle functions such as Smart Cruise Control (SCC) and Lane Keeping System (LKS). Came to have. On the other hand, in a limited situation such as a highway, it is possible to drive automatically without driver intervention simply by using a good combination of driver assistance systems (DAS). In order to implement this autonomous driving function, it is necessary to be able to select an appropriate driver assistance system (DAS) function at the appropriate time.

The following embodiments may provide a technique for learning a supervisor to select an appropriate driver assistance system (DAS) through deep reinforcement learning. Driving policy can be operated based on camera images and LIDAR data accessible from autonomous vehicles.

More specifically, embodiments provide an autonomous driving frame using an existing driver assistance system (DAS) and in-depth reinforcement learning. The proposed technology can determine driver assistance system (DAS) functions such as lane keeping, lane change and cruise control that maximize average speed and number of overtakings along with lane changes. The action space is defined according to the behavior level and the driving policy can be learned to determine the level of action such as lane keeping, lane change and cruise control.

Unlike the method of directly learning throttle, brake, and steering values, because the vehicle behaves based on the core functions of the existing commercially available driver assistance systems (DAS), it can guarantee the safety of driving. I can. Driving policies can be designed to operate based on camera images and LIDAR data that are easily accessible from autonomous vehicles. Through Unity, a simulator that implements the movement of roads and multiple vehicles was produced, and learning and verification of the deep reinforcement learning algorithm can be performed using this.

의 튜플(tuple) <S, A, T, R,

>로 정의될 수 있다. MDP에서 풀고자 하는 문제는 의사 결정을 위한 정책(policy)

를 찾는 문제로 주어진 보상 함수 R에 대해 예상 합(expected sum)

를 최대화하는

를 찾는 것이다. Markov Decision Processes (MDP) is a mathematical framework for decision making, a set of states S, a set of actions A, a transition model T, and Reward function R, discount factor

Tuple of <S, A, T, R,

Can be defined as >. The problem MDP is trying to solve is the policy for decision-making

The expected sum for a given reward function R as a problem of finding

To maximize

Is looking for.

를 이용하여 MDP를 풀 수 있게 되었다. 실제로 컴퓨터 비전 분야에서는 손으로 만들어진 특징(handcrafted feature)을 사용하는 것보다 신경망을 통해 보다 나은 표현(representation)을 학습할 수 있다. On the other hand, in recent deep reinforcement learning, it is possible to stably learn a deep neural network (DNN) from a large data set, so that even if the state S _t is not known directly, it is possible to obtain fixed data from raw input. Fixed state representation

It became possible to solve the MDP by using. In fact, in the field of computer vision, better representations can be learned through neural networks than using handcrafted features.

Driving policy learning is conducted by interacting with a host vehicle with the surrounding environment including surrounding vehicles and lanes based on the MDP. Taking advantage of the advantage that deep reinforcement learning can learn the representation itself, observation states S, action space A, and reward function R for driving policy learning can be defined as follows. .

1 is a diagram illustrating a configuration of a sensor and a camera of a vehicle according to an exemplary embodiment.

Referring to FIG. 1, in an embodiment, observation states may be configured using sensor data and camera data of the vehicle 100.

The sensor is a sensor that simulates a LIDAR detecting sideways, and by firing particles at a specific angle, the distance at which the particles collide with an object in the sensor coverage 110 may be returned.

In addition, the camera may be positioned in at least one of the front and rear of the vehicle to provide the front and rear image information 120. There are two network models that process the obtained sensor data and camera data, respectively, and the information refined through the results of each model can be combined and used as observation, an input of deep reinforcement learning.

2 is a diagram for describing an example of a LIDAR sensor for a nearby vehicle according to an exemplary embodiment.

As shown in FIG. 2, for example, each of the left and right sides of the vehicle 200 is sensed in a range of 90 degrees, and a ray 210 may be emitted every 1 degree. The distance from the LIDAR sensor to the obstacle for each degree can be obtained, and a raw image of the frontside can also be obtained to construct the observation.

Since range data from LIDAR and image data from a camera have completely different characteristics, a network having a multi-modal input method can be used.

An action space for driving policy may be defined in a discrete action space. Since these actions are performed based on the conventional driver assistance system (DAS), the action can be performed within the range of preferentially performing the functions of the driver assistance system (DAS).

In the longitudinal case, there are three kinds of behavior. 1) Here, cruise control is performed with speed v + v _cc , where v _cc may be an additional target speed set to 5 km/h. 2) Cruise can be controlled at the current speed v , and 3) Cruise can be controlled at the speed v - v _cc . These longitudinal actions may include Autonomous Emergency Braking (AEB) and Adaptive Cruise Control (ACC). In this way, an action that increases or decreases the target speed is performed through the longitudinal actions, and the vehicle drives according to the target speed, but in a dangerous situation where a collision with the vehicle in front may occur, the vehicle performs deceleration control to prevent a collision. Can be prevented.

In the lateral direction, there are three kinds of actions: 1) Keep Lane, 2) Change Left Lane, and 3) Change Right Lane. In the lateral action, the left and right lane change action is performed, but when the vehicle is within a certain distance to the left of the host vehicle, the Driver Assistance System (DAS) for the lane change operates and the left lane change action is performed. Even if is given, the vehicle does not move to the left. The right side can also operate in the same way as the left side. In addition, when performing an action that maintains the current state, the control to align the center of the lane is performed.

Accordingly, since the autonomous vehicle must drive in the longitudinal and transverse directions at the same time, five actions such as increasing the target speed, decreasing the target speed, changing the left lane, changing the right lane, and maintaining the current state may be included. Driving policy can secure robustness by using the existing driver assistance system (DAS) system, which basically guarantees longitudinal stability.

When you choose an action in relation to reinforcement learning, you are rewarded as a result of the action. As described above, the problem to be solved on the MDP is to find a driving policy that maximizes the expected value of future compensation. In other words, the optimal driving policy may be completely different depending on the design method of the compensation policy. Therefore, it is important to properly design compensation in order to learn the proper driving policy of autonomous vehicles.

When a vehicle is driving in a dense traffic situation, it is desirable to satisfy the following three conditions. The three conditions are: 1) finding a policy to drive the vehicle at high speed, 2) driving on a non-collision trajectory, and 3) not changing lanes too often. We designed a compensation that satisfies these three conditions, and can be expressed as the following equation.

[Equation 1]

[Equation 2]

[Equation 3]

[Equation 4]

[Equation 5]

Here, v is the current speed of the vehicle, v _max is an appropriate maximum speed, and v _min is an appropriate minimum speed. In addition, r _v,max is the compensation for driving at the maximum speed, v _lc is the penalty value when changing lanes, and r _collision is the penalty value received by the agent when colliding, r _overtake is a reward for overtaking another vehicle.

In terms of safety, collision is the most avoidable behavior when driving a vehicle, so r _collision The value can be given a higher value than any other reward value to prevent conflicting behavior first. In addition, it is desirable to travel as fast as possible at the optimum maximum speed v _max or less, so linear compensation can be obtained according to the current speed at less than the appropriate maximum speed. An optimal driving policy can allow the host vehicle to overtake slower driving vehicles. Finally, it is not desirable to do lane-changing behavior too often, so you can set the value of r _lc (or r _lanechange ) to learn how to drive without unnecessary lane changes.

Table 1 can define parameters for compensation.

[Table 1]

3 is a diagram for explaining immediate compensation according to overtaking of a slow vehicle according to an exemplary embodiment.

Referring to FIG. 3, the example of the immediate rewards of overtaking slow vehicles, change the vehicle was traveling at 60km / h lane to pass a slower vehicle, receives when traveling faster after 70km / h r _Here is an example of _tot value. Here, the reward decreases with lane changes, but can increase with increasing speed and overtaking.

Deep reinforcement learning that combines deep neural networks and reinforcement learning has been studied in various ways. In particular, in the case of value-based deep reinforcement learning based on DQN, a large number of studies have been conducted to develop this. Here, among various value-based in-depth reinforcement learning algorithms, DQN (Non-Patent Document 1), Double DQN (Non-Patent Document 2), and Dueling DQN (Non-Patent Document 3) are used to derive and compare results of an autonomous driving agent. Will perform.

기법에 따라 선택되어 수행되며 해당 행동(action)을 수행하고 나면 다음 관측 O_{t +1}과 보상 R_t를 구할 수 있게 된다. 이렇게 얻은 정보들을 하나의 경험(experience)

로 설정하여 데이터셋(dataset)

에 저장하였다가 학습을 할 때에는 데이터셋 안의 데이터를 랜덤 샘플링(random sampling)하여 미니-배지 업데이트(mini-batch update)를 수행하는 경험 리플레이(experience replay) 기법을 이용한다. CNN의 네트워크는 확률적 기울기 하강법(stochastic gradient descent)을 이용하여 손실을 최소화하는 방향으로 신경망의 파라미터들을 학습할 수 있으며, 다음 식과 같이 나타낼 수 있다. DQN (Non-Patent Document 1) is a technique that combines CNN (Convolutional Neural Network) and reinforcement learning.Only the raw pixel frame of the game screen is observed as O _t and substituted into the CNN as input, and the CNN is each action. We derive the Q value for (action) (A _t ). At this time, the action is

It is selected and performed according to the technique, and after performing the action, the next observation O _{t +1} and the reward R _t can be obtained. The information obtained in this way is an experience

Set to dataset

When learning after storing in the data set, an experience replay technique is used that performs a mini-batch update by random sampling the data in the dataset. The CNN network can learn the parameters of the neural network in the direction of minimizing the loss using stochastic gradient descent, which can be expressed as the following equation.

[Equation 6]

는 할인 계수(discount factor)로서 미래에 받을 것으로 예측되는 보상의 합을 네트워크 학습에 얼마의 비율로 반영할지를 결정할 수 있다.

는 네트워크의 파라미터이며,

는 목표 네트워크(network)의 파라미터로 일정 단계(step)마다 네트워크(network)의 파라미터를 그대로 복사해오며 따로 학습은 수행하지 않는 파라미터이다. here,

As a discount factor, it is possible to determine a percentage of the sum of rewards expected to be received in the future to be reflected in network learning.

Is a parameter of the network,

Is a parameter of the target network, the parameter of the network is copied as it is at every certain step, and no additional learning is performed.

In the case of Double DQN (DDQN) (Non-Patent Document 2), the overestimation problem for the action value of the DQN algorithm was solved. In DQN, both of selecting an action and evaluating an action can be performed using the same Q-function when calculating a target value, and can be expressed as the following equation.

[Equation 7]

As a result, an overestimation of the value occurred, which caused the performance of DQN to deteriorate. In DDQN, by separating the process of action selection and evaluation when determining a target value through a method using two Q-functions, the problem of overestimating values can be solved, and can be expressed as the following equation.

[Equation 8]

In the case of Dueling DQN (Non-Patent Document 3), the convolution part of the DQN network is used as it is, and the fully connected layer part is used to provide a state-value and an advantage for each action. A network divided into two parts to evaluate is available. Accordingly, the final formula for calculating the Q-value can be changed into a form that aggregates the state-value and the action advantage, and can be expressed as the following equation.

[Equation 9]

는 이점 함수(advantage function) A를 구하기 위한 완전히 연결된 레이어(fully connected layer)의 파라미터이며,

는 값-함수(value-function) V을 구하기 위한 완전히 연결된 레이어(fully connected layer)의 파라미터이다. 이렇게 2 개의 스트림(stream)으로 구성된 네트워크를 이용한 dueling DQN은 DQN에 비해 월등한 성능향상을 보여주었으며, 특히 행동(action)의 수가 많아져도 성능의 강인함이 확보되는 특징을 보여주었다. here,

Is the parameter of the fully connected layer to obtain the advantage function A,

Is a parameter of a fully connected layer to find the value-function V. Dueling DQN using a network composed of two streams showed a superior performance improvement compared to DQN, and in particular, it showed a characteristic that the robustness of performance was secured even when the number of actions increased.

According to an embodiment, an optimal driving policy may be found in a simulation by combining the above-described deep reinforcement learning algorithms (DQN, DDQN, and Dueling DQN). At this time, the purpose of the deep reinforcement learning algorithm is to drive at high speed without collision with other vehicles and unnecessary lane changes.

4 is a diagram illustrating an example of a simulator according to an embodiment.

Referring to FIG. 4, an example of an autonomous vehicle environment may be configured by using a simulator Unity or Unity ML-Agents to test the embodiments. A vehicle indicated by an agent 400 is a vehicle that is directly controlled, and there are a number of vehicles around it, and lane change, acceleration, deceleration, etc. can be arbitrarily performed. The goal of learning using this simulator is to drive in a highway environment with a large number of vehicles at high speed and minimize unnecessary lane changes. For testing in this environment, among the deep reinforcement learning algorithms, the above-described DQN-based algorithm can be used, and other deep reinforcement learning algorithms as well as DQN can be applied.

The simulated road environment is a highway driving environment consisting of five lanes. Eight types of vehicles, which are visually different from other vehicles in the vicinity, may be randomly generated in the center of a lane within a certain distance of the host vehicle. In addition, a collision avoidance system is strictly applied to each vehicle in the longitudinal direction, so it is possible to drive so as not to cause a collision between nearby vehicles. In addition, the surrounding vehicles randomly performed acceleration, deceleration, lane change to the left, and lane change to the right, respectively, while changing the simulation environment so that the vehicle learning the driving policy can experience various environments.

Since various actions by different vehicles change the simulation environment in various ways, the agent can experience various situations. There are two types of observations in the simulator, one is an image and the other is a LIDAR type range array. Since there is a camera in the front, etc., a raw pixel image can be observed at every step. The LIDAR sensor is capable of sensing a range of 360 degrees with one light beam provided per degree of angle. When the light beam hits an object (eg, an obstacle such as another vehicle), the distance between the host vehicle and the object may be returned. If there are no obstacles, the maximum sensing distance can be returned for every step of the simulator.

5 is a flowchart showing a method of controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment.

5A, a method for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment receives measured sensor data and photographed image data as a deep reinforcement learning algorithm. Step (S110), determining an action for vehicle control using sensor data and image data received from the deep reinforcement learning algorithm (S120), and selecting a Driver Assistance Systems (DAS) according to the determined action Thus, it may be accomplished including the step of controlling the vehicle (S130).

Referring to FIG. 5B, the step of determining an action for vehicle control (S120) includes refining the input sensor data and image data (S121), connecting the refined sensor data and image data to form connected data. Step S122, inputting the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value (S123), and determining an action according to the Q value (S124). I can.

The embodiments provide a method of controlling a vehicle by applying an artificial neural network technique and a reinforcement learning technique. Like artificial intelligence, learning-based methods do not have to be directly set by humans, but perform learning while experiencing various experiences by themselves and find the optimal control method accordingly. Because they learn through various experiences, they can cope with various situations that are difficult for humans to set up, and more optimal behavior can be selected.

In addition, the embodiments do not simply control steering wheel, brake, accelerator, etc. through reinforcement learning, but select a driver assistance system (DAS) that has already been developed to stably control the vehicle. In this way, when the driver assistance system (DAS) is used, the vehicle can be safely controlled without causing discomfort to the driver.

Each step of a method for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment will be described below.

6 is a diagram schematically showing the structure of an autonomous vehicle control apparatus using in-depth reinforcement learning and a driver assistance system according to an embodiment.

Referring to FIG. 6, a structure of an autonomous vehicle control device using in-depth reinforcement learning and a driver assistance system is shown, and the structure of a multi-modal driving policy network can be confirmed. Here, two inputs are required: a raw image of a grayscale camera and LIDAR sensor data having a distance arrangement in meters.

A method of controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment may be described in more detail by using a control device for an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment. . An apparatus for controlling an autonomous vehicle using in-depth reinforcement learning and a driver assistance system according to an embodiment may include an input unit, an in-depth reinforcement learning unit 600, and a vehicle control unit. In particular, the deep reinforcement learning unit 600 includes a sensor data purification unit 610, an image data purification unit 620, a connection data forming unit 630, fully connected layer units 640 and 650, and an action determination unit 660. It may include.

In step S110, the input unit may receive the measured sensor data 602 and the photographed image data 601 as a deep reinforcement learning algorithm. For example, the input unit may receive sensor data 602 measured through a LIDAR sensor configured in a vehicle and image data 601 photographed through a camera as a deep reinforcement learning algorithm.

In step S120, the deep reinforcement learning unit 600 may determine an action 603 for vehicle control using the sensor data 602 and image data 601 received from the deep reinforcement learning algorithm.

More specifically, in step S121, the sensor data 602 and the refiner 610 refine the received sensor data 602, and the image data 601 and the refiner 620 are the input image data 601 Can be purified. At this time, the sensor data 602 refiner 610 may refine the sensor data 602 using Long Short-Term Memory (LSTM), and the image data 601 and the refiner 620 ) Can be purified using a CNN (Convolutional Neural Network).

In step S122, the connection data forming unit 630 may connect the refined sensor data 602 and the image data 601 to form connected data.

In step S123, the fully connected layer units 640 and 650 may input the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value.

In particular, the fully connected layer units 640 and 650 use a network that divides the fully connected layer into two parts: a state value and a behavioral advantage that evaluates the benefits for each action according to the deep reinforcement learning algorithm, and uses a state value and an action. We can combine these to get the Q value.

Meanwhile, the step of learning the deep reinforcement learning algorithm as an optimal driving policy may be further included. The in-depth reinforcement learning unit 600 designs compensation according to the result of the action 603 to find an optimal driving policy based on the high-speed driving of the vehicle, driving on a non-collision trajectory, and excluding unnecessary lane changes, and in-depth driving policy. Reinforcement learning algorithms can be trained.

In step S124, the action determination unit 660 may determine the action 603 according to the Q value.

In step S130, the vehicle controller may control the vehicle by selecting a driver assistance system (DAS) according to the determined action 603. The vehicle controller may include five actions of increasing a target speed, decreasing a target speed, changing a lane to the left, changing a lane to the right, and maintaining a current state so that the vehicle can be driven simultaneously in the longitudinal and transverse directions.

In this way, information necessary for recognition, such as sensor data 602 and camera image, is received as an input of the deep reinforcement learning algorithm, and the deep reinforcement learning algorithm selects the driver assistance system (DAS) that is considered the most appropriate based on the received information. And the vehicle can be controlled accordingly. At this time, the driver assistance system (DAS) is added, so the vehicle can drive safely and stably.

Through driving in various situations, the vehicle is increasingly learning the optimal driving method. According to embodiments, it is applied to an autonomous vehicle, and through in-depth reinforcement learning, it determines which driver assistance system (DAS) is appropriate to use in each situation, and controls the vehicle using a driver assistance system (DAS) built into the vehicle. can do.

According to embodiments, using a system for determining which driver assistance system to select based on an in-depth reinforcement learning algorithm, the vehicle behavior 603 suitable for each situation may be selected and the vehicle may be controlled accordingly. Through this, it is possible to stably perform path planning and control of the control of the autonomous vehicle. Moreover, since the embodiments develop algorithms based on learning, it is possible to appropriately respond to a vehicle driving environment in which humans cannot handle all situations.

Since the proposed deep reinforcement learning algorithm to learn driving policy uses two different inputs at the same time, the convolutional layer of the original DQN structure can be changed to analyze multiple inputs.

In the case of camera images, image data can be refined using a convolutional neural network (CNN) such as DQN, and LIDAR sensor data can be refined using long short-term memory (LSTM). After linking the two refined image data and sensor data, this data can be used as input to a fully connected layer to obtain a Q value that determines the action. In other words, after fusion of the two results, it is possible to finally obtain a Q value and determine an action by passing through a fully connected layer.

At this time, in the case of Dueling DQN, a structure consisting of two streams for obtaining a state-value and an action-advantage for the last fully connected layer may be used.

Table 2 shows the overall structure of the driving policy network and its hyperparameters.

[Table 2]

(Non-Patent Document 4) can be used to educate policy networks. Other hyperparameters are as follows: the learning rate is 0.00025, the batch size is 32, the playback memory size is 100,000, and the number of frames skipped is 4.

The deep reinforcement learning algorithm will be described in more detail below by way of example.

For example, a CNN network that processes image data may be configured in the same structure as a CNN network of DQN. First of all, in the case of image input, you can use 4 stacks of 80x80 grayscale images. In the case of the simulator here, assuming that the cameras exist in the front and rear, a total of 80x80x8 images can be entered as an input of the CNN. The first convolution hidden layer is composed of 32 filters having a size of 8x8, and the stride is 4. The second convolutional hidden layer consists of 64 filters with a size of 4x4, and the stride is 2. The third convolutional hidden layer consists of 64 filters with a size of 3x3, and the stride is 1. ReLU can be used as an activation function in all convolution hidden layers.

In the case of sensor input, it detects a range of 90 degrees for each of the left and right, and if one particle is emitted per degree and collides with an object, the distance to each object is returned, and if there is no collision, the maximum detection distance value can be returned. have. In this way, 180 distance data can be stored for 4 time steps as an image and then applied to an RNN structure using LSTM. The result derived from the last cell among LSTMs is determined as the output of the final sensor data. The number of cell states of the LSTM is 256.

After fusing the image data processing result and the sensor data processing result, a fully connected layer consisting of 512 ReLUs is applied to derive the Q value as much as the number of final actions. For other parameters, the learning rate is 0.00025, the batch size is 32, the size of replay memory is 100,000, the number of frames to be skipped is 4, and (Non-Patent Document 4) can be used for optimization.

7 is a diagram illustrating an example of an average speed through an algorithm for determining a driving policy according to an embodiment. Referring to FIG. 7, it is possible to check the average speed for five times by multiple input, image input, and sensor input.

8 is a diagram illustrating an example of an average number of lane changes through an algorithm for determining a driving policy according to an embodiment. Referring to FIG. 8, an example of the average number of lane changes for five times by multiple input, image input, and sensor input can be confirmed.

9 is a diagram illustrating an example of an average number of passing through an algorithm for determining a driving policy according to an embodiment. Referring to FIG. 9, an example of the average number of passing five times by multiple input, image input, and sensor input can be confirmed.

The graphs of FIGS. 7 to 9 represent the average speed, the number of lane changes, and the number of vehicles overtaking according to the learning step. As learning progresses, the average speed of the vehicle and the number of overtaking vehicles continue to increase, and the number of lane changes decreases. Through this, it can be seen that the lane change is performed only when necessary to meet the target set by the vehicle, and the vehicle operates to drive while maintaining the speed close to the maximum speed.

Meanwhile, the algorithm for determining the proposed driving policy may be implemented using (Non-Patent Document 5). The proposed driving policy learning algorithm was created on python using (Non-Patent Document 5) library. All simulations were conducted using an Nvidia GTX 1080Ti. All algorithms started learning from 250,000, which is thought to have sufficiently accumulated replay memory. The four in-depth reinforcement learning algorithms used to learn driving policies can be confirmed to have improved when compared with the initial values in terms of average speed by learning how to select the existing driver assistance system (DAS) well.

When the learned driving policy is checked in a simulation environment, it can be seen that the vehicle in front drives slowly, and when the lane next to it is empty, it changes lanes and proceeds quickly. The lane change behavior itself can be viewed as simply performing a lane change that maximizes the expected value of future compensation by considering the speed compensation that can be obtained when changing lanes despite receiving negative compensation.

In addition, two policy networks, camera input and LIDAR input, can be additionally implemented to confirm the advantages of a multi-input architecture that combines the functions of camera and LIDAR through CNN and LSTM, respectively.

Below you can compare three different network architectures for different inputs of camera, LIDAR, camera and LIDAR. As learning progresses, autonomous vehicles tend to overtake more vehicles and drive at higher speeds without unnecessary lane changes in all input configurations.

Table 3 shows the comparison of performance according to different inputs of camera, LIDAR, camera and LIDAR.

[Table 3]

As can be seen from the results of Table 3, the multiple input structure using both the camera and LIDAR shows the best performance of 73.54km/h and 42.2, respectively, with the average speed and average passing number. However, when using the architecture for multiple inputs, the number of lane changes is the most and the average value is 30.2.

The purpose of the proposed algorithm is to reduce the number of unnecessary lane changes, but the multi-mode input result is the highest in terms of the number of lane changes. However, in the case of LIDAR's architecture and the camera's architecture, even if the preceding vehicle is slow, there are cases where the vehicle in front is followed without changing lanes. Therefore, there are more lane changes than results of reasonable value to find the optimal policy.

Safe autonomous driving has been actively developed over the past few years. One of the important topics for autonomous driving is finding the best driving policies or supervisors that provide safety and comfort to drivers. At this time, the driving policy must meet robustness regardless of various traffic environments. However, it is difficult for rule-based algorithms to cope with various situations. In order to solve this problem, according to embodiments, it is possible to perform self-learning through various experiences using a deep reinforcement learning algorithm and exhibit excellent performance by using a high-dimensional input such as an original image.

As described above, in the embodiments, not only camera images but also data simulating sensors mainly used in actual autonomous vehicles are input to improve the accuracy of cognitive ability, and autonomous driving based on a driver assistance system (DAS) to secure safety. It is possible to solve the problem of optimizing the behavior of a driving vehicle through in-depth reinforcement learning.

According to embodiments, the autonomous vehicle learned through the deep reinforcement learning algorithm can successfully drive at high speed in a simulated highway scenario without unnecessary lane changes. Since the proposed driving policy network uses multi-modal inputs, the vehicle can drive better than with a single input in terms of average speed, number of lane changes, and number of overtakings.

The embodiments may be used for driving and controlling an autonomous vehicle when driving on a highway. It can also be applied to autonomous drones and robots in that it determines the most appropriate action for the current state and performs control accordingly.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It can be implemented using one or more general purpose computers or special purpose computers, such as a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

Software may contain a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. Can be embodyed in The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Although the embodiments have been described by the limited embodiments and drawings as described above, various modifications and variations can be made from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the claims to be described later.

Claims (15)

Receiving the measured sensor data and the captured image data through a deep reinforcement learning algorithm;
Determining an action for vehicle control using the sensor data and the image data received from the deep reinforcement learning algorithm; And
Controlling a vehicle by selecting a Driver Assistance Systems (DAS) according to the determined behavior
Including,
The step of determining the action for vehicle control,
Refining the input sensor data and the image data, respectively;
Forming connected data by connecting the refined sensor data and the image data;
Inputting the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value; And
Determining an action according to the Q value
Including,
Purifying the sensor data and the image data, respectively,
Purifying the sensor data using Long Short-Term Memory (LSTM); And
Purifying the image data using a convolutional neural network (CNN)
Including,
The deep reinforcement learning algorithm analyzes multiple inputs to use two different inputs at the same time to learn the driving policy, and after connecting the two refined sensor data and the image data, a fully connected layer ) To finally get the Q value and determine the action
A method of controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 1,
The step of receiving the measured sensor data and the photographed image data as a deep reinforcement learning algorithm,
Receiving the sensor data measured through the LIDAR sensor configured in the vehicle and the image data photographed through the camera as an in-depth reinforcement learning algorithm
A method of controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system, characterized in that. delete delete The method of claim 1,
The step of obtaining a Q value by inputting into the fully connected layer,
Using a network that divides the fully connected layer into two parts: a state value and a behavioral advantage that evaluates the benefits for each behavior, and obtains the Q value by integrating the state value and the behavioral advantage.
A method of controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 1,
The step of determining the action for vehicle control,
Including five actions: increasing the target speed, decreasing the target speed, changing lanes to the left, changing lanes to the right, and maintaining the current state so that the vehicle can drive in the longitudinal and transverse directions simultaneously.
A method of controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 1,
Learning the deep reinforcement learning algorithm as an optimal driving policy
A method for controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system further comprising a. The method of claim 7,
The step of learning the deep reinforcement learning algorithm as an optimal driving policy,
Designing a reward according to the result of the action to find an optimal driving policy based on high speed driving of the vehicle, driving on a non-collision trajectory, and excluding unnecessary lane changes, and learning the deep reinforcement learning algorithm with the driving policy
A method of controlling an autonomous vehicle using an in-depth reinforcement learning and driver assistance system, characterized in that. An input unit that receives measured sensor data and photographed image data through a deep reinforcement learning algorithm;
A deep reinforcement learning unit for determining an action for vehicle control by using the sensor data and the image data input from the deep reinforcement learning algorithm; And
Vehicle control unit that controls the vehicle by selecting a driver assistance system (DAS) according to the determined behavior
Including,
The deep reinforcement learning unit,
A sensor data refiner that refines the input sensor data;
An image data refiner that refines the input image data;
A connection data forming unit connecting the refined sensor data and the image data to form connected data;
A fully connected layer unit that inputs the connected data into a fully connected layer of the deep reinforcement learning algorithm to obtain a Q value; And
An action decision unit that determines an action based on the Q value.
Including,
The sensor data purification unit,
The sensor data is purified using LSTM (Long Short-Term Memory),
The image data refinement unit,
The image data is refined using a convolutional neural network (CNN),
The deep reinforcement learning algorithm analyzes multiple inputs to use two different inputs at the same time to learn the driving policy, and after connecting the two refined sensor data and the image data, a fully connected layer ) To finally get the Q value and determine the action
An autonomous vehicle control device using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 9,
The input unit,
Receiving the sensor data measured through the LIDAR sensor configured in the vehicle and the image data photographed through the camera as an in-depth reinforcement learning algorithm
An autonomous vehicle control device using an in-depth reinforcement learning and driver assistance system, characterized in that. delete delete The method of claim 9,
The fully connected layer part,
Using a network that divides the fully connected layer into two parts: a state value and a behavioral advantage that evaluates the benefits for each behavior, and obtains the Q value by integrating the state value and the behavioral advantage.
An autonomous vehicle control device using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 9,
The vehicle control unit,
Including five actions: increasing the target speed, decreasing the target speed, changing lanes to the left, changing lanes to the right, and maintaining the current state so that the vehicle can drive in the longitudinal and transverse directions simultaneously.
An autonomous vehicle control device using an in-depth reinforcement learning and driver assistance system, characterized in that. The method of claim 9,
The deep reinforcement learning unit,
Designing a reward according to the result of the action to find an optimal driving policy based on high speed driving of the vehicle, driving on a non-collision trajectory, and excluding unnecessary lane changes, and learning the deep reinforcement learning algorithm with the driving policy
An autonomous vehicle control device using an in-depth reinforcement learning and driver assistance system, characterized in that.

Images