KR101951595B1 - Vehicle trajectory prediction system and method based on modular recurrent neural network architecture - Google Patents

Abstract

A method for predicting a vehicle path based on a modular circulating neural network structure is disclosed. According to an embodiment of the present invention, a method for predicting a path performed by a system for predicting a path including an encoder and a decoder comprises the steps of: analyzing, in the encoder, sequence information based on driving data of a vehicle of a user or the vehicle of the user and a neighboring vehicle present around the vehicle of the user; and predicting, in the decoder, a future path of the neighboring vehicle by applying an analysis result of analyzing the sequence information to a beam search. Deep learning can be performed based on a combined circulating neural network composed of at least one circulating neural network structure.

Description

[0001] VEHICLE TRAJECTORY PREDICTION SYSTEM AND METHOD BASED ON MODULAR RECURRENT NEURAL NETWORK ARCHITECTURE [0002]

The following description relates to a method and system for predicting the path of a vehicle.

It is one of the most important goals to develop an autonomous driving vehicle with the ability to analyze and predict the motion of nearby vehicles. In order for autonomous vehicles to secure high level of safety, it is necessary to analyze the tendency inherent in the movement of nearby vehicles, to identify the main characteristics, and to predict the route to be developed in the future. However, it is a difficult problem to analyze the characteristics of the vehicle and to predict the future path because it is determined by interacting with the potential variables changing in real time according to traffic conditions. In order to solve the problem of the vehicle path analysis and prediction, there is a method of estimating the vehicle path based on the Kalman filter, the Gaussian probability process, the Gaussian mixture model based statistical model, and the dynamic Bayesian network structure. Have been proposed.

In recent years, machine learning techniques of deep neural networks called deep running have been developed, and techniques using the deep neural networks have been proposed. The analysis of the route of the surrounding vehicles in the running and the prediction of the future route are based on the basic technology which is essential for the implementation of the useful technology such as the path planning, the risk assessment, the maneuver estimation, And this technology can be applied to autonomous vehicles or driver assistance systems (ADAS) to develop a reliable system for safe driving. In addition, if the analysis and prediction of the vehicle movement route is performed correctly, it is possible to estimate the driver's state and to warn of danger such as drowsiness operation, and various application techniques such as a technique of selecting a proper base station by estimating the position of the mobile communication terminal Development is possible.

The techniques that use machine learning can be roughly divided into vidivine learning based technology and deep learning based technology. Based on the DBN model, the vehicle routing method is based on the DBN model. Deep learning based techniques include the vehicle position estimation method based on the LSTM circular neural network structure, the circular neural network structure and the CVAE (Conditional Variational Auto- Encoder based vehicle path prediction. All of these techniques have in common that they determine the parameters used in the algorithm through machine learning. As an example of the bidirectional learning method, the vehicle path prediction algorithm based on the dynamic Bayesian network (DBN) model expresses the latent variables involved in determining the path of the vehicle as a graphical model, and learns the interaction between the variables by machine learning. In the vehicle path prediction algorithm based on the dynamic Bayesian network (DBN) model, the feature points that are considered to be highly correlated with the vehicle path such as the road shape and the traffic law are set as human intuition. And learning the parameter of the function that represents the interaction between the minutiae.

On the other hand, the technique of deep learning follows a method of directly learning the feature points for analyzing the vehicle path through machine learning without designing in advance. First, the vehicle position prediction method based on the LSTM recurrent neural network structure predicts the future position of the surrounding vehicle by accepting the sensor input of the motion state such as the car speed, the steering angular velocity, the relative position and the relative speed of the surrounding vehicles. Based on deep learning, this technique learns feature points that are highly correlated with the future position of the vehicle by machine learning, and outputs a map showing the probability that the nearby vehicle will be at a specific point in the future based on the learned feature points. In addition, the vehicle path prediction method based on the cyclic neural network structure and the CVAE (Conditional Variable Auto-Encoder) was introduced with a sampling path by CVAE (Conditional Variable Auto-Encoder) together with a cyclic neural network structure The hypothesis with a certain amount of importance is output by using a separate neural network that gives various hypotheses about the future path and estimates the importance between the hypotheses.

It is difficult to flexibly cope with various traffic conditions because of the fundamental limitation that dynamic base network (DBN) -based technique can consider only human pre-designed feature points. This is because the prediction performance decreases if the pre-designed feature points do not have a significant correlation in determining the vehicle path in a specific traffic situation. On the other hand, the deep-learning technology that has been recently undergone a lot of research uses a method of learning the useful features directly from the vast amount of data, so that it can maintain relatively good performance in various environment changes.

However, Deep Learning based vehicle path analysis and prediction techniques also fail to generate future path sequences or to stochastically express various future paths. Vehicle location prediction method based on LSTM recurrent neural network structure is not able to generate a sequence and has a limit to predict only the position of a specific time point in the future route prediction of a vehicle. Uncertainty increases in path planning or risk measurement , It becomes difficult to ensure the safety of the passenger. In addition, since the vehicle path prediction method based on the cyclic neural network structure and the CVAE (Conditional Variable Auto-Encoder) can not represent the probabilities of various future paths stochastically, a separate apparatus is required to calculate the importance between the various predictions The amount of computation increases and the probabilistic analysis of the feasibility of the hypothesis is ambiguous because of the lack of stochastic representation. There are limits to the possibility that the probabilistic expression of various future path hypotheses is impossible in the vehicle path prediction technique because the moving path of the vehicle significantly contains the probabilistic characteristic that can be variously differentiated according to the driver's judgment from the same situation.

References: KR10-1795250, KR10-2018-0020615

We propose a vehicle path prediction method based on modular circular neural network structure for autonomous vehicle or ADAS (Advanced Driver Assistance System). Specifically, to solve the problem of the vehicle position prediction method using the deep learning that predicts only the position at a specific time point rather than the future path sequence or does not have a stochastic representation, the encoder-decoder combined cycle A neural network structure and a beam search algorithm can be used to provide a method for predicting future paths of nearby vehicles.

A path prediction method performed by a path prediction system including an encoder and a decoder includes analyzing sequence information based on a driving vehicle of a peripheral vehicle existing in a periphery or a car or a car in the encoder; And a step of estimating a future path of a neighboring vehicle by applying a beam search to the analysis result obtained by analyzing the sequence information in the decoder, So that the deep learning learning can be performed, respectively.

The step of analyzing the sequence information may include driving state information of the vehicle and travel route information of the surrounding vehicle by performing the deep learning learning based on the driving data of the surrounding vehicle existing in the vicinity of the vehicle or the vehicle And analyzing the sequence information.

The step of analyzing the sequence information may include: running data including a speed of the vehicle collected from the vehicle or a sensor of the nearby vehicle, a steering angular velocity of the vehicle, a relative coordinate of the nearby vehicle, and a relative speed of the nearby vehicle And extracting pattern information between the vehicle and the surrounding vehicles by analyzing the sequence information based on the speed of the vehicle, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle. And summarizing.

Wherein the step of analyzing the sequence information comprises a configuration in which the encoder is configured as a mixture of FC (Fully-Connected) neural network layer and LSTM (Long-Short Term Memory) , An affine transformation and a rectified linear unit (ReLU) activation function, extracts relationship information related to the motion between the car and the neighboring vehicle, coincides the dimensions of the data, and the LSTM recurrent neural network layer of the encoder A plurality of LSTM circular neural networks are connected to each other, and each cell vector can be obtained by performing a recursive operation on the input sequence in the plurality of LSTMs.

The step of predicting a future route of the neighboring vehicle may include repeating a recursive operation based on the beam search from the summarized summary sequence by analyzing the sequence information in the decoder until reaching a preset predicted time, And outputting a hypothesis over a predetermined probability based on the generated output sequence.

The step of predicting a future path of the neighboring vehicle may include at least one FC neural network layer, at least one LSTM loop neural network layer, a Softmax layer for calculating a probability distribution, and an embedding layer for feeding back an output And an output value recursively calculated using a plurality of cell vectors received from the encoder in the FC neural network layer and the LSTM loop neural network layer of the decoder is passed through the Softmax layer, Wherein the predetermined number of candidates is represented by a conditional probability occupying an Occupancy Grid Map, applying a beam search based on the conditional probability to select a predetermined number of candidates for a position of a vehicle or a nearby vehicle, Lt; RTI ID = 0.0 > a < / RTI >

Wherein the step of predicting a future path of the neighboring vehicle comprises a plurality of matrices corresponding to the number of grids in the vertical axis and the horizontal axis of the occupancy grid map in the embedding layer of the decoder, Extracting the column vectors corresponding to the column vectors from the respective matrices, and then performing vector concatenation of the extracted column vectors to feed back the input row vectors to the input of the LSTM recurrent neural network layer of the decoder.

The route prediction system includes: an encoder for analyzing sequence information based on traveling data of a vehicle or a neighboring vehicle that is present around the vehicle or the vehicle; And a decoder for predicting a future path of a neighboring vehicle by applying a beam search to an analysis result obtained by analyzing the sequence information. The apparatus of claim 1, Learning can be performed.

The encoder performs deep learning learning on the basis of the driving data of the surrounding vehicle existing in the periphery of the vehicle or the vehicle in the encoder, and analyzes the sequence information including the driving state information of the vehicle and the traveling path information of the surrounding vehicle can do.

Wherein the encoder receives driving data including a speed of the vehicle collected from the vehicle or a sensor of the peripheral vehicle, a steering angular speed of the vehicle, a relative coordinate of the surrounding vehicle, and a relative speed of the surrounding vehicle, The pattern information between the vehicle or the surrounding vehicles can be extracted and summarized by analyzing the sequence information based on the speed of the vehicle, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle .

Wherein the encoder comprises a mix of a FC (Fully-Connected) neural network layer and a LNS (Long-Short Term Memory) loopback neural network layer, wherein the FC neural network layer of the encoder comprises an affine transformation wherein the LSTM recurrent neural network layer of the encoder comprises a plurality of LSTM recurrent neural networks, each of the plurality of LSTM recurrent neural networks comprising: , And each cell vector can be obtained by performing a recursive operation on the input sequence in the plurality of LSTMs.

The decoder repeatedly performs a recursive operation based on the beam search from the summarized summary sequence by analyzing the sequence information until a predetermined predicted time is reached to generate an output sequence, and based on the generated output sequence, A hypothesis more than the set possibility can be outputted.

Wherein the decoder comprises at least one FC neural network layer, at least one LSTM loop neural network layer, a Softmax layer for calculating a probability distribution, and an embedding layer for feeding back the output, Layer and the LSTM loop neural network layer, the output value recursively calculated using a plurality of cell vectors received from the encoder is passed through the Softmax layer so that the own vehicle or the neighboring vehicle occupies an Occupancy Grid Map A predetermined number of candidates for a position of a vehicle or a nearby vehicle may be selected by applying a beam search based on the conditional probability, and the selected candidate may be fed back through the embedding layer.

Wherein the decoder comprises a plurality of matrices corresponding to the vertical axis and the horizontal axis of the occupancy grid map in the embedding layer of the decoder and the column vectors corresponding to the selected grid coordinates based on the beam search, After extraction from the matrix, the extracted column vectors may be vector concatenated and fed back to the input of the LSTM recurrent neural network layer of the decoder.

The path prediction system according to the embodiment can predict the future path of the neighboring vehicle more accurately by performing processing by classifying the encoder and the decoder constituted by the combined circular neural network structure.

The route prediction system according to the embodiment can predict and track the route of the neighboring vehicle continuously at a predetermined time, as opposed to predicting the future one point, for example, the future after a specific time.

The route prediction system according to an exemplary embodiment can provide more useful and reliable information to perform a traveling route plan of a nearby vehicle even when generating a plurality of hypotheses.

FIG. 1 is a diagram for explaining an operation of a path prediction system according to an embodiment.
2 is a view for explaining a method of performing a beam search in a decoder of a path prediction system according to an embodiment.
3 is a graph illustrating performance of a path prediction system according to an embodiment.
4 is a block diagram illustrating a configuration of a path prediction system according to an embodiment.
5 is a flowchart for explaining a path prediction method of the path prediction system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

In the following embodiments, a modular recurrent neural network structure based vehicle path prediction scheme for an autonomous vehicle or an ADAS (Advanced Driver Assistance System) will be described. An encoder for generating one state vector for a past sequence and a decoder for generalizing a sequence for future positions through a state vector may be provided as a module for modulating a route prediction system and a method for applying to a route estimation of a vehicle . The vehicle path prediction system proposed in the embodiment can be configured as a combined type module including a circular neural network module for analyzing vehicle motion and a circular neural network module for generating a future path based on the analysis. The multiple cyclic neural network module learns weight and bias coefficient through the deep learning learning, and the combined cyclic neural network in which the learning is performed calculates the yaw rate of the car, the yaw rate of the car, The sequence information on the relative coordinates and the relative speed can be analyzed and the future route of the neighboring vehicle can be generated based on the analysis.

FIG. 1 is a diagram for explaining an operation of a path prediction system according to an embodiment.

The path prediction system can perform the deep learning learning through the combined circular neural network 120 based on the traveling data 110 of the neighboring vehicle existing around the car or the car (vehicle).

The combined circular neural network 120 may include an encoder 121 and a decoder 122. The encoder 121 may generate a fixed-length vector that summarizes the time-series information based on the sequence information including the movement path of the surrounding vehicle and the motion state information of the vehicle. The decoder 122 may predict and generate a future path sequence (output sequence) of the nearby vehicle based on the fixed length vector. The decoder 122 may use a recursive prediction method of sequentially generating the future route sequence of the neighboring vehicle by using the predicted position of the vehicle as an input at the next prediction by feeding back the predicted position of the vehicle each time. In this case, a beam search algorithm can be applied as a recursive prediction method. Through this, it is possible to maintain K candidates (K is a natural number) having a probability of locally higher than a certain rank each time the prediction is performed have. Accordingly, the path prediction system has a system for generating a plurality of sequences for the future path of the vehicle in a stochastic manner.

Before explaining the combined circular neural network 120 including the encoder and the decoder proposed in the embodiment, the operation structure of the LSTM circular neural network will be described in order to facilitate understanding of the combined circular neural network. LSTM (Long-Short Term Memory) The LSTM model is one of universal recurrent neural network types. It is composed of a cell memory which stores information of input sequence, a gatekeeper device which controls the forgetting and inputting / gating mechanism. The following equations (1-1) to (1-5) are recursive equations describing the operation of the LSTM recurrent neural network.

는 sigmoid 함수,

는 요소별 곱셈(element-wise product),

는 입력 벡터,

는 선형 변환 행렬,

는 편향(bias) 벡터,

는 문지기(gating) 벡터,

는 셀 메모리(cell memory),

는 상태 출력 벡터를 의미한다. Hereinafter, definitions of the variables described in the respective equations will be described. here,

The sigmoid function,

Is an element-wise product,

Is an input vector,

Is a linear transformation matrix,

Is a bias vector,

A gating vector,

A cell memory,

Denotes a state output vector.

와 입력 문지기 벡터

를 이용하여 망각과 입력의 정도를 제어한다. 수학식(1-5)는 갱신된 셀 메모리 상태의 출력을 계산하는 방정식으로, 출력 문지기 벡터

를 통해 셀 메모리 상태의 정도를 제어한다. 여기서, 문지기 벡터들은 sigmoid 함수를 통해 계산되기 때문에 그 요소들은 반드시 폐구간 [0,1]에 속하는 실수를 가지며, 요소별 곱셈(element-wise product)을 통해 망각 및 입력값/출력값을 제어한다. 예를 들면, 만약, 망각 문지기 벡터

의 특정 요소(element) 값이 0에 가깝다면 해당 요소에 대응하는 셀 메모리 요소는 수학식 (1-4)의 계산을 통해 대부분 삭제될 수 있고, 반대로 특정 요소의 값이 1에 가깝다면 그에 대응하는 셀 메모리 요소는 대부분 유지될 수 있다. 나머지 두 개의 문지기 벡터

와

의 경우에도 이와 유사하게 작동하여 입력과 출력의 정도가 제어될 수 있다. The equations (1-1), (1-2) and (1-3) determine the forget gating vector, the input gating vector and the out gating vector, respectively . Equation (1-4) is an equation for updating the cell memory from the previous time based on time t when a new input is received at time t,

And input gatekeeper vector

To control the degree of forgetting and input. Equation (1-5) is an equation for calculating the output of the updated cell memory state. The output gatekeeper vector

To control the degree of cell memory state. Here, since the gatekeeper vectors are computed by the sigmoid function, they have a real number belonging to the closed interval [0, 1] and control the forgetting and input / output values through an element-wise product. For example, if the observer door vector

The cell memory element corresponding to the element can be deleted most of the time through the calculation of Equation (1-4). On the contrary, if the value of the specific element is close to 1, the corresponding Most of the cell memory elements can be maintained. The other two gatekeepers vectors

Wow

The degree of input and output can be controlled.

에 대한 출력 시퀀스

의 조건부 확률, 다시 말해서

를 모사하는데 그 목적을 두고 있다. 인코더(121)는 입력 시퀀스

(길이

)를 처리하기 위해 수학식(1-1) 내지 수학식(1-5)의 연산을 재귀적으로 T(T는 자연수)회 수행하여 셀 메모리

를 생성할 수 있다. 상기 생성된 셀 메모리

는 디코더로 전달되어 디코더 LSTM 셀 메모리

의 초기값으로 사용되어

와 같이 설정될 수 있다. 디코더(122)의 출력 시퀀스 생성은 인코더(121)와 유사하게 수학식(1-1) 내지 수학식(1-5)의 연산을 재귀적으로 수행하며 이루어지고, 이때 입력값으로는 특정 시간으로부터 기 설정된 시간 이전의 이전 시점의 출력값이 피드백되는 구조를 갖는다. 이에 따라 조건부 확률

는 수학식 2와 같은 방법으로 근사될 수 있다. As described above, the encoder 121 and the decoder 122 applying the LSTM recurrent neural network generate the input sequence

Output sequence for

Conditional probability of

The purpose of this is to simulate. The encoder 121 receives the input sequence

(Length

(1-1) to (1-5) is recursively performed for T (T is a natural number)

Lt; / RTI > The generated cell memory

Lt; RTI ID = 0.0 > LSTM < / RTI &

Is used as the initial value of

. ≪ / RTI > The output sequence generation of the decoder 122 is performed by recursively performing operations of equations (1-1) to (1-5) similarly to the encoder 121, And the output value of the previous time point before the preset time is fed back. Therefore,

Can be approximated by the same method as in Equation (2).

Equation 2:

에 따라서 출력에 대한 의사결정(decision making)을 수행해야 한다. 실시예에서 적용된 의사결정 방법으로 빔 탐색(beam search) 알고리즘을 사용할 수 있다. 도 2를 참고하면, 빔 탐색 알고리즘을 설명하기 위한 도면으로, 디코더에서의 빔 탐색을 예를 들어 설명하기로 한다. 빔 탐색 알고리즘은 디코더(122)의 출력을 결정할 때마다 빔 폭(beam width)이라 정의되는 K개(K는 자연수)의 후보를 유지하는 방식을 기초로 한다. 빔 탐색이 적용된 디코더는 매번 출력 생성 시 조건부 확률을 기준으로 기 설정된 기준에 기반하여 설정된 상위 K가지의 출력을 결정한 후, 다음 단계로 피드백할 수 있다. 다음의 출력을 생성할 때에는 피드백된 K개의 출력에 대하여 각각 조건부 확률을 계산하고, 그 중에서 확률이 높은 상위 K개의 출력을 정해 피드백하는 과정을 반복할 수 있다. In order for the decoder 122 to perform a recursive operation, the output of the previous stage must be written as the input of the next stage,

Decision making should be done according to the output. A beam search algorithm can be used as a decision method applied in the embodiment. Referring to FIG. 2, a beam search algorithm in a decoder will be described with reference to a beam search algorithm. The beam search algorithm is based on a method of maintaining K candidates (K is a natural number) defined as a beam width each time the output of the decoder 122 is determined. The decoder with beam search can determine the output of the top K sets based on the predetermined criteria based on the conditional probability at the time of outputting each time and then feed back to the next stage. When generating the next output, it is possible to calculate the conditional probability for each of the K outputs fed back, and to repeat the process of setting the upper K outputs with high probability and feeding back the output.

), 자차의 조향 각속도(

), 주변 차량의 상대 좌표(

) 및 주변 차량의 상대 속도(

)에 대한 시퀀스를 사용할 수 있다. 이때, 주변 차량의 상대 좌표는 자차를 기준으로 판단되는 위치 좌표를 의미할 수 있다. 경로 예측 시스템의 심층 신경망이 주변 차량의 미래 경로를 출력할 때, 로봇 분야에서 쓰이는 점유 격자 지도(Occupancy Grid Map)(130)를 사용할 수 있다. 점유 격자 지도는 자차 주변의 공간을 일정한 개수의 직사각 격자로 분할함으로써 형성되며 이를 통하면 주변 차량의 위치를 격자 중의 하나로 정하거나 격자 범위 밖으로 벗어나는 것으로 처리하여 표현하게 된다. 예를 들면, 경로 예측 시스템은 점유 격자 지도에 주변 차량의 미래 경로, 예를 들면, 방향, 위치 등을 표시할 수 있다. The path prediction system is an input for predicting the vehicle path,

), The steering angular velocity of the vehicle

), Relative coordinates of nearby vehicles (

) And the relative speed of the surrounding vehicles

) Can be used. In this case, the relative coordinates of the surrounding vehicles may be position coordinates determined based on the vehicle. When the in-depth neural network of the path prediction system outputs the future path of the neighboring vehicle, an Occupancy Grid Map (130) used in the robot field can be used. The occupancy grid map is formed by dividing the space around the vehicle into a certain number of rectangular grids. In this case, the position of the surrounding vehicle is defined as one of the lattices or processed by being out of the lattice range. For example, the path prediction system may display a future path of a nearby vehicle, for example, a direction, a location, and the like, on a occupied grid map.

동안의 경로를 재귀적으로 생성할 수 있다. 디코더(122)의 연산에 빔 탐색 알고리즘이 사용될 수 있다. 디코더(122)는 빔 탐색 알고리즘을 통하여 K개의 가장 가능성 큰 가설을 생성할 수 있다. 이와 같은 과정을 모든 주변 차량에 대하여 시행함에 따라 경로 예측 시스템은 최종적으로 각각의 주변 차량의 미래 경로에 대한 K개의 시퀀스를 획득할 수 있다. 이때, 획득된 시퀀스(결과)를 단일한 점유 격자 지도 위에 표현하면 자차 주변의 주변 차량이 미래

동안에 어떻게 이동할 것인지 예측 경로를 통합적으로 확인 및 추적할 수 있다. The path prediction system can generate a cell vector that summarizes the sequence information through the recursive operation in the encoder 121 as the collected sequence is input to the encoder 121 for at least one or more neighboring vehicles existing around the car have. The decoder 122 decodes the future

Can be recursively generated. A beam search algorithm may be used in the operation of the decoder 122. [ Decoder 122 may generate K most likely hypotheses through a beam search algorithm. As this process is performed for all the nearby vehicles, the route prediction system can finally obtain K sequences for the future route of each neighboring vehicle. At this time, if the obtained sequence (result) is expressed on a single occupied grid map,

You can identify and track the predicted path in an integrated way.

와

를 획득할 수 있다. 일례로, 자차 및 주변 차량으로부터 6개의 입력값을 수신할 수 있다. 이때, 각각의 입력값에 대하여 기 설정된 시간 동안 관측할 경우, 기 설정된 시간마다 시퀀스를 생성할 수 있다. 예를 들면, 자차 또는/및 주변 차량을 3초 동안 관측함에 따라 획득된 값에 대하여 0.1초마다 샘플링할 경우, 길이 30의 시퀀스를 생성할 수 있다. 이러한, 시퀀스를 인코더의 입력값으로 입력할 수 있다. 인코더의 FC 계층에 6차원의 입력값을 입력함에 따라 256차원의 시퀀스로 변환될 수 있다. 이때, FC 계층에서 FC 연산을 3회 수행함에 따라 생성된 256차원의 길이 30의 시퀀스가 LSTM 순환 신경망 계층의 입력값으로 입력될 수 있다. 이때, LSTM 2개의 레이어로 구성되어 스택되어 있을 수 있다. 이러한 과정을 일정 시간동안 관측함에 따라 압축된 256차원의 벡터가 디코더로 전달될 수 있다. The encoder 121 proposed in the embodiment may be configured as a mixture of a FC (Fully-Connected) neural network layer and an LSTM (Long-Short Term Memory) circular neural network layer. The FC neural network layer in the encoder consists of affine transformation and ReLU (Rectified Linear Unit) activation functions. FC neural network consisting of affine transformation and ReLU (Rectifed Linear Unit) activation function can help to extract complex relations included in motion between vehicles by increasing the model expressiveness by matching the dimension of data. The LSTM recurrent neural network layer in the encoder 121 may have a structure in which a plurality of (for example, two) LSTM circulating neural networks are connected. In the two LSTMs, a recursive operation is performed on the input sequence, Cell vector

Wow

Can be obtained. For example, six input values can be received from a car and a nearby vehicle. In this case, when observing each input value for a predetermined time, a sequence can be generated at predetermined time intervals. For example, a sequence of length 30 can be generated if sampling is performed every 0.1 seconds for the value obtained as a result of observing the vehicle and / or the surrounding vehicle for three seconds. This sequence can be input as the input value of the encoder. Dimensional input values into the FC layer of the encoder, and thus can be converted into a sequence of 256 dimensions. In this case, a sequence of 256 lengths of length 30 generated by performing FC operations three times in the FC layer can be input as an input value of the LSTM loop neural network layer. At this time, the LSTM may be composed of two layers and stacked. As this process is observed for a certain period of time, a compressed 256-dimensional vector can be transmitted to the decoder.

에 다다를 때까지 반복적으로 수행되며 최종적으로 K개의 출력 가설을 획득할 수 있다. 예를 들면, 인코더의 LSTM 순환 신경망 계층에서 출력된 출력값인 복수 개의 256차원의 벡터가 디코더의 LSTM 순환 신경망 계층에 입력될 수 있고, 디코더의 LSTM 순환 신경망 계층에 통과된 출력값인 256 차원의 벡터를 FC 신경망 계층에 입력시킬 수 있다. FC 신경망 계층에 256 차원의 벡터를 통과시킴에 따라 수행된 연산에 기초하여 256 차원의 벡터, 256 차원의 벡터 및 757 차원의 벡터가 출력될 수 있다. 이때, 자차 전방을 기준으로부터 일정 범위를 포함하는 특정 구역을 점유 격자 지도로 구분할 수 있다. 예를 들면, 점유 격자 지도를 가로 21, 세로 36 및 상기 특정 구역 이외로 벗어 나는 경우를 고려하여, 주변 차량의 미래 위치를 이산화하기 위한 총 757 가지의 경우를 예측할 수 있다. 이러한 벡터를 softmax 연산을 수행함에 따라 0 내지 1 사이의 범위로 정규화하여 확률 분포로 해석할 수 있다. softmax 연산을 수행함에 따라 빔 서치를 수행할 수 있다. 일례로, 점유 격자 지도에서 주변 차량이 특정 시점에 존재할 가능성이 제일 높은 확률에 해당하는 위치(지점)만 표시될 경우, 차량이 어느 방향으로 이동할 것인지, 멈출 것인지 등 한가지만의 예측으로 피드백되는 것을 발생하는 문제점을 해결하고, 주변 차량에 대한 다양한 이동 방향을 예측하기 위하여 빔 서치가 수행될 수 있다. 이러한, softmax 연산을 수행할 결과인 757 차원의 확률 분포를 빔 서치를 수행함에 따라 일정 순위 이상의 결과값만을 추출할 수 있다. 미래의 특정 시점의 위치를 설정할 때, 일정 순위 이상의 결과에 대한 지점을 임베딩에 통과시켜 다시 벡터화하여 LSTM 계층에 입력값으로 입력할 수 있다. The decoder 122 proposed in the embodiment includes a Fully-Connected layer, an LSTM loop neural network layer, a Softmax layer for calculating a probability distribution, and an embedding layer for feeding back the output. . ≪ / RTI > The LSTM layer and the FC layer of the decoder 122 may recursively calculate an output value using a plurality of (e.g., two) cell vectors received from the encoder 121. [ When this output value passes through the Softmax layer, it is expressed by the conditional probability that the surrounding vehicle occupies the grid of the occupied grid map. Next, if the beam search algorithm is applied based on the conditional probability, K (K is a natural number) candidates for the position of the vehicle (neighboring vehicle) can be selected. The selected candidates can be fed back through the embedding layer. The embedding layer can be composed of two matrices corresponding to the vertical axis and the horizontal axis lattice of the occupied grid map, extracting the column vectors corresponding to the grid coordinates selected by the beam search from the respective matrices, (Vector concatenation) to the input of the decoder. This process is based on the prediction range

, And finally obtain K output hypotheses. For example, a plurality of 256-dimensional vectors, which are output values output from the LSTM loop neural network layer of the encoder, may be input to the LSTM loop neural network layer of the decoder, and a 256-dimensional vector, which is an output value passed through the LSTM loop neural network layer of the decoder, Can be input to the FC neural network layer. A 256-dimensional vector, a 256-dimensional vector, and a 757-dimensional vector may be output based on the operation performed by passing a 256-dimensional vector through the FC neural network layer. At this time, the specific area including a certain range from the reference ahead of the vehicle can be divided into the occupied grid map. For example, considering the case where the occupied grid map is 21, 36, and the case where the map is deviated from the specific area, a total of 757 cases for discretizing the future position of the nearby vehicle can be predicted. Such a vector can be normalized to a range of 0 to 1 by performing a softmax operation and interpreted as a probability distribution. The beam search can be performed according to the softmax operation. For example, if only the position (point) corresponding to the probability that the nearby vehicle exists at a specific time in the occupancy grid map is displayed, the vehicle is fed back to only one prediction such as which direction the vehicle will move or stop A beam search can be performed to solve the problems that occur and to predict various directions of travel for the surrounding vehicles. By performing the beam search on the probability distribution of the 757-dimensional result that is the result of performing the softmax operation, only the result value of a certain rank or more can be extracted. When setting the position of a specific point in time in the future, a point for a result of a predetermined rank or more can be passed through the embedding and re-vectorized and input as an input value to the LSTM layer.

의 합인 (

) 크기로 분할하여 생성할 수 있다. 센서에서 수집된 주변 차량 움직임이 충분히 정확하다면, 기 설정된 시점 이후에 대한 후반부

만큼의 데이터는 점유 격자 지도에 대한 격자 좌표로 변환하여 지도 학습을 위한 정답으로 사용할 수 있다. 신경망 학습에 사용되는 손실 함수는 수학식3과 같은 음의 로그 우도 함수가 사용될 수 있다.Neural network parameters, including the embedding matrix, can be learned end-to-end. The learned data includes a record of the movement of the surrounding vehicle as an input length M, an output length

(

) Size. If the motion of the surrounding vehicle collected by the sensor is sufficiently accurate,

Can be converted into grid coordinates for the occupancy grid map and used as the correct answer for the map learning. A negative logarithmic likelihood function as shown in Equation (3) can be used for the loss function used for neural network learning.

Equation (3)

는 예측 길이, Q는 격자 클래스 개수이다.

는 원 핫 인코딩(one hot encoding)으로 표현된 라벨로서, 만약 j번째 샘플에서 주변 차량이 q번째 격자를 점유하고 있다면

는 1이 되고, j번째 샘플에서 주변 차량이 q번째 격자 이외의 다른 격자를 점유한 경우에는 0이 된다. 마지막으로

는 Softmax 계층의 출력으로 획득된 주변 차량이 q번째 격자를 점유할 확률을 의미한다. In Equation (3), J is the number of learning samples,

Is the predicted length, and Q is the number of grid classes.

Is a label expressed by one hot encoding, and if the neighboring vehicle occupies the qth lattice in the jth sample

Becomes 1, and becomes 0 when the neighboring vehicle occupies another lattice other than the q-th lattice in the j-th sample. Finally

Denotes the probability that the neighboring vehicle obtained as the output of the Softmax layer occupies the q-th grid.

3 is a graph illustrating performance of a path prediction system according to an embodiment.

=20)로 설정하고, 적절한 초기 학습속도(learning rate)의 설정 하에 검증 데이터에 대한 오차 개선이 정체될 때마다 학습 속도를 절반으로 낮추는 방식을 이용하며 모델의 파라미터를 최적화할 수 있다. The path prediction system can predict the path of the neighboring vehicle by using the combined circular neural network structure and the beam search algorithm. An encoder composed of a combined circular neural network structure analyzes motion data of a vehicle, and a decoder composed of a combined circular neural network structure can generate a future path based on analysis. The encoder can sequentially analyze the sequence of the movement path of the peripheral vehicle and the motion state of the vehicle collected by the sensor to extract and summarize the inherent pattern. The decoder may receive information summarizing the input sequence of the encoder and generate an output sequence containing the future path of the surrounding vehicle. At this time, a recursive beam search algorithm is used when the decoder generates the output sequence, which can output the K hypotheses most likely to the future route of the neighboring vehicle. For example, machine learning and performance evaluation can be carried out using the traveling data of a car and a nearby vehicle collected through a highway driving. In this case, the range of the path prediction is 2 seconds (

= 20), and the parameter of the model can be optimized by using a method of lowering the learning rate by half each time the improvement of the error of the verification data is stalled under the setting of an appropriate initial learning rate.

We define Top-K Mean Absolute Error (MAE) as a measure of the performance of the model. Equation (4) can be obtained by calculating the Mean Absolute Error between the correct answer and the hypothesis by selecting one of the top K hypotheses obtained through the proposed model, which is closest to the correct answer.

Equation 4:

)는 정답에 가장 가까운 가설 시퀀스 샘플의 격자 좌표, (

)는 대응하는 정답 시퀀스 샘플을 의미한다. 이 척도는 모델이 생성하는 K개 가설 중 하나가 정답을 얼마나 정확하게 예측할 수 있는지 보여준다. Top-K MAE의 단위는 '격자'이다. 성능의 비교를 위하여 전통적인 등속 모델에 의한 칼만 필터, 주변 차량이 Occupancy Grid Map 격자를 점유할 확률을 예측하도록 학습시킨 기본적인 LSTM 모델, 종래의 LSTM 신경망 구조의 차량 경로 예측 기법에서 제안된 보다 정교한 형태의 LSTM 모델을 대조군으로 설정할 수 있다. 이에 따라, 실시예에서 빔 탐색 결과 출력하는 가설 개수 K는 1, 3, 5 세 가지 설정을 비교할 수 있다. 표 1은

=0.4, 0.8, 1.2, 1.6, 2.0초에서 경로 예측 기법들의 MAE를 확인할 수 있다. In Equation 4, L is the number of verification data samples, (

) Is the lattice coordinate of the hypothetical sequence sample closest to the correct answer, (

) Denotes a corresponding correct answer sequence sample. This scale shows how accurately one of the K hypotheses generated by the model can predict the correct answer. The unit of Top-K MAE is 'grid'. In order to compare the performance, a Kalman filter with a conventional constant velocity model, a basic LSTM model in which a neighboring vehicle learns to predict the probability of occupying an Occupancy Grid Map grid, and a more sophisticated form proposed in the conventional LSTM neural network structure The LSTM model can be set as a control group. Accordingly, in the embodiment, the number of hypotheses K for outputting the beam search result can be compared among the three settings of 1, 3, and 5. Table 1

= 0.4, 0.8, 1.2, 1.6, and 2.0 seconds, respectively.

Table 1:

에 대하여 값을 구하여 표기하였다. 반면, 기본 LSTM 대조군과 종래의 LSTM 신경망 구조의 차량 경로 예측 기법에 대한 대조군의 경우 오로지 단일한 시각에서의 위치만 예측할 수 있기 때문에

=2.0초에 대한 MAE 만을 표기하였다. 실험 결과에 따르면 실시예에서 제안된 방법은 오직 한 가지의 가설만 고려하는 K=1설정만으로도 칼만 필터 대조군과 LSTM 대조군에 비해 현저하게 우수한 성능을 보여준다. 또한, 종래의 LSTM 신경망 구조의 차량 경로 예측 기법의 대조군과 비교하여도 근소하게 우수한 성능을 보이며, K를 3, 5 등으로 변경하여 가설의 개수를 증가시키게 되면 더욱 우수한 성능을 보이는 것을 확인할 수 있다.In the case of the Kalman filter control, the sequence can be generated through recursive operations,

And the values were obtained. On the other hand, in the case of the control group for the vehicle path prediction method of the basic LSTM control and the conventional LSTM neural network structure, only the position at a single time can be predicted

= 2.0 sec. ≪ / RTI > Experimental results show that the proposed method shows significantly better performance than the Kalman filter control and the LSTM control with only K = 1 setting considering only one hypothesis. In addition, the performance of the conventional LSTM neural network structure is slightly better than that of the control group of the vehicle path prediction method, and it is confirmed that the performance is improved when the number of hypotheses is increased by changing K to 3, 5, .

와 가설 수 K에 따라서 어떻게 변화하는지를 확인할 수 있다. 도 3을 참고하면, K에 따른 MAE의 변화와 전체 시각에 대한 평균을 나타낸 그래프이다. K가 증가함에 따라 모든 시각에서 MAE 성능이 향상됨을 확인할 수 있다. 이때, K의 증가로 인한 성능 향상 정도가 더욱 더 먼 미래 시각일수록 뚜렷하게 나타나게 된다. 예를 들면,

=2.0초에서 K=1을 K=3으로 증가시켰을 때, 약 0.2 정도의 MAE 감소를 확인할 수 있다. 이는 가설의 개수를 늘릴수록 모델이 실제 정답에 가까운 예측을 도출할 가능성이 커진다는 것을 의미한다. 이에 따라 다수의 가설을 생성하게 되어도 차량의 주행 경로 계획을 수행하는데 더욱 유용하고 신뢰성 있는 정보를 제공할 수 있다. Also, if the performance proposed in the embodiment is within the prediction range

And the number K of hypotheses. Referring to FIG. 3, it is a graph showing a change in MAE according to K and an average over the entire time. As K increases, MAE performance improves at all times. At this time, the degree of performance improvement due to the increase of K becomes more prominent as the future view becomes farther. For example,

= 2.0 seconds, when K = 1 is increased to K = 3, a decrease of MAE of about 0.2 can be confirmed. This means that as the number of hypotheses increases, the likelihood of the model becoming more predictable is close to the actual answer. Accordingly, even if a plurality of hypotheses are generated, it is possible to provide more useful and reliable information for carrying out the traveling route planning of the vehicle.

FIG. 4 is a block diagram illustrating a configuration of a path prediction system according to an embodiment. FIG. 5 is a flowchart illustrating a path prediction method of a path prediction system according to an embodiment.

The path estimation system 100 is for predicting a future path of a neighboring vehicle by performing a deep learning learning based on a joint type circular neural network composed of at least one circular neural network structure and includes an encoder 410 and a decoder 420 . These components may be representations of different functions performed by the processor in accordance with control commands provided by the program code stored in the path prediction system 100. [ The components may control the path prediction system 100 to perform the steps 510 to 520 that the path prediction method of FIG. 5 includes. At this time, the components may be implemented to execute an instruction according to code of an operating system and code of at least one program included in the memory.

The processor of the path prediction system 100 may load the program code stored in the file of the program for the path prediction method into the memory. For example, when the program is executed in the path prediction system 100, the processor can control the path prediction system to load the program code from the file of the program into the memory under the control of the operating system. At this time, each of the encoder 410 and the decoder unit 420 included in the processor and the processor executes the instructions of the corresponding part of the program code loaded in the memory, so as to execute the steps 510 to 520 of the processor Other functional expressions.

In step 510, the encoder 410 may analyze the sequence information based on the traveling data of the surrounding vehicle existing in the vicinity of the car or the car. The encoder 410 can analyze the sequence information including the driving state information of the vehicle and the traveling route information of the surrounding vehicle by performing the deep learning learning based on the traveling data of the surrounding vehicle existing around the vehicle or the vehicle . The encoder 410 receives driving data including the speed of the own vehicle, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicles, and the relative speed of the surrounding vehicles collected from the sensors of the vehicle or the surrounding vehicle, The pattern information between the vehicle and the surrounding vehicles can be extracted and summarized by analyzing the sequence information based on the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicles, and the relative speed of the surrounding vehicles. At this time, the encoder 410 may be configured as a mixture of FC (Fully-Connected) neural network layer and LSTM (Long-Short Term Memory) circular neural network layer. The FC neural network layer of the encoder 410 is composed of affine transformation and ReLU (Rectified Linear Unit) activation functions, and it can extract the relationship information related to the motion between the car and the surrounding vehicle, . The LSTM recurrent neural network layer of the encoder 410 has a structure in which a plurality of LSTM recurrent neural networks are connected, and each cell vector can be acquired by performing a recursive operation on an input sequence in a plurality of LSTMs.

In operation 520, the decoder 420 may perform a beam search on the analysis result of analyzing the sequence information to predict the future path of the neighboring vehicle. The decoder 420 repeats the recursive operation based on the beam search from the summarized summary sequence as it analyzes the sequence information until it reaches a predetermined predicted time to generate an output sequence and generates a preset sequence based on the generated output sequence It is possible to output more hypotheses than possibility. At this time, the decoder 420 may include at least one FC neural network layer, at least one LSTM loop neural network layer, a Softmax layer for calculating probability distribution, and an embedding layer for feeding back the output. Specifically, the decoder 420 passes the recursively calculated output value to the Softmax layer using a plurality of cell vectors received from the encoder in the FC neural network layer of the decoder and the LSTM circular neural network layer, A predetermined number of candidates for a position of a vehicle or a neighboring vehicle are selected by applying a beam search based on a conditional probability, and the selected candidate is subjected to the embedding of the occupancy grid map, Feedback can be made through the layer. The decoder 420 is composed of a plurality of matrices corresponding to the vertical axis and the horizontal axis of the occupancy grid map at the embedding layer of the decoder and corresponding to the selected grid coordinates based on the beam search from each matrix After extraction, the extracted column vectors can be combined (vector concatenated) and fed back to the input of the LSTM recurrent neural network layer of the decoder.

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device As shown in FIG. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (14)

A path prediction method performed by a path prediction system including an encoder and a decoder,
Analyzing sequence information on the basis of the driving data of the peripheral vehicle existing around the vehicle or the vehicle; And
Estimating, in the decoder, a future path of a neighboring vehicle by applying a beam search to the analyzed result of analyzing the sequence information
Lt; / RTI >
Wherein deep learning learning is performed based on a combined circular neural network composed of at least one or more circular neural network structures,
Wherein analyzing the sequence information comprises:
Wherein the encoder comprises a mix of FC (Fully-Connected) neural network layer and LSTM (Long-Short Term Memory)
The FC neural network layer of the encoder is composed of an affine transformation and a rectilinear linear unit (ReLU) activation function, extracts relationship information related to movement between the car and the neighboring vehicle,
The LSTM recurrent neural network layer of the encoder has a structure in which a plurality of LSTM circulating neural networks are connected, and each cell vector is obtained by performing a recursive operation on the input sequence in the plurality of LSTMs
Wherein the path prediction method comprises: The method according to claim 1,
Wherein analyzing the sequence information comprises:
Analyzing the sequence information including the driving state information of the vehicle and the traveling route information of the surrounding vehicle by executing the deep learning learning based on the traveling data of the surrounding vehicle existing around the vehicle or the vehicle
And estimating the path. 3. The method of claim 2,
Wherein analyzing the sequence information comprises:
The traveling data including the speed of the vehicle, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle are input from the vehicle or the sensor of the peripheral vehicle, Extracting and summarizing the pattern information between the vehicle and the surrounding vehicles by analyzing the sequence information based on the speed, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle
And estimating the path. delete The method according to claim 1,
Wherein the step of predicting a future route of the neighboring vehicle comprises:
And generating an output sequence by repeating a recursive operation based on the beam search from the summarized summation sequence by analyzing the sequence information until a predetermined predicted time is reached by analyzing the sequence information to generate an output sequence based on the generated output sequence, Outputting a hypothesis more than the set possibility
And estimating the path. A path prediction method performed by a path prediction system including an encoder and a decoder,
Analyzing sequence information on the basis of the driving data of the peripheral vehicle existing around the vehicle or the vehicle; And
Estimating, in the decoder, a future path of a neighboring vehicle by applying a beam search to the analyzed result of analyzing the sequence information
Lt; / RTI >
Wherein deep learning learning is performed based on a combined circular neural network composed of at least one or more circular neural network structures,
Wherein the step of predicting a future route of the neighboring vehicle comprises:
And generating an output sequence by repeating a recursive operation based on the beam search from the summarized summation sequence by analyzing the sequence information until a predetermined predicted time is reached by analyzing the sequence information to generate an output sequence based on the generated output sequence, Wherein the decoder comprises at least one FC neural network layer, at least one LSTM loop neural network layer, a Softmax layer for computing a probability distribution, and an embedding layer for feedbacking the output
≪ / RTI >
An output value recursively calculated by using a plurality of cell vectors received from the encoder in the FC neural network layer and the LSTM loop neural network layer of the decoder is passed through the Softmax layer so that the own vehicle or an adjacent vehicle can be classified into Occupancy Grid Map), applying a beam search based on the conditional probability to select a predetermined number of candidates for the position of a vehicle or a nearby vehicle, and transmitting the selected candidate to the feedback via the embedding layer Step
And estimating the path. The method according to claim 6,
Wherein the step of predicting a future route of the neighboring vehicle comprises:
And a plurality of matrices corresponding to the number of grids in the vertical axis and the horizontal axis of the occupation grid map in the embedding layer of the decoder, wherein a column vector corresponding to the selected grid coordinates is extracted from each matrix based on the beam search , Coupling the extracted column vectors (vector concatenation), and feeding back to the input of the LSTM loop neural network layer of the decoder
And estimating the path. In the path prediction system,
An encoder for analyzing the sequence information based on the traveling data of the vehicle or the surrounding vehicle existing around the vehicle or the vehicle; And
A decoder for predicting a future path of a nearby vehicle by applying a beam search to an analysis result obtained by analyzing the sequence information;
Lt; / RTI >
Wherein deep learning learning is performed based on a combined circular neural network composed of at least one or more circular neural network structures,
The encoder comprising:
Wherein the encoder comprises a mix of FC (Fully-Connected) neural network layer and LSTM (Long-Short Term Memory)
The FC neural network layer of the encoder is composed of an affine transformation and a rectilinear linear unit (ReLU) activation function, extracts relationship information related to movement between the car and the neighboring vehicle,
The LSTM recurrent neural network layer of the encoder has a structure in which a plurality of LSTM circulating neural networks are connected, and each cell vector is obtained by performing a recursive operation on the input sequence in the plurality of LSTMs
The path prediction system comprising: 9. The method of claim 8,
The encoder comprising:
The encoder analyzes the sequence information including the driving state information of the vehicle and the traveling route information of the surrounding vehicles by performing the deep learning learning based on the traveling data of the surrounding vehicle existing in the vicinity of the vehicle or the vehicle
The path prediction system comprising: 10. The method of claim 9,
The encoder comprising:
The traveling data including the speed of the vehicle, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle are input from the vehicle or the sensor of the peripheral vehicle, Extracts and summarizes the pattern information between the vehicle and the surrounding vehicles by analyzing the sequence information based on the speed, the steering angular speed of the vehicle, the relative coordinates of the surrounding vehicle, and the relative speed of the surrounding vehicle
The path prediction system comprising: delete 9. The method of claim 8,
The decoder includes:
Generating an output sequence by repeating a recursive operation based on the beam search from the summarized summary sequence by analyzing the sequence information until a predetermined predicted time is reached, and generating an output sequence based on the generated output sequence, To output
The path prediction system comprising: In the path prediction system,
An encoder for analyzing the sequence information based on the traveling data of the vehicle or the surrounding vehicle existing around the vehicle or the vehicle; And
A decoder for predicting a future path of a nearby vehicle by applying a beam search to an analysis result obtained by analyzing the sequence information;
Lt; / RTI >
Wherein deep learning learning is performed based on a combined circular neural network composed of at least one or more circular neural network structures,
The decoder includes:
Generating an output sequence by repeating a recursive operation based on the beam search from the summarized summary sequence by analyzing the sequence information until a predetermined predicted time is reached, and generating an output sequence based on the generated output sequence, Wherein the decoder comprises at least one FC neural network layer, at least one LSTM loop neural network layer, a Softmax layer for computing a probability distribution, and an embedding layer for feedbacking the output,
An output value recursively calculated by using a plurality of cell vectors received from the encoder in the FC neural network layer and the LSTM loop neural network layer of the decoder is passed through the Softmax layer so that the own vehicle or an adjacent vehicle can be classified into Occupancy Grid Map), applying a beam search based on the conditional probability to select a predetermined number of candidates for the position of a vehicle or a nearby vehicle, and transmitting the selected candidate to the feedback via the embedding layer doing
The path prediction system comprising: 14. The method of claim 13,
The decoder includes:
And a plurality of matrices corresponding to the number of grids in the vertical axis and the horizontal axis of the occupation grid map in the embedding layer of the decoder, wherein a column vector corresponding to the selected grid coordinates is extracted from each matrix based on the beam search , Combines the extracted column vectors (vector concatenation), and feeds them back to the input of the LSTM recurrent neural network layer of the decoder
The path prediction system comprising:

Images