KR102552051B1 - Vehicle risk prediction device and method therefor - Google Patents

Abstract

In one aspect of the disclosure, at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations, wherein the operations include: First input data and second input data are applied to a first long short term memory (LSTM) and a second LSTM model, respectively, to obtain 1-1 output data and 1-2 output data, and the 1-1 output The regression model is trained based on data, the 1-2 output data and risk label data, and the first input data is first array data determined based on object data in front of the vehicle and CAN (controller area of the vehicle) It is generated by combining second arrangement data determined based on network) data, and the second input data is a vehicle risk prediction device determined based on driver data of the vehicle.

Description

Vehicle risk prediction device and method {VEHICLE RISK PREDICTION DEVICE AND METHOD THEREFOR}

The present disclosure relates to an apparatus and method for predicting vehicle risk.

Advanced Driver Assistance System (ADAS) is a driver assistance system that allows the vehicle to recognize and determine dangerous situations while driving through state-of-the-art detection sensors and control them on its own or informs the driver of risk factors in advance. Despite the fact that many automakers are applying intelligent driver assistance systems to their vehicles, the number of traffic accidents in 2019 increased by 5.7% compared to the previous year, and 75.5% of them were vehicle-to-vehicle accidents. Therefore, in order to more accurately recognize the risk situation of a vehicle-to-vehicle accident, research on vehicle risk prediction is being conducted in various fields.

Republic of Korea Patent Publication 10-2019-0103078 US Patent Publication 2017-0286826

Various examples of the present disclosure are to provide an apparatus and method for predicting vehicle risk using a front camera, an inside camera, and CAN (controller area network) data.

The technical problems to be achieved in various examples of the present disclosure are not limited to those mentioned above, and other technical problems not mentioned above can be solved by those skilled in the art from various examples of the present disclosure to be described below. can be considered by

In one aspect of the disclosure, at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations, wherein the operations include: First input data and second input data are applied to a first long short term memory (LSTM) and a second LSTM model, respectively, to obtain 1-1 output data and 1-2 output data, and the 1-1 output The regression model is trained based on data, the 1-2 output data and risk label data, and the first input data is first array data determined based on object data in front of the vehicle and CAN (controller area of the vehicle) It is generated by combining second arrangement data determined based on network) data, and the second input data is a vehicle risk prediction device determined based on driver data of the vehicle.

For example, the object data may include type data of at least one object included in front image data photographed by a front camera of the vehicle and coordinate data of the at least one object.

For example, the first arrangement data may be determined based on the remaining objects except for an object whose coordinate data is not included in a preset area and an object whose type data does not match a preset type among the at least one object. there is.

For example, the second arrangement data may be determined based on whether the CAN data is mapped to at least one of a plurality of risky action types included in a preset risky action type table.

For example, the driver data may include gaze angle data of the driver and headpose data of the driver included in internal image data captured by an internal camera of the vehicle.

For example, the risk label data may correspond to the risk level of the vehicle classified based on the CAN data and the driver data.

For example, the dimensional sizes of the 1-1 output data and the 1-2 output data may be the same.

For example, the operations may smooth prediction data obtained in the process of learning the regression model in units of a preset window.

In another aspect of the present disclosure, a transceiver; at least one processor; and at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations, wherein the operations include: First input data and second input data are applied to a first long short term memory (LSTM) and a second LSTM model, respectively, to obtain 1-1 output data and 1-2 output data, wherein the first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on controller area network (CAN) data of the vehicle, and the second input data is the driver of the vehicle data, learning the regression model based on the 1-1 output data, the 1-2 output data and risk label data, and receiving real-time object data, real-time CAN data and real-time driver data through the transceiver, , A vehicle risk prediction device that predicts the risk of the vehicle from the first LSTM model, the second LSTM model, and the regression model based on the real-time object data, the real-time CAN data, and the real-time driver data.

In another aspect of the present disclosure, a vehicle risk prediction method performed by a vehicle risk prediction device, by applying first input data and second input data to a first long short term memory (LSTM) and a second LSTM model, respectively Obtaining 1-1st output data and 1-2nd output data; and learning the regression model based on the 1-1 output data, the 1-2 output data, and the risk label data, wherein the first input data is determined based on object data in front of the vehicle. A method for predicting vehicle risk, generated by combining first sequence data and second sequence data determined based on CAN (controller area network) data of the vehicle, wherein the second input data is determined based on driver data of the vehicle am.

For example, the object data may include type data of at least one object included in front image data photographed by a front camera of the vehicle and coordinate data of the at least one object.

For example, the first array data is determined based on the remaining objects except for an object whose coordinate data is not included in a preset area and an object whose type data does not match a preset type among the at least one object, and , The second array data may be determined based on whether the CAN data is mapped to at least one of a plurality of risky action types included in a preset risky action type table.

For example, receiving real-time object data, real-time CAN data and real-time driver data through a transceiver; and predicting the risk of the vehicle from the first LSTM model, the second LSTM model, and the regression model based on the real-time object data, the real-time CAN data, and the real-time driver data.

The various examples of the present disclosure described above are only some of the preferred examples of the present disclosure, and various examples in which the technical features of the various examples of the present disclosure are reflected are detailed descriptions to be detailed below by those of ordinary skill in the art. It can be derived and understood based on.

According to various examples of the present disclosure, the following effects are obtained.

According to various examples of the present disclosure, an apparatus and method for predicting vehicle risk using a front camera, an inside camera, and CAN (controller area network) data may be provided.

Effects obtainable from various examples of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned are clearly derived to those skilled in the art based on the detailed description below and can be understood.

The accompanying drawings are provided to aid understanding of various examples of the present disclosure, and provide various examples of the present disclosure together with detailed descriptions. However, technical features of various examples of the present disclosure are not limited to specific drawings, and features disclosed in each drawing may be combined with each other to form a new embodiment. Reference numerals in each figure mean structural elements.
1 is a block diagram of an apparatus for predicting vehicle risk according to an example of the present disclosure.
2 is a schematic diagram of a vehicle risk prediction model performed by an apparatus for predicting vehicle risk according to an example of the present disclosure.
3 illustrates a risk behavior type table for determining second arrangement data according to an example of the present disclosure.
4A and 4B are for explaining driver data.
5 is for explaining a risk level of a vehicle according to an example of the present disclosure.
6A to 6C are for explaining pre-processing of risk label data.
7A to 7B are for explaining performance of a regression model according to an example of the present disclosure.
8 is a flowchart of a vehicle risk prediction method according to an example of the present disclosure.

Hereinafter, implementations according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to describe exemplary implementations of the invention, and is not intended to represent the only implementations in which the invention may be practiced. The following detailed description includes specific details for the purpose of providing a thorough understanding of the present invention. However, one skilled in the art recognizes that the present disclosure may be practiced without these specific details.

In some cases, in order to avoid obscuring the concept of the present disclosure, well-known structures and devices may be omitted or may be shown in block diagram form centering on core functions of each structure and device. In addition, the same reference numerals are used to describe like elements throughout the present disclosure.

Since various examples according to the concept of the present invention can be made with various changes and have various forms, various examples will be illustrated in the drawings and described in detail in the present disclosure. However, this is not intended to limit the various examples according to the concept of the present invention to specific disclosed forms, and includes modifications, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component, Similarly, the second component may also be referred to as the first component.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle. Expressions describing the relationship between components, such as "between" and "directly between" or "directly adjacent to" should be interpreted similarly.

In various examples of this disclosure, “/” and “,” should be interpreted as indicating “and/or”. For example, “A/B” may mean “A and/or B”. Furthermore, “A, B” may mean “A and/or B”. Furthermore, “A/B/C” may mean “at least one of A, B and/or C”. Furthermore, “A, B, C” may mean “at least one of A, B and/or C”.

In various examples of this disclosure, “or” should be interpreted as indicating “and/or”. For example, “A or B” may include “only A”, “only B”, and/or “both A and B”. In other words, "or" should be interpreted as indicating "in addition or alternatively."

The terms used in this disclosure are only used to describe specific various examples, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, the terms "comprise" or "having" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers, It should be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present disclosure, it should not be interpreted in an ideal or excessively formal meaning. don't Hereinafter, various examples of the present disclosure will be described in detail with reference to the accompanying drawings.

Vehicle risk prediction device

1 is a block diagram of an apparatus for predicting vehicle risk according to an example of the present disclosure.

Referring to FIG. 1 , an apparatus 100 for predicting vehicle risk according to an example of the present disclosure includes a transceiver 110 , a processor 120 and a memory 130 .

The transceiver 110 may be connected to the processor 120 and may transmit and/or receive wired/wireless signals. For example, the transceiver 110 may be connected to various electronic control units (ECUs) of the vehicle through an in-vehicle communication network. Here, the in-vehicle communication network may be, for example, CAN (controller area network), FlexRay, Lin (local interconnect network), or Ethernet, but is not limited thereto.

Alternatively, the transceiver 110 may be connected to a server and/or a terminal through a wired/wireless communication network. Here, the wireless communication network may include a mobile communication network, a wireless LAN, a local area wireless communication network, and the like. For example, a wireless communication network may include LTE, LTE Advance (LTE-A), code division multiple access (CDMA), wideband CDMA (WCDMA), universal mobile telecommunications system (UMTS), wireless broadband (WiBro), or global system (GSM). for Mobile Communications) and the like. For example, the wireless communication network may include at least one of wireless fidelity (WiFi), Bluetooth, Bluetooth Low Energy (BLE), Zigbee, near field communication (NFC), or radio frequency (RF).

A server and/or a terminal connected to the transceiver 110 through a wired/wireless communication network, for example, data collected from a vehicle to be used as input data of an artificial intelligence model according to various examples of the present disclosure, learning data, and / Or map data and the like can be exchanged with the transceiver 110 .

The transceiver 110 may include a transmitter and a receiver. The transceiver 110 may be used interchangeably with a radio frequency (RF) unit. The transceiver 110 may exchange signals or data with various ECUs, servers, and terminals of the vehicle under the control of the processor 120 .

The processor 120 may include at least one, controls the memory 130 and/or the transceiver 110, and may be configured to implement the descriptions, functions, procedures, suggestions, methods and/or operational flowcharts of the present disclosure. can For example, the processor 120 may receive signals or data through the transceiver 110 and store information included in the signals or data in the memory 130 . Also, the processor 120 may process information stored in the memory 130 to generate signals or data, and transmit the generated signals or data through the transceiver 110 .

The memory 130 may include at least one, may be connected to the processor 120, and may store various information related to the operation of the processor 120. For example, memory 130 may perform some or all of the processes controlled by processor 120, or may provide instructions for performing descriptions, functions, procedures, suggestions, methods, and/or operational flowcharts of the present disclosure. It can store the software code that contains it. Alternatively, the memory 130 may store data according to various examples of the present disclosure or a vehicle risk prediction model according to various examples.

Hereinafter, various operation examples of the vehicle risk prediction apparatus 100 will be described. The following various operation examples may be included in the operation of at least one processor 120 described above.

2 is a schematic diagram of a vehicle risk prediction model performed by an apparatus for predicting vehicle risk according to an example of the present disclosure. In the present disclosure, the vehicle risk prediction model 200 may refer to an artificial intelligence model used by the vehicle risk prediction apparatus 100 to predict vehicle risk. In other words, the vehicle risk level prediction model 200 may be an artificial intelligence model for performing various operation examples of the vehicle risk level prediction apparatus 100 according to various examples of the present disclosure. Operations of each artificial intelligence model included in the vehicle risk prediction model 200 may be included in the operation of the processor 120 described above.

Referring to FIG. 2 , the vehicle risk prediction model 200 includes a first long short term memory (LSTM) model 210, a second LSTM model 220, and a regression model 230.

In the present disclosure, an LSTM model (or LSTM structure) is a type of recurrent neural network (RNN) model and includes a hidden-state and a cell-state responsible for long-term information. In the hidden-state, h _t-1 is the hidden-state initial vector, and h _t is the new hidden-state vector. In cell-state, c _t-1 is the cell-state initial vector, and c _t is the new cell-state vector. The LSTM model determines how much to reflect the input data (u _t ), the hidden-state initial vector, and the cell-state initial vector with respect to the new vector based on the hidden-state and cell-state, and generates an output value (h _t ).

The first LSTM model 210 receives the first input data and generates 1-1 output data, and the second LSTM model 220 receives the second input data and generates 1-2 output data. That is, the processor 120 applies the first input data and the second input data to the first LSTM model 210 and the second LSTM model 220, respectively, to obtain the 1-1 output data and the 1-2 output data Acquire The 1-1 output data and the 1-2 output data are features of the first LSTM model 210 and the second LSTM model 220 .

The regression model 230 receives the 1-1st output data and 1-2nd output data, and predicts the risk level of the vehicle based on the authorized 1-1st output data and 1-2nd output data and the risk label data. carry out the learning process for

For example, the regression model 230 compares the output values obtained based on the 1-1st output data and the 1-2nd output data, that is, the prediction data and the risk label data, and performs a learning process through backpropagation. can be performed. Backpropagation is performed based on a loss model associated with the error between prediction data and risk label data. At this time, backpropagation is performed in the direction of the regression model 230 as well as the first LSTM model 210 and the second LSTM model 220 described above. Therefore, risk label data is involved in learning the entire AI model.

The regression model 230 may have a structure including a dense layer.

Summarizing the above-described vehicle risk prediction model 200 of the present disclosure, different input data are applied to the first LSTM model 210 and the second LSTM model 220 to obtain respective output data (ie, features) and has a structure in which the regression model 230 is trained by applying the obtained output data to the regression model 230 .

Hereinafter, each artificial intelligence model included in the vehicle risk prediction model 200 will be described in detail.

- 1st LSTM model and 2nd LSTM model

As described above, the first LSTM model 210 and the second LSTM model 220 receive first input data and second input data, which are different data, respectively.

The first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on CAN data of the vehicle.

The object data includes type data of at least one object and coordinate data of at least one object included in front image data photographed by a front camera of the vehicle. That is, the object data is data associated with an object such as an obstacle located in front of the vehicle while the vehicle is driving.

The first array data is determined based on object data and may be an array having a size of (n, o). Here, n is a value corresponding to the number of first input data and the number of second input data, and is a value corresponding to the number of time series data, and o is the number of object types used to determine the first array data.

For example, the first array data is based on the remaining objects except for objects whose coordinate data is not included in a preset area among at least one object included in the front image data and objects whose type data does not match a preset type It is decided. In this case, the number of types of the remaining objects may correspond to the above-mentioned o.

A value of each element of the first array data may correspond to the number of each object included in the front image data at a specific point in time. For example, based on the coordinate data and the type data, 12 objects among all objects included in the front image data are used to determine the first array data, and only two first objects and two twelfth objects are detected among the 12 objects. In this case, the first array data may be determined as (2, 0, ..., 0, 2).

CAN data is data acquired about a vehicle through CAN communication, and may include, for example, various parameters used in CAN communication, such as a steering angle, acceleration, speed, and heading of a vehicle.

The second array data is determined based on the CAN data and may be an array having a size of (n, d). Here, d is the number of risk behavior types.

3 illustrates a risk behavior type table for determining second arrangement data according to an example of the present disclosure.

Referring to FIG. 3 , a total of 11 risky behavior types for determining second array data are classified according to the above-described CAN data, and at least one risky behavior type used for determining second array data among the 11 risky behavior types can be selected. The number of selected at least one type of risky behavior may correspond to d described above.

The second arrangement data is determined according to whether CAN data is mapped to at least one of a plurality of risky action types included in the preset risky action type table as shown in FIG. 3 .

A value of each element of the second array data may be 1 when the CAN data is mapped to a risky behavior type at a specific time point, or 0 otherwise. For example, when d = 10 and CAN data at a specific time point is mapped only to the first risk behavior type, the second array data may be determined as (1, 0, ..., 0, 0).

First input data obtained by combining the above-described first array data and second array data is used as input data of the first LSTM model 210 . The first input data is converted into an array extended by a time step having a size of s before being applied to the first LSTM model 210 . Here, s is an integer greater than or equal to 1 and may have the same value as the size of a smoothing window to be described later. Accordingly, the first input data may be an array having a size of (n, s, o + d).

The second input data is determined based on driver data of the vehicle.

The driver data includes the driver's gaze angle data and the driver's headpose data included in the internal image data captured by the vehicle's internal camera. Here, the internal image data may include a front image and/or a side image of the driver.

4A and 4B are for explaining driver data.

Referring to FIG. 4A , gaze angle data included in driver data may be [Lphi, Ltheta, Rphi, Rtheta]. Here, Lphi and Ltheta are the y-axis angle and the x-axis angle of the left eye, respectively, and Rphi and Rtheta are the y-axis angle and the x-axis angle of the right eye, respectively.

Referring to FIG. 4B , head pose data included in driver data may be [Roll, Pitch, Yaw]. Here, Roll, Pitch, and Yaw are the roll angle, pitch angle, and yaw angle of the driver's head, respectively.

The second input data is determined as an array having as an element a value obtained by dividing each of the parameters included in the above-described driver data by 360. The second input data may be an array having a size of (n, 7) according to the number of gaze angle data and head pose data. Like the first input data, the second input data is converted into an array extended in units of time having a size of s before being applied to the second LSTM model 220 . Accordingly, the second input data may be an array having a size of (n, s, 7).

The processor 120 combines the first arrangement data determined based on the object data in front of the vehicle and the second arrangement data determined based on the CAN data of the vehicle before applying the first input data and the second input data to the LSTM model to obtain First input data may be generated, and the generated first input data may be applied to the first LSTM model 210 . Alternatively, the processor 120 may apply the first input data pre-stored in the memory to the first LSTM model 210 .

The processor 120 may determine second input data based on driver data of the vehicle before applying the first and second input data to the LSTM model. Alternatively, the processor 120 may apply the second input data pre-stored in the memory to the second LSTM model 220 .

The first LSTM model 210 and the second LSTM model 220 generate 1-1 output data and 1-2 output data based on the authorized first input data and second input data. Dimensional sizes of the generated 1-1 output data and 1-2 output data are identical to each other. For example, the dimensional size of each of the 1-1 output data and the 1-2 output data may be identically (n, m). Here, m is a preset feature size (ie, output size) for each LSTM model.

According to the first LSTM model 210 and the second LSTM model 220 of the present disclosure described above, the input data applied to each LSTM model, that is, the first input data and the second input data have the same weight and each is authorized. Therefore, in the regression model 230 to be described later, the importance of driver data associated with the second input data becomes higher than the importance of object data and CAN data associated with the first input data, and accordingly, the performance of risk prediction can be improved. .

- Regression model

The regression model 230 receives 1-1 output data and 1-2 output data from the first LSTM model 210 and the second LSTM model 220 . The input data of the regression model 230 may be an array having a size of (n, 2m) because it is data obtained by combining the 1-1 output data and the 1-2 output data.

The regression model 230 performs a learning process for predicting the risk level of the vehicle based on the 1-1 output data, the 1-2 output data, and the risk label data.

The risk level label data corresponds to the level of risk of the vehicle classified based on the above-described CAN data and driver data. The risk level of the vehicle may have a plurality of levels according to values of each of the CAN data and the driver data.

5 is for explaining a risk level of a vehicle according to an example of the present disclosure.

Referring to FIG. 5 , the risk level of a vehicle may belong to dangerous, normal, and safe areas according to a driver's condition (ie, driver data) and a vehicle's dynamic status (ie, CAN data). In addition, although not shown, a specific risk level in each area may be represented by a value of 0 to 5. That is, the risk label data may refer to a series of data sets having risk levels for each time series data (n) based on the first input data and the second input data.

Risk label data can be preprocessed.

6A to 6C are for explaining pre-processing of risk label data.

Referring to FIGS. 6A to 6C , for example, the processor 120 may exclude data having a risk level of 0 from risk label data. When there is a plurality of risk label data excluding the risk level 0 data, the processor 120 may perform an average operation on the plurality of risk label data. The processor 120 may perform average operation-processed risk label data for each preset window unit. The processor 120 may scale the smoothed risk label data to have a value of 0 to 1.

으로 나타낼 수 있다. 여기서,

는 예측 값(예를 들어, 예측 데이터)이고,

는 실제 값(예를 들어, 위험도 라벨 데이터)이다. 이와 같은 RMSLE 모델은 아웃라이어(outlier)에 강건하고, 저측정(under estimation)에 보다 큰 패널티를 부여할 수 있다. 따라서, 상대적으로 적은 양의 위험 데이터도 커버할 수 있다.The regression model 230 uses a loss model in a learning process. For example, the regression model 230 may use a root mean squared log error (RMSLE) model. The RMSLE model is

can be expressed as here,

is the predicted value (e.g., predicted data),

is the actual value (eg, risk label data). Such an RMSLE model is robust to outliers and can impose a larger penalty on under estimation. Therefore, even a relatively small amount of risk data can be covered.

The regression model 230 performs backpropagation based on the loss model as described above. Backpropagation is performed in the direction of the first LSTM model 210 and the second LSTM model 220 in the regression model 230 as described above.

Prediction data obtained in the process of learning the regression model 230 may be post-processed. For example, the processor 120 may smooth the prediction data acquired from the regression model 230 in units of a preset window. For example, when the predicted time point is t and the preset window unit is 5, the processor 120 may perform an average operation on the predicted data of time points (t-4) to time t and use it as the predicted data of time t.

7A to 7B are for explaining performance of a regression model according to an example of the present disclosure.

Referring to FIG. 7A , when the loss model is adopted as a mean squared error (MSE) model and the predicted data is not post-processed, it can be seen that there is a slight difference between the correct value and the predicted value.

Referring to FIG. 7B , when the loss model is adopted as the RMSLE model and the predicted data is post-processed as in the present disclosure, it can be seen that the gap between the correct value and the predicted value is reduced and the bouncing value is reduced.

The processor 120 may learn the vehicle risk level prediction model 200 according to various examples of the present disclosure and predict the risk level of the vehicle based on the trained vehicle risk level prediction model 200 .

For example, the processor 120 receives real-time object data, real-time CAN data, and real-time driver data through the transceiver 110, and generates a first LSTM model 210 based on the real-time object data, real-time CAN data, and real-time driver data. ), the second LSTM model 220 and the regression model 230 can predict the risk of the vehicle.

According to the above-described vehicle risk predicting device of the present disclosure, when predicting the risk of a vehicle, the importance level may be differently considered for each type of data and input to an artificial intelligence model. Accordingly, it is possible to have more improved risk prediction performance.

Hereinafter, a vehicle risk prediction method performed by the above-described vehicle risk prediction apparatus will be described. Hereinafter, detailed descriptions of overlapping parts with those described above will be omitted.

Vehicle Hazard Prediction Method

8 is a flowchart of a vehicle risk prediction method according to an example of the present disclosure.

Referring to FIG. 8 , in S110, the vehicle risk prediction apparatus applies the first input data and the second input data to the first LSTM and the second LSTM model 220, respectively, to generate the 1-1 output data and the 1-2 output data. Get the output data. Here, the first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on CAN data of the vehicle, and the second input data corresponds to driver data of the vehicle. is determined based on

In S120, the vehicle risk prediction device applies the 1-1 output data and the 1-2 output data to the regression model 230, and based on the 1-1 output data and the 1-2 output data and the risk label data. Thus, the regression model 230 is trained.

In S130 , the vehicle risk prediction apparatus receives real-time object data, real-time CAN data, and real-time driver data through the transceiver 110 .

In S140, the vehicle risk prediction apparatus calculates the vehicle risk from the first LSTM model 210, the second LSTM model 220, and the regression model 230 based on the received real-time object data, real-time CAN data, and real-time driver data. predict

Since the examples of the proposed schemes in the above description may also be included as one of the implementation methods of the present disclosure, it is obvious that they can be regarded as a kind of proposed schemes. In addition, the above-described proposed schemes may be implemented independently, but may also be implemented in a combination (or merged) form of some proposed schemes.

Examples of the present disclosure disclosed as described above are provided to enable those skilled in the art to implement and practice the present disclosure. Although the above has been described with reference to examples of the present disclosure, a person skilled in the art may variously modify and change the examples of the present disclosure. Thus, the present disclosure is not intended to be limited to the examples set forth herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

100: vehicle risk prediction device
110: transceiver 120: processor
130: memory

Claims (13)

at least one processor; and
at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations;
These actions are
Obtaining 1-1 output data and 1-2 output data by applying the first input data and the second input data to a first long short term memory (LSTM) and a second LSTM model, respectively;
Learning a regression model based on the 1-1 output data, the 1-2 output data and risk label data;
The first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on CAN (controller area network) data of the vehicle,
The second input data is determined based on driver data of the vehicle.
Vehicle risk prediction device.
According to claim 1,
The object data includes type data of at least one object included in front image data photographed from a front camera of the vehicle and coordinate data of the at least one object,
Vehicle risk prediction device.
According to claim 2,
The first arrangement data is determined based on the remaining objects except for objects whose coordinate data is not included in a preset area among the at least one object and objects whose type data does not match a preset type,
Vehicle risk prediction device.
According to claim 1,
The second arrangement data is determined based on whether the CAN data is mapped to at least one of a plurality of risky action types included in a preset risky action type table.
Vehicle risk prediction device.
According to claim 1,
The driver data includes driver's gaze angle data and headpose data of the driver included in internal image data captured from an internal camera of the vehicle,
Vehicle risk prediction device.
According to claim 1,
The risk label data corresponds to the risk level of the vehicle classified based on the CAN data and the driver data.
Vehicle risk prediction device.
According to claim 1,
The dimensional size of each of the 1-1 output data and the 1-2 output data is the same as each other,
Vehicle risk prediction device.
According to claim 1,
These actions are
Smoothing the prediction data obtained in the learning process of the regression model in units of preset windows,
Vehicle risk prediction device.
transceiver;
at least one processor; and
at least one memory operatively connected to the at least one processor to store at least one instruction that causes the at least one processor to perform operations;
These actions are
Obtaining 1-1 output data and 1-2 output data by applying the first input data and the second input data to a first long short term memory (LSTM) and a second LSTM model, respectively;
Here, the first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on controller area network (CAN) data of the vehicle,
The second input data is driver data of the vehicle,
Learning a regression model based on the 1-1 output data, the 1-2 output data and risk label data;
Receiving real-time object data, real-time CAN data, and real-time driver data through the transceiver;
Predicting the risk of the vehicle from the first LSTM model, the second LSTM model, and the regression model based on the real-time object data, the real-time CAN data, and the real-time driver data,
Vehicle risk prediction device.
A vehicle risk prediction method performed by a vehicle risk prediction device,
obtaining 1-1 output data and 1-2 output data by applying the first input data and the second input data to a first long short term memory (LSTM) model and a second LSTM model, respectively; and
Learning a regression model based on the 1-1 output data, the 1-2 output data and risk label data;
The first input data is generated by combining first arrangement data determined based on object data in front of the vehicle and second arrangement data determined based on CAN (controller area network) data of the vehicle,
The second input data is determined based on driver data of the vehicle.
Vehicle Hazard Prediction Method.
According to claim 10,
The object data includes type data of at least one object included in front image data photographed from a front camera of the vehicle and coordinate data of the at least one object,
Vehicle Hazard Prediction Method.
According to claim 11,
The first arrangement data is determined based on the remaining objects except for the object in which the coordinate data is not included in a preset area and the object whose type data does not match the preset type among the at least one object,
The second arrangement data is determined based on whether the CAN data is mapped to at least one of a plurality of risky action types included in a preset risky action type table.
Vehicle Hazard Prediction Method.
According to claim 10,
receiving real-time object data, real-time CAN data, and real-time driver data through a transceiver; and
Predicting the risk of the vehicle from the first LSTM model, the second LSTM model, and the regression model based on the real-time object data, the real-time CAN data, and the real-time driver data,
Vehicle Hazard Prediction Method.

Images